Hi everyone!
Another Downloader release has just arrived, and it's perhaps the most important one since its inception. At least for me!
The first release of Downloader happened in the summer of 2021 as a much more stable and reliable substitute for the older updater. Initially, the focus was on delivering features and writing specifications, but I always had a clear plan to make this tool, not only more robust but also much quicker than any updater running in this kind of device.
Once I delivered all the features (2022 - 2023), I started executing my plan to make this tool as fast as possible. And this release is the result.
With this 2.3 release, Downloader is now over 10x faster than 3 years ago, and 5x faster than a year ago, in a no-updates scenario. Compared to the previous version 2.2.1 from a few weeks ago, this new one is over 2.5x times faster, as you can see in the following graph.
No-updates comparison between v2.2.1 and v2.3 with "balanced" file_checking option.
Yes, you read that right. It can now take just two seconds!
After this, there are no realistic hopes of making it significantly faster. There are a few things here and there that can be tweaked to get marginal improvements but speedrunning is over guys. So time to move on to other things!
From now on, my future work on Downloader will mostly be on new features and small refinements. There is a very cool feature I want to implement, that wasn't possible before, but now it is thanks to this much quicker runtime: An integration with the firmware UI, to show whether there are updates available or not with a new icon.
There is also a chance that I may turn Downloader into a multi-purpose library for a wide range of platforms. This tool has become a really good updater for a certain niche of systems and I believe there is value in generalizing it a bit more, but this is food for another post.
So how I got this to be quicker this time? Introducing the File Checking options.
File Checking options
"file_checking" is a new field for the downloader.ini file that goes under the [mister] section, and right now it can take four different values: "fastest", "balanced", "exhaustive", and "verify_integrity".
By default, it'll use "balanced". This means, it will skip checking for the presence of installed files (which can be slow), unless Downloader detects changes in the incoming databases, or that you freed quite a bit of space in your SD or your external drives.
With "exhaustive", we configure Downloader to have the older behavior, meaning, it will check the existence of all installed files all the time. By doing so, it will try to restore installed files that have been deleted manually. This option is useful for people who are messing with installed files often. But if you treat MiSTer as a closed platform where you only place games and little more, you probably don't need this option.
With "fastest", you are skipping file checking unless strictly needed. This saves around 0.2-0.5s compared to "balanced". It's an interesting option for people that treat MiSTer 100% like a console, but it's not super impactful, so keeping it in "balanced" is a good idea.
And last, with "verify_integrity", Downloader will verify that all already installed files have the proper checksum, which takes a lot of time. This is useful if you suspect that your SD card got corrupted. MiSTer uses ExFAT partitions, which is an old format, and corruptions on it are quite rare but do happen sometimes, unfortunately.
You may find more info about these options in the README.md file.
To implement "file_checking", some architectural changes had to be introduced, as you can see in the documentation (docs folder).
Worker flows in Downloader 2.2.1 (left) and Downloader 2.3 (right)
Compatibility with other regions
In the previous version, we introduced a HTTP Proxy to work around region blockades, but we missed something important that affects users from countries like China: A proper multi-region NTP setup.
So in this new version, I changed the launcher so that it tries to connect to NTPs from different regions if the internal clock doesn't sync correctly. This is critical, because without this, the affected users can't do HTTPS requests, and that means the updater will not work at all.
Thanks to this, users in China with a working HTTP Proxy can finally use Downloader reliably in their MiSTers for the first time.
There is still work to do in this front, since the requirement of a HTTP Proxy is not ideal. I'm also exploring the implementation of mirrors for such regions, which would be the best solution to these problems. If you are interested in helping with this, please feel free to contact me.
Wrapping up
And that's it for now! As always, if you want to check all the details, please check the CHANGELOG (there is a deprecation this time!).
The next update is gonna likely be about Update All, stay tuned, and see you soon!