Hey everyone!
Today I’m sharing a big milestone: a binary version of Downloader that -you guessed it- runs much faster. This is a release‑candidate version, not enabled by default in your setups, as I’d like to see broader testing before rolling it out in all channels.
After running Update All, you’ll find it installed as downloader_bin in your Scripts menu. Or you can download it from this post and install it manually by placing it in the Scripts folder on your SD card. It can’t be run on the Desktop, only on MiSTer.
A binary version means the program is compiled to raw machine code, whereas previously it ran under a Python interpreter. This eliminates all interpreter overhead, which is often a significant cost in Python applications. However, compiling Python code into a standalone binary is not straightforward, so most Python projects do not attempt it.
Before commenting on the speed improvements (which are huge), I’d like to explain how I came to work on compiling Downloader into a binary. It’s going to be a bit of a technical explanation, so feel free to jump to the next section if you like.
The unexpected bottleneck
While working on the Downloader 2.0 release, I ran into an unexpected issue.
I managed to clean up the code and optimize all the I/O operations thanks to the new worker‑based architecture (see the 2.0 post linked above for more details). Focusing on I/O is crucial when improving performance for this kind of application, since the network and filesystem layers are the main limiting factors. After all, we’re fetching and installing thousands of files.
But even after getting that part right, I didn’t see the speed improvements I expected. In a no‑updates scenario, I wasn’t saving any time compared to Downloader 1.8, despite refactoring all the relevant parts and simplifying the logic.
Something else was at play: the DE10‑Nano’s slow CPU playing with python's file import system.
My cleanup had introduced more files (while keeping roughly the same total lines of code) and a few extra intermediary objects. Benchmarks showed that the overhead of importing them outweighed the savings from my other improvements.
Sure, it’s well known that Python under an interpreter is slower than compiled C, but you rarely see such a pronounced effect just from importing a few extra modules—unless your CPU is a bottleneck. On the DE10‑Nano, that slowdown was magnified.
As I hinted in the Downloader 2.1 release notes, the solution was to compile the Python modules. That change brought additional benefits, too, so I went all in. The result? Boot time plummeted from over 4 seconds to just 1 second. Compiling virtually eliminated import overhead.
During the work for the 2.0, I also implemented a rather savage and undeniably un‑pythonic optimization for instantiating those additional intermediary objects I mentioned without paying the price. But that story will have to wait for another post. Let me know if you are interested!
Speed improvements
This binary version is about 3 seconds faster than Downloader 2.1, putting us around 7 seconds in a no‑updates scenario. Compared to Downloader 1.8, which was over 13.5 seconds, that’s roughly a 45 % speed increase. In other words, nearly 2× faster than just a month ago.
And it’ll get faster soon.
Closing up
To summarize: after installing this release candidate with update_all, just run downloader_bin to enjoy the speed improvements.
Please report any errors you encounter, that will help me a lot. I didn’t experience any issues during the last week of testing, so I don’t expect it to cause hassle for users. That’s why I call it a release candidate, not an alpha, beta, or experimental release.
That’s all for today.
See you soon!