Hey everyone!
A week ago, I've mentioned that updates will follow much sooner thanks to the new architecture I introduced in Downloader 2.0—and as expected, Downloader 2.1 is already here!
While this is a much smaller release compared to the previous one, it builds on its foundations with several key improvements.
The most notable changes? Speed, once again. Check out this patrons-only poll to let me know whether you'd prefer future updates to focus on speed improvements or new features.
Even Faster in Short Runs
As explained in the Downloader 2.0 post, short runs are the priority for optimization. So I've moved the internal local store loading (which takes ~1 second) to a worker. This means it's now also running on parallel, while databases are being fetched.
Thanks to this every theoretically parallelizable task now runs in parallel which is great milestone. Here’s a flow diagram showing the execution flow of the update process:
This diagram illustrates the flow of the execution. Each job is executed by a worker in parallel.
As some of you might have guessed, there is another more complex diagram that illustrates what happens when unzipping is required by a database, but that's a story for another post.
Result: 30% speed increase when compared to Downloader 1.8.Or, in other words, you save 4+ seconds each time you run update.sh, downloader.sh or update_all.sh when there are no files to update—compared to previous months.
More Realistic Runtime Output
You’ve likely noticed the runtime summary at the end of executions:
It turns out that this time is over 4 seconds behind real time because it doesn't account for the time Python takes to import modules, and in this system, that time is pretty massive. We are talking about 3 seconds, which was indeed a surprise for me. To show a more realistic output, I'm now accounting for that, so that the mismatch between real time and the output run time is just around 1 second—the time the OS takes to run the launcher and boot the process.
I've found a way to reduce this importing time penalty dramatically, and this is something I'm experimenting with right now. Hopefully, it will be the seed of more speed improvements in the near future.
Closing up
There are more bug fixes and improvements that you can read about in the CHANGELOG.
I hope you appreciate this update. I'm already working on the next ones which will hopefully arrive pretty soon.
Again, check the poll to let me know where to put my working time toward.
See you soon!