Here we go! The update I have been working on for the last two months is now running in production. Pixeldrain now uses Reed-Solomon encoding to store files. This encoding scheme has a lot of benefits over the way pixeldrain used to store files:
Files are stored with a 8/4 data/parity ratio. This means that 4 disks would have to fail simultaneously before any data is lost (for reference, pixeldrain has had one disk failure since the project started in 2015).
Since files are equally distributed over 12 different servers, we can use the combined bandwidth of those servers to read the files. Files which were previously only accessible at 1 Gbps can now be read at 8 Gbps.
This saves a lot of storage space. Pixeldrain used to store two copies of every file on two different servers in RAID5. Now that pixeldrain has its own parity calculations we don't need RAID anymore, this saves about 25% storage space. And we don't need redundant copies anymore which saves another 50%.
Because this new system is so space efficient I can afford to store larger files for longer periods of time. Soon I will be increasing storage limits.
Merge request for the new storage system. 3039 lines of code added and 1683 lines removed.
Storage servers migrating files to the new storage system. This uses a lot of bandwidth!