Guess who said he wouldn't be hacking on his vacation and who hasn't kept his promise.
And I am bringing VERY good news: Donkey Kong DX refactor is finished!
Today I will be explaining why two obstacles made me lose my motivation a year ago. I cannot promise I'll be short!
Due to the hardware limitations, the Game Boy memory map was made so the console can only run 32kb ROMs. So most of you will now be wondering: why are my Game Boy cartridges are bigger than that?
The answer is the Memory Bank Controller (MBC for short). MBC is a little electronic component assembled in the cartridge board that allows the game code to read data further than those 32kb by redirecting the data read calls to other slices of 16kb of data (called banks) of the ROM. It came with a significant caveat, though: you can only map 16kb of data at the same time (the other 16kb is called bank 0 and cannot be remapped).
So, in order to read different data from your ROM, you keep switching banks. That's not a big deal when you are developing your own game from scratch, as you've got full control over how to store and manage your game's data and code.
However, when hacking Game Boy games, you don't own the original game's code and you need to learn how the game was structured and coded if you really want to switch banks without making it crash.
Luckily, DK94 had a lot of free space in bank 0 (which is a precious thing when hacking GB games, trust me) so that wasn't the issue. The problem was the rest of the data.
How graphics are stored in DK94
As you know, the game has different biomes (city, jungle, desert, iceberg...). Every biome has its own tileset: solids, platforms, walls, levers, items, and even some enemies are different in every biome. But there is even more: there are always at least two variations of every biome, and some biomes even have three variations.
If you are curious, in total, there are 33 different biomes. And there are more than 100 different elements (enemies, doors, keys, levers, platforms, etc.) that can be placed in a level. That would be like 3300+ different tilesets but, fortunately, 'a few' of them are shared, so that reduces the number to 'just' 2300+. But then, I am not counting here the tilesets for static screens (title screen, world maps, bonus screens, 24 cutscenes...). I am going to be practical and round up: let's say the total tileset number is 2500 (which is not true, they are more).
All those 2500 tilesets couldn't be stored in a single 16kb bank, of course. So developers had to scatter them across the ROM in different banks.
In order to store so much information and being able to access it, the game has two different databases of graphics. The first and biggest one is for the gameplay. The second one is for static screens, cutscenes, etc. Those databases are pointers to where every tileset is stored (their banks and their offsets in that bank).
But those databases order makes no sense. All graphics are stored in an arbitrary order. Of course, the original developers had the sourcecode that had all tilesets properly labelled, so they could easily move, rearrange, etc. And, probably, they could build the ROM with a single command that distributed the graphics automatically without having to worry where everything was stored ...but, sadly, I don't have the sourcecode.
I am aware there is a disassembly project for this game. A disassembly is a deconstruction of a piece of software you don't own the sourcecode of, so when done properly, you can build an exact 1:1 copy of the original game and do cool things like fixing bugs, improving the game or even port it to other platforms.
But, in the case of DK94 dissasembly, it is mostly incomplete. Code is more or less disassembled (though it is missing some labels I had already documented back in 2023). But graphic banks and the databases pointing to every tileset are not disassembled, they are just binary duplicates of data, and that's exactly the same as grabbing an hexadecimal editor and extracting that bank manually. What I really needed was all pointers and tilesets to be labelled.
Hey, I haven't finished yet with the negativity!
Compression
It was very common to use compression back in the day so they could decrease the ROM size, thus decreasing the cartridges' costs.
I had already colorized other games that used compression, like Amazing Penguin, Spanky's Quest or our most recent release Ninja Gaiden Shadow (whose development started a year before DKDX). All three happen to be developed by Natsume, so they shared the same compression algorithm.
I never got to crack the compression algorithm for those, though, because I didn't exactly need it. All I did was to ignore the algorithm and store our new graphics uncompressed in another bank. Then, modified the game's code so it loaded the new graphics instead of the old ones. That was easy because their graphic tilesets were not 2500+, they were less than 64 for Penguin and Spanky, and less than 96 for Ninja. And, guess what, they were stored in an order that made sense!
When I started DK94 hacking I decided to follow the same route that had given me good results with previousprojects. Since I am not mass-producing cartridges (I leave that to AliExpress and Etsy pirates!), I didn't have to bother about size and I could just increase the ROM size and store the new edited sprites and backgrounds decompressed in new ROM banks. Then, create my own database that redirected the original tileset load calls to my new banks.
And that was the biggest mistake I made with this project.
Because, yes, it was exciting to see promising results in little time. But the truth was that it was making the project exponentially grow into a huge frankenstein that was becoming more and more difficult to maintain every time I replaced a new tileset. Not only because it was a pain in the ass to manually add a new tileset, but also because the code had to select a different database depending on the current stage biome, aaaaaand, some specific stages needed exceptions because, when adding color, not all tiles look the same in all stages.
And you just thought that was all. But no, there is even more.
Tilesets are stored both compressed and uncompressed. Mixed. Again, the developers knew what they were doing. They compressed only the ones that needed compression. Because a Game Boy tile data weigh 16 bytes and compressing so little data, usually can lead to compressed data being bigger than the original decompressed one. Some elements like keys or doors are always stored compressed because they are not a tile but a combination of various tiles and, in those cases, the compression algorithm can find more repeated patterns and give better results..
That means, even if I had cracked the compression algorithm, chances were I wouldn't be able to just replace the graphics in the ROM because if our new edited tiles weighed more than the original game's compressed ones, that would overlap the next tile data and break the entire game.
Good news
I hope this drag has helped you to understand how frustrated I felt after the hack became impossible to maintain. Now, let's go for the good news.
A year and a half ago, it was already obvious that I couldn't keep developing this using the same structure. Believe it or not, every now and then I thought of alternatives and ways to fix, and even take notes of ideas. The project required a total overhaul. I had to start from scratch but doing it the right way. The to-do list was this:
- to reverse engineer the compression algorithm- to develop my own tool to recompress graphics- to reverse engineer and learn how the two big graphics databases in the game work- to rebuild the project so I could inject both compressed and uncompressed graphics into their ORIGINAL banks- to let the project automatically recalculate all graphic databases pointers every time I replace a new tileset
Today, I can say all of these have been crossed out in my notebook :-)
I started on January during my vacation, but it's been the last two weeks when I've been working really hard to get everything done.
I've got a partial disassembly of the graphic banks of DK94. I can replace graphics easier than ever and let the assembler recalculate all graphic pointers for me. I don't have to mess with hundreds of arrays of pointers, all dirty workarounds are removed.
The ROM hack size has decreased from nearly 3 megabytes to less than 1 megabyte! That will allow the chinese pirates to make bootleg cartridges more easily, yay!
Not only that, but I've also disassembled the stage database (stage maps were also compressed using the same algorithm) so we can edit the stages (again, without worrying about their size and their pointers). Stage editing is needed in order to introduce new background tiles that weren't in the original game. I did this before with dirty workarounds, of course, but like the tilesets, now I can edit them easily and store them again using the original game's structure.
What happens now
I have already started redoing all stages using the new format, replacing the tiles in the internal game's database. It will be a long process that might take weeks. But afterwards, I'll be able to insert new colorized content at last!
On the other hand, I have also improved my custom colorization tool with new features to ease the stage colorization process to the graphic artists. They have got all the required stuff to resume work at any time.
In other words: the project has revived at last. I will be fully focusing on this from now on. I hope to bring you even more good news next month with new screenshots at last.
As always, thank you for you support, take care!