So the shader compiler is moving forward. A lot of this work isn't new, but migrating from MojoShader, where one day I decided, hey, I'll just knock out an HLSL compiler, no biggie.
First step is migrating the preprocessor and lexer, which are pretty much what you'd expect for C-style languages, so beyond SDLifying (de-Ryanifying?) them, they translated over pretty cleanly.
I even have a small suite of unit tests for it! So obviously it's going to work great!
...but it's never long before reality harshes my goddamn mellow...
First thing I realized: Microsoft HLSL's preprocessor is so darn weird, and I had taken several steps to imitate it. Removing those ifdefs just made a bunch of other things fail, though, since parsing source code is always a fragile effort. Fixing one obvious issue tends to call three more less-obvious ones into existence, but with some perseverance, I got through it all and everything was right in the world.
And then I thought, you know, since we want to parse shaders at runtime, I should run this through some fuzzing tools, which was where my troubles began.
Do you know about American Fuzzy Lop? If you write software in C, you should.
What AFL does is fuzz software. You write a simple little program that takes an input and passes it to the code to test. If you were writing code that implements a PNG loader, you might write a couple of lines of code that loads a file into memory, calls into your PNG loader code, and exits immediately after. You build everything with AFL's compiler wrapper (which instruments the compiled code), give it a few tiny, valid PNG files to use as a basis for generating test cases, and then let it just absolutely cook your CPUs for 24 hours or so. It'll try to find variations on those test cases that crash your program, make it allocate a ton of unexpected memory, or hang in an infinite loop.
What you'll invariably find is that your extremely robust code, written defensively to distrust all inputs, still has dozens of crash bugs when genuinely bogus data is handed to it.
As an example, here's what AFL turned my simple unit tests into that triggered an assertion in the preprocessor:
You know, a perfectly common piece of C code that a preprocessor expects to chew on.
And the tests got wilder as AFL kept altering the inputs looking for variations that snuck through our defenses.
The theory here is that corrupt data (or, more importantly, carefully-crafted malicious data) can cause all sorts of real-life problems. What if you shipped a multiplayer game that let people supply their own PNG files for their avatar and they could crash everyone that's playing by uploading a bogus file?
Worse, what if they could execute arbitrary code with data that overflows a buffer, and own a dozen players' machines at once? Fuzzing your code up front is the first and best defense against this danger. And since we want to let shader code compile at runtime, it's not inconceivable that a multiplayer game could want to let people supply a mod or character personalization that has its own shaders, and thus has the same risks as our hypothetical PNG loader.
Many many commits later, the preprocessor is actually robust, or at least not finding any more issues with AFL! Flexing my muscles, I enabled AddressSanitizer and immediately found I botched a simple string alteration in two separate ways. I sat there for a while, staring at the horizon and wondering if all these companies racing to ship the first self-driving cars are writing in C.
So now, I will definitely be fuzzing (and sanitizing, and Valgrinding...) the shader parser from the start as I continue to build.
You can see the ongoing work here. More to tell soon!