I'm revisiting the forest project as I plan to use it for a public Raytraced Audio demo... but I got carried away with graphics!
This post is an overview of what's changed and improved. There's so much more to write about - I'll share more detail in follow-up posts for water reflections, shadows and more.
All benchmarks are at 1080p on my RTX 3070 GPU.
Water
Water now has reflections, distortion, caustics and displacement maps!
I initially tried screen-space reflections, where the above-water pixels are reflected in the water. This is the cheapest approach, but breaks when looking down into the water as there are no above-water pixels on screen to reflect.
I looked at how games like Valheim render water reflections, and discovered they render the entire scene twice - once for what you see, and once mirrored from underneath the water.
This is called Planar Reflection, which renders the scene from a second camera that's placed under the water and looking upwards. Rendering the entire scene again is expensive, so to speed it up I reduced it to half resolution and disabled shadows.
This scene renders in 0.5ms to 1.0ms depending on the angle you're looking at the water (more on frustum culling later in this post)
The reflection is distorted based on a normal map that moves across the surface, as well as a displacement map. Distortion and caustics are also applied to underwater geometry:
I also wanted water to react to the player and other objects moving through the water. I implemented this using a compute shader that runs on a 2D displacement texture, where each pixel represents the height of the water and the up/down velocity of it.
The compute shader simulates wave physics, where tall water pixels are pushed down and low water pixels are lifted up.
When an object lands in the water, it writes a low pixel into this displacement texture with downward velocity. The compute shader then spreads this 'energy' out over time, causing ripples to form.
The texture is 1024x1024 pixels, and updates in 0.07ms.
The angle you view the water at also determines what you see:
glancing angles have strong reflections
steeper angles let you see through the water
Buoyancy
Water should also be functional, with objects floating on its surface. I'm using BepuPhysics for physics, which unfortunately doesn't support buoyancy by default.
To simulate buoyancy, each frame I apply an upward force to the corner of each object that counteracts gravity. On the surface the force is light, and increases dependeing on how far underwater the object is.
Wood pieces have mass depending on their size, meaning you can stand on larger pieces of wood. It's a bit slippery at the moment, I need to tweak the friction:
Lighting
Lighting uses the same ambient/diffuse/specular system, but I've changed the 16 bit RGB framebuffer to use floats instead of unsigned shorts. This means brighter colours can be stored with values higher than 1.0, and tonemapping can be added.
I'm using ACES tonemapping, which adjusts the hue/saturation of light/dark areas to give it a more filmic/cinematic look.
I've also changed the sun colour to be off-white during the day and orange at sunset, which produces these nice warm scenes.
Paired with the skybox it produces some beautiful colours.
Fog also uses the same sun colour, which helps differentiate between objects before the sun rises above the horizon.
New Trees
I'm using TreeIt to generate trees and it is much easier than the Sapling plugin and Geometry Nodes in Blender.
I created this bendy water tree by adjusting the 'curviness' of the tree trunk and increasing leaf gravity.
Some of the default trees are excellent too, like this Beech tree:
To use this tree in game, I exported it as an .obj file, loaded it into Blender, then separated the leaves into a separate object (leaves are rendered separately as they use alpha-cutout textures).
I also created a separate low-quality mesh that's used for physics interactions, as well as a 'shadow leaf' model that has 75% of its leaves removed - it's 4x faster to render and the shadows look nearly identical.
Wind
Previously I implemented wind as a simple horizontal movement based on the height of the tree - the base stays stationary, and the top sways horizontally. However in strong winds they became skewed, since they only moved left/right rather than actually bending.
The solution was to split the tree into 10 segments vertically, then incrementally rotate each segment further than the last.
This all runs in the vertex shader, and suprisingly had no effect on performance. The vertex shader mustn't be the bottleneck here.
The trees bend nicely now, but they're still moving as one solid object. In reality the leaves should shake around as well.
I used the leaves' UV coordinates to implement this, since every leaf texture is identical with the base of the branch at (0, 0) and the tip of the leaf at (1, 1).
This means the shake strength can be controlled by the UV length - the base will stay stationary at (0, 0) and the rest of the leaf will move.
Leaf Performance
Leaves and grass take the longest to render as they require alpha-cutout textures.
Leaves take 0.6ms to render and grass takes 0.7ms to render, which is 1.3ms out of the 1.7ms it takes to render the geometry.
Without alpha-cutout, they would look like this:
This is slow to render because it can't take advantage of Early Depth testing. Normally, before running the fragment shader, the GPU compares each pixel's depth against the depth buffer to determine if it's occluded. If it is, the fragment shader is skipped entirely. However, because the shader uses alpha-cutout textures, the GPU can no longer do this test early — it has to run the fragment shader first to find out whether the pixel will be discarded, and then perform the depth test.
Since we're using forward rendering and doing expensive calculations on each pixel - and since leaves overlap a lot - this means we're spending time shading pixels that will likely not even be visible.
The solution is to do a depth-only pass, where we render the leaves and only write to the depth buffer:
Then we render the leaves again, but only shade pixels that exactly match what's in the depth buffer. This means leaves that are in the distance are culled before the fragment shader runs, so we're only running the fragment shader once per visible pixel.
Shadows
After working on 3D rendering for 8 years, I think I finally understand shadows and have implemented them correctly.
The demo now has smooth, performant shadows using 3 cascades / shadow maps. This means both close-up and distant shadows look nice without glitches or artifacts.
Shadows take 2.5ms to render at sunrise (ouch), and 0.4ms to render at midday. 75% of this is rendering leaves - I need to implement LOD for trees in the distance.
The basic approach is:
Figure out the max bounds of the world you're rendering (black box below)
Calculate your view frustum (yellow lines below)
Calculate the intersection between the view frustum and the world bounds, then split it 3 times (red, green, blue) to produce 3 cascades
Based on the sun direction, position the shadow camera precisely for each cascade, so we only render the cascade with minimal waste (0:10 in the video below)
Render the scene's depth from the perspective of each shadow camera, and save each to a texture
Finally when rendering the scene, use these shadow map textures to figure out if each pixel is in shadow
I'll write another post about this because I've been confused by shadows for a while now, and think it'll be a beneficial post for everyone. Below is an animation that may help explain this:
Rain
Rain particles are updated using a compute shader, which iterates over the buffer and moves each particle based on gravity and wind, and respawns them when they fall below the map
Currently there's no collision - rain particles fall through buildings and trees. I'd like to see if I can sample the shadow depth map in the compute shader to figure out when a rain particle hits an object.
Rain particles also write to the water displacement map, causing the surface to shimmer:
Performance
My performance target is 144 FPS at 1080p with my RTX 3070 graphics card. This new forest scene renders at 180 FPS at sunrise (long shadows are slow to render) and 280 FPS at midday.
This is much faster than the previous forest demo, which ran at 160 FPS with a single shadow cascade and no water reflections.
The main performance improvements were:
less leaves / smaller trees
compute frustum culling
layered rendering
Trees
The old forest demo had huge trees and leaves practically filled the screen. Since the leaf shader uses alpha-cutout textures, the GPU can't render them as quickly as solid geometry. The only workaround to this that I've found is to render less trees.
The shadow pass is also affected by alpha-cutout textures, so I'm also using a separate 'shadow leaf' model that has 4x less leaves. This is 3-4x faster to run, and the shadows look nearly identical.
Compute Frustum Culling
Each frame the world geometry is rendered multiple times:
once in the main view (what you see)
once in the water reflection
three times in the shadow cascades
six times for each point light (torches)
If we have 4 point lights, this means the geometry is being processed 1 + 1 + 3 + 4 x 6 = 29 times each frame.
Frustum culling the entire world on the CPU isn't that slow, but doing it 29 times takes a hit. So I removed frustum culling from the CPU and instead enqueue every object in the scene for rendering.
Then I use a compute shader to analyse:
the indirect draw buffer, which holds the number of each object (tree, rock, etc) to render
the instance buffer, which holds the position (XYZ) and scale (W) of every object
Then it compares each object against the 29 view frustums. If it passes, it writes its info to a new draw command and instance buffer for that frustum.
The output of this compute shader is 29 draw buffers and 29 instance buffers, which can be used to render each stage (main, water reflection, shadow, etc). The compute shader is essentially splitting the scene into 29 different render calls, one for each stage.
Layered Rendering
To render shadows faster I'm using the gl_Layer shader variable, which was designed for rendering the same geometry twice for VR (once for each eye) to two separate textures.
We can use this to render multiple shadow cascades with one draw call.
The shadow textures are stored as layers in a GL_TEXTURE_2D_ARRAY attached to a framebuffer, and the vertex shader sets gl_Layer to 0, 1, or 2 based on the current instance draw index.
For this to work we need to modify the compute shader to store the frustum-culled commands in one large draw buffer and one instance buffer, instead of 29 separate buffers. Then we can render all 3 cascades with one draw call, since all of their commands live in the one buffer.
This is a quick overview of how layered rendering works - I'll write it in more detail in a follow-up post.
Raytraced Audio
This forest demo will be available publicly as a demo, where you can experience Raytraced Audio in a non-voxel environment. You can explore the forest and build structures to test out occlusion and reverb.
The Raytraced Audio SDK has also received some major upgrades, including:
multiple listeners (for split-screen gaming, or server-side raytracing)
casting rays between sounds (e.g. to determine if two NPCs can hear each other)
two new ray types:
Audio Visualisation rays, for visualising audio for deaf players
Ambient Permeation rays, so rain is louder inside a thin metal shack vs a concrete bunker
different bounce counts for each feature - previously all occlusion/reverb/ambience rays were limited to the same bounce count, and permeation rays only bounced once
memory and CPU optimisation for even faster raytracing
API changes for easier integration and less boilerplate code
That's all for this post, thanks for reading!
High quality images and videos for this post are available here.