In basic development, I balance new subclasses by having a whole array of spreadsheets, charting out "optimal" damage numbers for each level. For the game's base classes, I try to graph out all of WotC's subclasses as well as all of my own. This lets me check my math with a pretty simple question: is my new subclass stronger than the weakest thing and still weaker than the strongest thing? If no, then it's back to the drawing board.
This month I produced a bunch of supporter-exclusives, all as prework for Machines of Bone & Blood. And one of the ones that gave me a little grief was the Way of the Rotting Hand, a necromancy-themed monk subclass. As I was writing it, I was comparing to WotC's official Way of Mercy, and decided that it was time to refresh my monk spreadsheet.. And then I realized that the how of that math would make for a good design diary for the month. Abandon hope, all ye who enter here because inside you're gonna find some math. And fair warning, Patreon doesn't give me any reasonable way to write that math out, so it's just gonna look ugly.
Simple Assumptions
In my damage charts, I tend to make the same few simplifying assumptions every time.
Every combat lasts 3 rounds.
This is a pretty standard simplifying assumption; I forget if it comes out of the DMG or somewhere, but ~everybody has used this one forever.
Every day has 4 combats.
This is maybe a little controversial, but I find it works well. The official guideline is 6-8 encounters per day, but that's always been laughably high; it's only mechanically achievable if lots of them are easy, and only narratively achievable during a dungeon crawl. A lot more of the DMs I know do something more like 2-6 encounters, with the combats skewing to the Hard/Deadly ranges.
Every day has 1 short rest.
While the "ideal" adventuring day always includes 2, there are lots of tables where this is uncommon. We'll touch on this a bit more later, though.
For the monk chart in particular, my original version made two more obviously-wrong-but-maybe-not-too-wrong assumptions:
Monks use Flurry of Blows every single round.
This is obviously untrue; Flurry is, after all, only one of at least 4 options for your bonus action. But these are damage charts and are generally aimed at somewhat-but-not-totally optimized builds, so we want to plan for a high-damage playstyle.
Monks have essentially infinite ki.
This is again obviously untrue. But our main priority here isn't necessarily comparing to other classes, it's comparing monk subclasses to each other — I use this chart to write new monk subs. And in my original draft of this chart, trying to model variable ki-consumption rates was a giant pain, so I didn't do it.
Uh-oh Steroids
The simple math above works pretty well for the PHB monk subclasses. Shadow monks don't meaningfully get other damage boosts. Way of the Four Elements monks (mostly) do better by Flurrying than by using their subclass features. Drunken Master monks have their main feature tied to Flurry. Simply put, we can call these subclasses Flurriers. Their "optimal" play pattern (again, in terms of damage per round) is to use Flurry of Blows as much as possible. (We can put Open Hand in a special subcategory of its own, where its Flurry is stronger than a basic one.)
But there's another type of monk subclass, which we can call Non-flurriers. (I know, creative.) These are monks who get a better damage steroid than just spamming out Flurry. They break both the two monk-behavior assumptions above, because they no longer want to use Flurry every round, and their steroid features generally cost ki, which means they have less ki available to spend on Flurry when they want to Flurry. If you look at the early years of 5E, Wizards of the Coast wasn't publishing any monk subclasses in this category; all the damage steroids in early subs are carefully tuned to be just less than flurry, and are meant to add tactical versatility, not damage. They threw this rule out with Tasha's, though, and every subclass since has had a real steroid.
In my supporters channel on Discord, I previewed that Mercy monks were going to blow everything else out of the water; this turned out to be completely wrong. After I rewrote my math away from the very simplistic versions up top, they basically track exactly with every other sub. The standouts are Open Hand (strongest at low levels, though they're all pretty close), Drunken Master (goofy-strong at 17+), and Long Death (which never receives a single worthwhile damage feature).
The Simple Math Doesn't Cut it Anymore
In order to accurately model the non-flurriers, I needed to change the way the spreadsheet worked. So I wrote a couple of fairly simple functions that are able to apply a multiplier to your damage based on how ki-expensive a feature is. They assume you use the feature every time you can until you either run out of ki or you're using it every round. The functions wouldn't work for features you can use more than once per round, but those aren't really a thing. While they'd be great for nova damage, nobody really wants a monk that burns through all their resources for the day in a single round.
Consider the Way of Mercy monk. At level 3, it gets Hand of Harm: once per turn when you hit with an unarmed strike, you can spend 1 ki to deal damage equal to your Martial Arts die + your Wisdom modifier. Hit-confirmed like a smite, but with a 1/turn limit. When you get the feature at level 3, your Dexterity and Wisdom are probably both a +3, so the damage is equal to an unarmed strike (but hit-confirmed). Using Hand of Harm every round deals more damage and for less ki than using Flurry of Blows. (Not by a lot, but it does. And if you want to nova, you use them both.)
But that relationship doesn't stay fixed. For one, you're probably putting your ASIs into Dexterity, not Wisdom, so your attack damage goes up while Hand of Harm lags behind. And for two, you can only use Hand of Harm once each turn; after you're putting all your ki into it, you have leftover ki that's not doing any damage. So you start to mix more Flurry of Blows in.
What this produces in the math is a seesaw pattern. For a non-flurrier subclass, I write the math assuming it prioritizes its steroid until it can't anymore, then it switches to putting extra ki into Flurry of Blows. (Almost all the steroids cost 1 ki, but the functions I wrote can handle variable costs). The spreadsheet can't optimize the seesaw by itself (it's possible to write linear solvers in google docs, but I don't care enough to relearn the math and then learn how to do it in docs). But the seesaw basically always switches over around level 6 or 7. If you're getting 2 short rests in, then it would switch over around level 4.
There is an alternate approach I didn't try, which might actually work out easier: just calculate a damage-value-per-ki. For example, at level 3 you can spend 1 ki to Flurry of Blows, which nets you 1 extra unarmed strike, which is 5.5 (1d4+3) damage. If you apply accuracy math, you get 3.7 damage per ki. In theory, at every level you could calculate every subclass's optimal damage-value-per-ki and just do the math with those everywhere. (I'm not sure it'd save you much, though, you'd still need to figure out the seesaws above.)
Chart Me, Bro
Now, caveat that there are almost certainly errors in there — that's over 500 data points, mostly calculated with pretty gnarly formulas, so I'm sure there are logical bugs and typos. And there are other shortcuts and assumptions, things like "this feature probably doesn't add significant damage", or "the math on this one is much too hard so I'll just add a flat bonus instead that's approximately right". But broadly the chart shows the same distinction we described above: Flurriers vs. non-flurriers, or to put it another way, pre-Tasha's monks and post-Tasha's.
I'll be making a few adjustments based on the chart, though not too much dramatic. One thing worth noting is that of my own monk subs, you can broadly say that Predation, Progress, and Stormdancer follow the pre-Tasha's philosophy more, while the newer subs (Steel, Silver Glimmer, Forge, and Rotting Hand) follow post-Tasha's balancing. I also note that while my subs tend to hit pretty hard, they also tend to be very ki-hungry, which should help constrain it some.
Do as I Say, Not as I Do
Now, if you want to repeat this exercise for yourself, I would suggest doing much simpler charts than these. Crucially, two things you don't usually need to model are to-hit chance and crit chance.
Hit chance is unimportant to model as long as you're comparing things with equivalent hit chances. For example, two fighters wielding different weapons do different damage, but will both see the same percentage drop in damage when you account for their chance to miss. Where you do need to care about hit chance is when you stop comparing apples-to-apples; give one of those fighters advantage and their damage goes way up, even if they're rolling the same size die.
Second, critical hits basically don't matter. A critical hit happens 5% of the time, so when you factor in its added damage, you need to reduce it by 95%. Even with advantage, you still only get to (almost) 10%. At this point, while it's fun gameplay, it's meaningless as part of an average damage analysis.
Now, there are two exceptions here that do force you to model crits. Expanded critical hit ranges (the Champion Fighter or my entire Swashbuckler class) can change that 5% up to 10, 15, 20, or even in one extreme case 25%. As this number goes up, the added damage starts mattering more, especially when advantage applies. (My charts model crits, but mostly because I had to do the math to make the Swashbuckler).
Second, critical hits matter when critical hits are strong. You might think this would be something like a barbarian's Brutal Critical, which scales up their crit dice, eventually as high as 4x damage. This matters, but not a ton; even at level 17, with a greataxe (1d12) and using Reckless Attack every round for advantage, this only works out to a bit more than 1 damage per round extra. If you look at a rogue or a divine-smiting paladin, and then the crits get strong. Our barbarian above deals 19.5 (3d12) extra damage on a crit, while a level 17 rogue does 31.5 (9d6). The rogue's bonus damage gets closer to 1.5-2 damage per round. Still not a huge effect, but getting into the territory where it starts to matter; that works out to a 7% damage increase overall.
For a significantly smaller thing that's really not worth the trouble: hit-confirmed effects are very annoying. A hit-confirmed effect is something you decide to use after you know if your attack hit - Divine Smite is the gold example. There are a few different ways you can build it into a model. The easiest and simplest is to just give it bonus damage. For example, a basic Divine Smite is 13.5 (3d8) damage. But because it's hit-confirmed, there's no miss chance. An ordinary attack you'd wind up haircutting by about 35%, so instead you can increase Divine Smite by an equivalent amount. 13.5/0.65 = 21 damage, give or take. If you want to simplify this rule even more, you can treat hit-confirmed things as being about 150% as strong as they look on paper.
If you'd rather be more exact, you can instead weight the hit-confirmed attack by the odds that any attack hits that turn; after all, the paladin can't smite if they miss all their attacks. This is easy math when you've only got 1 attack per turn. Once Extra Attack kicks in, it requires real probability math, but the easiest kind, probably something you learned in high school. You just have to calculate the chance that any attack hits, which means 1 - (the chance that no attack hits). The chance of no attack hitting is multiplying their miss chances - so for two attacks, it's (1 - [hit chance]) * (1 - [hit chance]). Or in more mathy terms, (1 - [hit chance]) raised to the [number of attacks] power.
In Conclusion
I hate doing monk-math. But sometimes you have to, and sometimes it leads you to interesting places. I recall a college course I took once, where the entire structure of the class was:
Solve some particular series of equations, with an acceptable error included.
Declare the error unacceptable and solve another series of equations to eliminate it, with its own acceptable error included.
Repeat until the semester ends.
I treat these spreadsheets a little bit the same way. They're full of assumptions and have shortcomings I'm aware of, but I live with it and work around it until I can't. Then it's time to spend half a week rebuilding the sheets to be more accurate. (Fortunately, I don't have to do forty hours of calculus this time around).