This week I've been living inside the last few playtest writeups, trying to tease them apart and figure out how to handle them. So I thought it'd be fun to continue that into this month's design notes, and talk about what's in a playtest report and how I handle it.
What is this "Playtest"?
First, set the stage. The hardest part of making anything is getting reliable feedback. In a perfect world, every released piece of homebrew would get 100 excited players with welcoming DMs, and within a month you'd get 100 full session writeups, complete with general impressions and specific issue lists. I say often that feedback is worth its weight in gold; that is not an exaggeration. (Actually, I have just checked, and you can get a sheet of gold leaf for about $6; printing out a spreadsheet of feedback would weigh about the same, and I'm paying significantly more than $6 for it!).
Free, enthusiasm-supplied feedback is amazing, and I would develop entirely off of it if I could. But I can't; there's just not enough. So I supplement with paid playtests. For Spellbound Sea, I brought on five coordinators (basically hired on the core of MCDM's regular playtesting crew, though I didn't set out to do that from the first). The structure of all these deals is similar: the playtest coordinator gets a pretty good hourly rate, the testers get free copies of the eventual product, and I get a spreadsheet full of line-by-line feedback along with some general notes. The rate can vary by country, and the feedback can vary in format some, but because these coordinators all come from the same circles, they're decently standardized. Right now I'm only working with one of the coordinators; I haven't decided how far I'll scale back up as I get into full production.
The Report
The report itself is just a spreadsheet. The recent abomination test took place at level 9, with three encounters (2 combats) and the party composed of:
Human Abomination (Beast)
Awakened Tree Fighter (Freeknight)
Goliath Warlock (Beastlord, Pact of the Cannon)
Human Ranger (Deathhunter)
Aasimar Necromancer (Resurrectionist)
A lot of testers will keep in a little WotC content as a control/leveling tool; this coordinator doesn't bother with that, and I'm fine, because we've got a pretty good sense for how this stuff goes relative to baseline.
The Awakened Tree
Again, there are a lot of different formats for playtest feedback. With this coordinator, I'm getting individual items, line by line. For example,
That's pretty tough to read, but it's a summary on the Awakened Tree; the tester thought it worked pretty well overall and was a lot of fun. A couple lines down they noted that the Fear of Fire drawback seemed good, very fair and not as punishing as fire vulnerability would be. But even in a dungeon populated with fire giants, it managed not to trigger. They did detect an exploit with it, though - the Fire-Hardened feat grants you temps when you take fire damage, but doesn't have any limits. So if an enemy multiattacks you with a fire attack (e.g. a flaming weapon), you can potentially be generating a whole lot of temps, very fast. So that's getting a 1/turn limit.
The Freeknight
The same tester was playing a Freeknight and was pleasantly surprised. Their overall impressions said they worried it would be a lot to track, but the overhead was not an issue, and they found the core reacting-and-marking action a lot of fun. Their negative notes included a distaste for the GM being in charge of defining "regions" for the Freeknight's utility feature; both the coordinator and I agree that's not a big deal.
For negatives, let me throw in another screenshots.
Again that's very small: the gist is they're not sure if the language around getting a second reaction is quite correct. I think they're probably right, though I haven't seen someone get confused about the intent yet. I should probably rephrase that language — checking how I wrote it for the Duelist Swashbuckler, there I just used "once per round, you can use [ability] without consuming your reaction." So that's probably a good change to make.
The Abomination
Ok, there's a lot more like that and I won't go over every single item; this sheet wound up with 109 items of feedback! But I will jump ahead to the headline and the most important thing that got tested: the Abomination class. Broadly, the tester...didn't love it.
As a whole the player liked somethings about the class, and felt a lot of others weren't fulfilling the fantasy. Overall they felt they were playing two halves of two different classes. They felt there wasn't a good balance between able to do thing while not transformed and while transformed and found the class was lacking in non-combat abilities. They found the class wasn't keeping up with other classes at this level, and didn't feel it was fulfilling the fantasy it was presenting well, currently.
Oof. That's great feedback, but not exactly pleasant to hear. Digging through the feedback line-by-line, though, showed up several things that went wrong in the test.
The player didn't actually bother to use Sinister Influence very much, instead generating ~all their frenzy through Sacrificial Frenzy. The presence of a Resurrectionist in the party may have been part of the issue here, as that subclass can be a very efficient healer, making Sacrifical Frenzy's downside (burning your own hp) less significant.
The player didn't use Channeled Curse at all. Something clearly went wrong here: the player didn't think it was a good option to grant their teammates attacks, when I know from both math and play experience that it's numerically powerful to do so. After all, an abomination can attack themselves with a spear or something for 8.5 (1d6+5) damage; or they can grant an attack to their checks note Awakened Tree Fighter with a greatsword or something for 14.5 (2d6+5+1d4) damage. (I don't actually know what weapon that tree had).
The player discovered a nasty loop where they were reverting and retransforming every single round to keep getting max temporary hit points.
Because they were generating so much frenzy by using Sacrificial Frenzy at maximum constantly, they judged all the rest of their features oddly; a bunch of stuff relies on frenzy economy to function properly.
There's a bunch of other feedback in there too: 48 separate line items, a mix of bugs, good things, and other observations. But what we have here is a tester who had a bad experience because the class wasn't functioning the way it's intended to, and it needs some combination of design and communication changes to get the class played "correctly". I don't want this to sound like I'm saying it's the tester's fault; it's my fault. But some of this stuff is loopholes in my design, and some of it is that the design didn't really get exercised properly because the player didn't perceive what I intended to get across.
So what's the fix? As you can see by the Abomination post earlier today, I'm not making a major change right away. But I'm going to try some small patches to fix those holes and nudge the player in a better direction.
Limiting Sacrificial Frenzy so you can't rely just on it for frenzy generation.
Rereading this feedback, I think my original intent also was that its damage would bypass temporary hit points; that's not clear in the doc, so I'll need to go fix that too.
Buffing Channeled Curse to make it more obvious that it's an effective tool.
As a designer, having a player write a feature off because "it doesn't sound good" is hugely frustrating. Maybe the feature is bad, or maybe the feature is good and I just didn't present it well.
Breaking that revert-transform loop.
In v0.8 I put a 1-minute cooldown on your transformation; that's a sledgehammer here and one I think I will likely take back out. I think the proper fix is in how Sacrificial Frenzy works. It should still be possible, and fun, to get knocked out of your transformation and get back into it to finish the fight.
Fixing the frenzy economy. Hopefully the Sacrificial Frenzy change above helps (now you can't get all the way to your limit just by using it). But this one really needs more playtest data to suss out properly. Fortunately, I'll have an abomination starting in one of my own regular games next week, so I can do some closer observation.
What's it All Mean?
I dunno, man. Maybe if this month's design-notes isn't a nice and cohesive essay like some of them are, that's because playtest feedback isn't nice and cohesive. It's messy, with players' and DMs' perceptions getting skewed by their experiences and preferences, with tests going haywire to to bad rules interpretations or misunderstanding a feature or just terrible dice. Especially with tough feedback, the important thing is to sit with it for a little bit, let it ferment, and then say "how can I make this player's experience better?" Sometimes you can't — this playtest feedback was good and actionable and gives me a lot to chew on, but I've gotten stuff like "we totally misread this feature and so we thought the subclass didn't work, lol sorry". But all you can do is go out, get the feedback, and then do what you can to address it.