Rating Messages with S.T.A.R.S.
Hey, hola, it's me. Don't you find that when you start doing one thing, another thing immediately needs to be done? To the point that there are so many things that you change priorities and in the end, you don't get anything done? Well, not absolutely nothing, but you leave things incomplete? Welcome to my world.
Anyway, the post isn't about that... or is it?
Well, I've just implemented (not entirely) a star system. Something fairly well-known, although sometimes we don't know what they mean. Because I like giving 5 stars, even though the food was over-salted, and the crab from the other customer tried to use me as a hostage to escape the restaurant.
Let's review what the stars in each written AI message mean.
☆ This is an empty star, meaning it is not present. And ★ means it is present. Wait, that's very obvious... or maybe not?
By clicking on the star, you can toggle the rating on or off.
The position of the star you select indicates the number of stars you want to rate the message with.
One star is equivalent to Broken / Awful. This indicates that the message is terrible, poorly written, out of context, out of character, poorly formatted, has grammatical errors, etc.
Two stars are equivalent to Weak. This indicates low quality in terms of content; for example, it hasn't written actions when it should have, or hasn't written character dialogue when there should be a verbal exchange. Or it has stopped doing anything, or it has "trolled" you with actions to avoid writing. In short, a vague response.
Three stars are equivalent to Descent. This refers to normal messages that continue the acceptable line of what the AI should respond, respecting the same format, amount of actions, without spelling or grammar errors.
Four stars are equivalent to Great. This is a rating to highlight that you liked the response; not only did it follow the message format, but it also added value and interest to the conversation by responding as the character or narrator would, in an immersive way, or as expected.
Five stars are equivalent to Perfect. Not only did the response hit the nail on the head, but you also liked it so much that it brought a smile or a laugh (or a tear?) to your face. This is when the response perfectly follows the dialogue and action format, sometimes surprising you and generating a lot of interest in continuing the chat.
Now, so far so clear. But I also have to mention when not to rate certain stars.
For example, sometimes a message is so perfect that it hits the nail on the head regarding how the character should respond. However, you had expectations for something else. A villain being kind, a hero swearing. You were looking for something out of the ordinary.
Well, you understand where I'm going. Despite the AI responding in a way we didn't want, it responded in the way the character should, so we should refrain from marking the message based on "the moment" and focus more on the message itself.
Is it mandatory? Oh, no, it's just a way to explain how it works... or should be. Yes, the system is incomplete; remember to refresh the page at least once.
"Wait, is this all an illusion?"
That's right, and I have a lot of work to do on the server. But the functional foundations are already there, and although they do nothing at this moment, they will serve in the future to improve feedback on when the AI is generating poorly.
And I have to repeat this because I know many are worried and fearful: No, I am not going to read the messages rated by users. There are millions! What is done is saving certain values in the messages, like the configuration used in the AI, and then extracting statistics on which were the best and worst according to the stars. And in that way, improving the quality of the messages.
It could go well, or it could go poorly. It depends on everyone (specially me if I nuke the server by mistake).
— Slow steps, but in the right direction.