How to Use Feedback

Jay is fresh from Unpub 3 where he took a few of his new prototypes to test with a flood of designers and gamers alike. He's also been participating in the PPP program. Therefore, Jay's input on gathering and using testing feedback is useful and should be read!

Guest Column by: Jay Treat

You’re more likely to be struck by lightning twice than you are to design a flawless game without any playtesting. Yet, figuring out how to improve your game after a playtesting session is often a mystery. How can you turn your playtesters' nebulous comments, insane ideas, and bad advice into useful feedback? I’ll share a few tricks with you and I hope you’ll share yours in the comments.

Ask the Right Questions

“Yeah, that was pretty good.”

The first piece of useless feedback every designer hears is the patronizing reassurance that your game isn’t 100% crap. You ask, “What did you think?” or “How did you like it?” and 99/100 people will answer the same way. What they’re really saying is “It wasn’t good enough. I can imagine a worse way to spend as much time, but it involves fire and shackles. I can’t tell you exactly how bad it was because I was taught if I can’t say something nice, don’t say anything at all.”

While you will get more pointed feedback from your peers in the industry who know what you need to hear, and you must seek that level of feedback out at some point, you need the perspective of real players first. If you want more useful responses, you have to ask better questions. Crap in, crap out.

Be as specific as you can. If you’re aware of any points that might be less than perfect, call them out and ask for opinions. “What did you think of mechanic X? Was it fun? Balanced? Confusing? In-Theme?” Once players realize you know there might be a problem, they’re no longer the messenger and are more willing to discuss it.

There are other staple questions that can lead to more useful information. “Was the game too long?” “How much would you pay for this?” “Who do you think would enjoy this?” There’s a subtle difference between these questions and the following: “Was anything confusing?” “Would you buy this?” “Was the game too hard?” Take a moment and think about why the first three might be good and the last three might not.

“Was the game too long?” Unlike “Did you like the pacing?”, this question acknowledges that the game might be too long. Answering “a bit” doesn’t make the respondent a villain, it lets them confirm your line of reasoning. More on what that answer means soon.

“Would you buy this?” is a binary and personal decision. The player only has two answers to give; one of which will be a lie and the other will hurt your feelings.

“How much would you pay for this?” gives them a scale to grade on, the upper and lower bounds of which are not clearly defined, so whatever they answer can’t be interpreted as an insult. Even though ‘you’ is the same word, this question feels more like an abstract question where the ‘you’ is more of a general ‘one.’ I might not buy your game for $40, but I know people who would. Note that this question is less to help you set a price point and more as a secret gauge for how good your game is.

“Was anything confusing?” is an attack. Your player hears “Are you too stupid to play this game?” They will always answer ‘no,’ unless they made a bad play during the game and would rather pawn off responsibility for it to the game. A better question is “Where can I improve rules or game text to be more clear?” Or “Can anyone word this better?” Or “Did scoring seem cumbersome?”

“Was the game too hard?” Also an attack:. “Was the game too hard for you?” Instead ask, “What felt imbalanced?” or “The game is meant to be a challenge. Do you think I should increase or decrease [some element]?”

“Who do you think would enjoy this?” allows the player to give you a positive response without committing themselves to a lie about how much they hate your game. They might say “hardcore gamers” which tells you the game is too complicated or “kids” which tells you it’s too simple or luck-based. If they say “my play group back home” you can do a little dance inside.

Under the Veil

Very few of the comments your testers give you will be directly applicable in tweaking your design, but all of it is useful data. Your job is to interpret what they’re saying (the hard part) and decide how to use that information (the other hard part).

“The game was too long.”

Your player(s) weren’t engaged the whole time. While we all prefer short games, that’s a learned defense mechanism against bad games. No one who enjoys Cards Against Humanity minds playing for two hours straight or D&D for four or more hours. A game is only too long when it’s boring.

The more players there are in your turn-based game, the longer a player has to wait before her next turn comes around again. If the game isn’t highly interactive, all of that time is boring. Try to make turns shorter by giving players less to do and preventing analysis-paralysis by limiting their options. Try to keep players engaged when it’s not their turn by giving them stuff to do out of turn, or making sure they care desperately how the current player takes his turn.

If a player feels overwhelmed or doesn’t get the game, they will tune out. “I don’t know how to improve my score/position because I don’t understand everything” leads almost instantly to “I don’t care. Please let me stop.” You probably need to simplify your game. Consider making a basic/introductory game or improving your initial presentation, but probably, you need to simplify your game.

If a player feels they can no longer win the game, or that there’s so much luck that their choices don’t matter, they will disengage and want to escape. Make sure that you have an appropriate amount of luck in your game for the length and gravity of the game (they’re inversely proportional) and that the amount of work a player has to do to take their turn (analyze the situation, make a decision, move pieces around) doesn’t exceed the impact of their choice on the game. It’s fine if your game is mostly luck, provided it’s short, light-hearted and easy.

“I didn’t like this mechanic.”

Many comments mean very different things depending on who said them and the context. You have to be reading your players throughout the game so that you will know how they mean what they say afterward. Honestly, you shouldn’t even need to ask “did you like the game?” If your players were laughing and smiling, they definitely did* and if they were checking their watches or nodding off, they definitely didn’t.

*Good company can make a game the same way bad company can ruin a game, but you can tell how much of the fun was just friends joking with each other as if they weren’t playing at all.

If a player lost the game because he misplayed or because another player leveraged a mechanic against them well, “I didn’t like this mechanic” often just means “I didn’t get this mechanic” or “I lost to this mechanic and I don’t like losing.” If they play again (good luck with that) and have the same opinion, consider upgrading them from ‘sore loser’ to ‘onto something.’

If a casual player hates your mechanic, it’s probably too novel/complicated for him. Reconsider your audience, or simplify your game.

If a hardcore player hates your mechanic, it’s probably too old/shallow for her. Reconsider your audience, or look for a unique twist to cast a new light on this classic mechanic. If you weren’t aware the mechanic was old, ask for at least two games that use it and go play them. Also, play more games. Musicians don’t not listen to music. Boxers don’t not watch boxing.

If an industry peer hates your mechanic, ask them why. They could fall into any of the above categories, but if not, they probably have a really good reason and whether you agree or not, you need to understand their reason.

“There’s too much luck.”

Funny how you almost never hear the opposite. The reason is that in a game with luck, players can blame their loss on something other than themselves; in a game with no luck, your only recourse is to blame the game balance or complexity; it is a rare player that acknowledges their own mistakes.

That said, there’s probably too much luck. We game designers love our dice and cards and rarely do the math to see exactly how much impact they have on the game. It’s okay, that’s what playtesting is for. Just don’t be afraid to make some big tweaks (always start with big tweaks and iterate your way down to small tweaks—it’ll save you time, guaranteed). Editor's Note: Famous video game designer Sid Meier of Civilization fame has a classic rule for tuning: double it, or cut it in half. I've been using this for years and it has served me well.

Games need variance in order to create suspense, so don’t remove all the luck. Just make sure players can’t invest more work into the game than you can guarantee they will be rewarded for. Again, shorter and easier games can have more luck, but longer and harder games can’t. Think of a bell-curve. If your game is 5 minutes, it’s okay if the more skilled player only wins 60% of the time, but if your game is 5 hours, it’s not okay if the player who played best doesn’t win at least 95% of the time.

Note that luck and skill are not opposite ends of a single axis. There are games with low luck and low skill (Tic Tac Toe) and games with high luck and high skill (Poker). One does not preclude the other. There are also vastly different sorts of variance. That’s a whole other article, but consider for one example the difference between Candy Land and To Court the King. Candy Land’s dice decide who wins the game, while To Court the King’s dice usually just push toward one strategy or another.

“Have you considered X?”

Players will suggest ways to change your game or things to add. Especially industry peers. Some of these suggestions will be so dumb, you have to lock off the part of your brain that heard them so it doesn’t infect what’s left. Some of these will be so brilliant, they turn your game from “ehhh” to “yeahhh.” You have to listen to these suggestions and even harder, you have to consider them.

Suggestions like these are to be treated like a brainstorm. No matter how bad they sounds, you must not criticize or dismiss them, or it will be the last suggestion you get from that playtester. You can either write them down and promise to think about them later (which you should actually do) or talk them through on the spot. Make sure you understand both the core concept of the proposal as well as the reason the suggester thinks it might improve the game. Often, his solution will be unusable, but the problem he’s trying to solve will be a very real problem that you need to address.

This actually tends to be the easiest way to tease out a problem from your testers. “What would you change/add/remove?” will often lead to terrible ideas that pinpoint with laser accuracy a deficiency in your game. For example, “I would make separate cards for cowboys’ horses” :: “You haven’t integrated the Western theme enough.” And “eliminate the bidding step” :: “the bidding is unoriginal and doesn’t really impact the game.”

Remember that no one understands your design or your vision as well as you do. Just because Richard Garfield would add powers to your character cards, doesn’t mean you should. But if you don’t understand exactly why he suggested that (the game’s too simple) and exactly why you shouldn’t (your audience is the elderly), then you need to find out.

Is there another comment you’ve gotten you’re not sure how to interpret? Let’s discuss it in the comments.

What Next

You’ve got your feedback. You think you know what it means. How should you change your game? Try all of it—one at a time. So many of man’s greatest achievements have come from accidents or from crazy ideas that someone wouldn’t give up on. Don’t let your instinct throw away an idea because it doesn’t sound helpful. If you don’t know exactly how that would work out in every possible situation, you owe it to yourself to find out. Most of what you try won’t improve the game, but everything will improve your understanding of the game.

Design is an exploration. You are entering an unknown reality where you don’t even know the rules of physics. You can only discover the boundaries by pushing ahead in a direction until you hit a wall. The more you push, the better intuition you have of what’s possible and what’s not, but if you stop pushing when you find something good, you’ll never know if you missed a secret passage to something vastly better.

Design with the intention of failing, or you’ll never have the perspective at the end to know if you’ve truly succeeded or merely stopped designing.**

That said, don’t try everything at once. If you try three changes at the same time, and the game improves, how do you know if all three helped or if one helped a lot and the other two held it back. If the game worsens, how do you know all three changes were bad? More importantly, how do you understand how each change impacts your game and what that means about your game? Editor's Note: The scientific method will treat you well.

You don’t have to prototype every change. Most games and most changes have to playtested with other humans to see their true impact, but sometimes you can do that without making new playing pieces. Just tell the players X is Y this time. Some changes you can eliminate socratically: What if players could pay $10 to the bank to roll an extra die in Monopoly? They wouldn’t bother in the early game, and would know exactly when to use it in the late game. Does the mid-game matter? Do you want players paying marginal amounts to the bank to completely bypass their opponents’ hotel chains? You have to be really careful with this type of thought because it’s ridiculously easy to miss important details and still be 100% certain your right. If you are going to skip this test, at least run it by another player to see if they agree with your conclusion.

**How do you know when you’re done? If you blind playtest the game with at least three new groups of your target audience and they all love it, and you can’t find any way to make the game better (without making it worse), you’re probably done.

Multiple Paths

You’ll often reach a fork in the road. Perhaps your game is half-way between a party game for gamers and a fun party activity for non-gamers. You have to go one way or the other because the split won’t please anyone, but how do you choose? This is the other other hard part.

Sometimes, you have to go back to your original vision for the game. If you set out to make the fastest, lightest tactical wargame ever and you have the chance to instead make a meatier game that really stands on its own, maybe you should stick with your vision and complete your original goal. Put your thumb in the page of this choose-your-own-adventure, because you can always go back and explore the other path when you finish this one.

Sometimes, I’m thinking most of the time, you have to forget where you were coming from and listen to what the game wants to be. Like carving stone, of course you have to start with a plan in mind but you might get half way there and just see something else waiting to be revealed. Something better. It’s rare that the act of executing one game idea doesn’t lead me to discover something different, something a little more natural and unique. Unlike carving stone, you can always undo a change and go back to what worked better.

Ultimately the decision is yours and yours alone to make, but remember that you can always ask friends and peers for their opinion. The value of another perspective is immeasurable.

Calm Down

As a parting note, I wanted to touch briefly on designer ego. It’s impossible not to be invested in your game. It’s your baby and if you aren’t invested in it, it’s going to be trash and you’re wasting the playtesters’ time. What you have to learn to do, though, is to divorce the quality of your game in its current state from your self-confidence as a designer. A bad playtest doesn’t make you a bad designer. It doesn’t even make the game bad, because your game isn’t finished. It just means you’ve learned something and found a way to make your game better.

The only way to be a bad designer is to release only bad games, and the only way to do that is to ignore your playtesters when they tell you what’s wrong with your game. Or to not playtest.

Comments

Good stuff Jay! BTW been meeting to thank you for the 2nd email on HN.

I've been thinking about the awkward moment at the end of the test and what to say, and I think I have decided on a new approach. The first thing I am going to say is "What did you absolutely HATE about the game?"

I think this will do a few things. First it will take them off guard a little bit and they may be caught speechless, but it also reflects me totally lowering my guard and saying (without saying) "any comments you want to make are fair game."

This also forces me to take my ego out of it and hear what their biggest problem is which I can then drill into the "why" it matters to them.

The natural inclination of most will probably be to point out something positive after hitting you with their biggest complaint. And that is fine, it is always good to get validation of what DID work for them.

Thanks for the strong article.

Thanks, AJ. I've been getting a lot of positive feedback about—waaaiiitaminute! Was the article too long? It was too long, wasn't it? Why won't you people admit my article was too looong?!

I've tried the "What did you hate" angle once or twice. I'm not sure it works. It's better than "Did you like it" because it shows you expect there to be problems, but still makes the first criticism a bit of a neck-out-sticking situation. I would follow up with "Was it too long / too fiddly / too luck-based / too easy" depending on the game; something concrete for them to latch onto.

Candy Land has dice?

Regardless, thank you for this very insightful breakdown. I will definitely be taking many of your points to heart, and incorporating them during my future play-testing sessions!

Crap, Candy Land uses a spinner or colored cards, doesn't it? Sigh. At least the points stands.

BANNED FROM WRITING JAY

Yes, Candy Land uses colored cards, and, yes, your point--which is a very good one--still stands.

Again, this is a great article and I look forward to reading more down the road!

Great article, Jay. This is the sort of information that every game designer needs to get the most out of those precious playtesting sessions. I'm definitely going to employ your advice my next time around, and make sure I ask questions that are concrete, admit there can be weaknesses, and give players space to offer opinions without putting themselves out there too much.

What terrific advice, especially for new designers like us. What I would give for an upPub con here in Denver -*hint, hint*!
We are considering starting a Game Board Developers group around here just so we can have and offer support in our games. Finding people to play test isn't easy when the game is in early stages!

Excellent article; thank you so much for taking the time to share what you've found to work! I'll definitely keep your suggestions in mind as I go through my own playtesting.

"While you will get more pointed feedback from your peers in the industry who know what you need to hear, and you must seek that level of feedback out at some point, you need the perspective of real players first."

Hmm, I'd have to disagree with the notion of getting "real players" to test your game before your game designing/industry peers. Indeed, I take the exact opposite approach. Feedback from designers and experienced play testers is far better for shaping a game, plus they're more open to changing a game mid-stream or ending it when you've learned enough for that particular test run. Admittedly, finding or training such play testers is not easy.