Testing Well

Post by: Grant Rodiek

I believe testing is the most important aspect of game design. Perhaps it’s because I lack the brilliance of Feld or Don X., so I require testing to improve my ideas. Ah, if only I were brilliant. Nonetheless, I fundamentally believe that an idea is just that until it’s proven with actual players.

Many people don’t test enough. In my past, I haven’t tested enough. However, testing is one of the few things within our control throughout this process.

We cannot control the opinions and tastes of our players. We cannot dictate the whims of consumers and the environment in which people play our game. But, through testing we can ensure we have perfect rules, streamlined mechanics, and focused games for our players. Here are some of the things I’ve learned through testing. Note that some comments pertain to iteration, but I often link testing and iteration as one.

Know your game and your goals for it. Do not implement a tester’s feedback because they gave it. I don’t know the exact quote, but Matt Leacock said that many games meander for years because they lack focus. Don’t meander!  Implement and adhere to suggestions because they are right for your game.

Every tester has a different favorite mechanic (“I love drafting. Have you thought of that!”), favorite way of doing things, or something they hate in games (“Ugh, luck? I hate luck.”). Keep this in mind. If you chase after every tester’s favorite thing, your game will become a hodge podge of mismatched mechanics. Use their input to make your vision better and more fun.

Here’s a favorite tactic of mine: When a tester says “I didn’t like this mechanic,” instead of telling them they are wrong, I explain to them why it exists. My exact phrasing is often “Here’s the reason I did that. Perhaps you can help me improve it?”

For example, in a recent test of Field Marshals I was told the math for calculating the results of a battle was clunky (truth). They suggested something that was indeed simpler, but it did away with the attrition model (based on the Franco-Prussian War, among others) I sought to recreate. I explained my goals and through this we arrived at an implementation that was far simpler and met my goals for the feature.

If you’re curious, here are my guiding goals for Field Marshals:

  • Create a highly accessible war game ( i.e not much more difficult than Memoir ’44)
  • Plays in an hour or less
  • Plays with 2 or more players (many war games are just 2 players)
  • Use a card driven mechanic (instead of dice)
  • Leverage the tactics and weaponry of the 19th century

Know the difference between a goal of yours and a bad feature. This comment is meant to balance the previous one. If you let yourself, you can counter every tester suggestion with “that conflicts with my vision.” Don’t sell yourself on the superiority of your own ideas. Listen to your testers, ask questions to understand their perspective and comments, and evaluate it as openly and reasonably as possible. Use the data, don’t thrash against it.

Do not insult your testers or let them feel insulted. Nobody wants to test for a jerk. Don’t get angry when you receive harsh feedback. Instead, push yourself to make the game much better. People will only tell you they love they game if they fear you won’t accept their criticism. Make sure they feel comfortable and appreciated for telling you what you really need to hear. Trust me, you need to hear it. As a side note, I feel incredibly blessed to be surrounded by an awesome group of testers. I’ve been a professional game developer for 7 years, so I have a fantastic group of about 10 people who are incredibly enthusiastic about games, are well-versed in design, and aren’t afraid to be honest with me. 

Enter your tests with a purpose. Try not to change everything before a new test. Obviously, game design isn’t science, but the closer we can move towards a “control” environment with a few variables, the better our data will be. If you change everything, it’ll be really difficult to know what worked, what didn’t, and more importantly, why for either.

If possible, have a tester read your rules, then explain them to the group. This more closely emulates the real world playing of your game. Most people learn to play a game after one person reads the rules. Unfortunately, you don’t ship with each copy of your game that’s sold. If you cannot find a tester to read the rules beforehand, try to explain the game in the same order of your rules. This will help you get a feel for how to better organize the rules. Plus, when people ask questions like “How do I use that thing you just mentioned,” you’ll get the tip to bring that information forward in the rules.

Test with a wide variety of people, including people you think will love your game, hate your game, and somewhere in between. One of the most interesting tests I’ve held for Field Marshals so far was with two incredibly competitive gamers. That’s not the type of gamer I am, nor is it really my focus when developing games. But, observing how they viewed the game and broke down every element to maximize their chances for victory was fascinating. Had I only tested with casual, lightweight gamers (like me), I would have missed out on this perspectives.

If someone asks a question, take a note of it. Questions and confusion are a great indication that a feature was poorly explained or may need to be streamlined. Before I put Poor Abby on hold, I conducted a test with my new shiny prototype. The tester asked questions on the Argument cards for a solid 5 minutes. He just did not get them! My takeaway wasn’t that he was dense. Instead, I realized that if this veteran, hardcore board gamer couldn’t grasp the mechanic, perhaps it was due for a complete reconsideration.

It’s not you, it’s me.

Take notes on what features/mechanics/player actions are used and which ones are ignored. If a feature is ignored by the majority of your testers, it is either too clunky or doesn’t clearly provide enough value. I’ve found that 9 times out of 10, testers will take a less ideal move using a mechanic that’s more clear. If it’s confusing, they’ll just ignore it. Regarding the value comment, Sid Meier often notes that when balancing a game, either double the value, or cut it in half. If nobody is using a feature or action, you may need to double its value. This tip has been ridiculously useful in refining the cards in Poor Abby and picking which Tactics to list for Field Marshals.

Review your notes with the testers at the end of a test. I always like to quickly read over all the things I observed and changes I intend to make. This serves a few purposes. One, it demonstrates to them that you intend to make changes based on their feedback and didn’t just waste their time. It also gives them a second chance to ponder and think about things you may have been discussing the entire time. You shouldn’t end a test without discussing the mechanics, the flow of the game, and everyone’s opinion. This is a good way to start that conversation.

Blind Test. In case you aren’t familiar with the term, a blind test is when you send your game to someone who isn’t familiar with the game, has to read the rules to learn to play on their own, and ideally, doesn’t even know you. A blind test is as close to a live fire exercise.  I found blind testing on Farmageddon to not only be invaluable for improving my rules, but often, when you have a stranger play your game, they’ll be frank with you. I typically found my blind testers by checking to see who was interested in Farmageddon. People would mark the game as “Want to Play” or “Want in Trade” on BGG. I would send them a message and if they were interested, a copy of the game.

Be willing to kill your favorite features. When you enter a test, do so with the resolution that nothing is sacred. Wait, that’s wrong. What’s sacred is that your game should be fun and that nothing should hinder that. I had this nifty random turn order mechanic for Field Marshals that I kept trying to tweak, streamline, and improve. Unfortunately, after a dozen or so tests it was clear that, while the rest of the game was really improving, the turn order mechanic wasn’t the sharpest tack. I finally threw it away and re-examined how to determine player turn order. Guess what? The new mechanic actually adds depth and strategy to the game. Shocking!

Test Hunches. You want to be careful about fiddling just to fiddle. The majority of the time, you want to make changes based on problems you witness in tests. You want to fix issues, not create new ones. However. While you’re testing, if you have a hunch or think something might improve the game, try it. For example, I was curious whether changing a player’s limit of “play 4 cards per turn” to “play 3 cards” would improve the game. In reality, it made the game significantly worse. Now I know! I also better understand why it made sense to not just revert to 4 cards, but modify a few things and make it 5 cards (with other changes). My bad idea led to the right idea.

What have I missed? Care to add anything?

6 thoughts on “Testing Well

  1. Man, it took me so long to learn all these things on my own. So many game design articles focus on getting your game in front of a publisher, or how to write rules… so very few tell you how to properly test. Exactly ZERO I have read until today talked about humbling oneself to better accept flawed/failed ideas and correct them. Awesome article!

    • I tested Farmageddon a bit before I put it on TGC for sale. I then sent out a bunch of copies to reviewers and sold 150+ copies, so I thought I was pretty awesome. The initial feedback was intensely humbling. It was difficult, but over the next 8 or so months I tested and tested. Eventually, the feedback shifted to positive and enthusiastic, not “I was confused by…” or “I didn’t like this…”

        • Farmageddon was in development for 14 months before Kickstarter. In fact, about 85% of the art was already done as well. The new art, done during the KS campaign, was for the final few cards I added/tweaked over the course of its lifespan.

  2. Be Willing to Kill Your Favorite Features

    So absolutely true, as I have pointed out many times. I design a game by putting in everything I want it to have and that I think is great. Then I slowly take that stuff out until what I am left with is a game that other people think is great.

    Also if something about your game design nags at you, don’t just blindly change it, think about it in depth, till a true solution presents itself. This has taken me months in some cases. Just fiddling with stuff and hoping to find a fix is a waste of your playtesters time and enthusiam. Do that too much and pretty soon you won’t have any playtesters left.

  3. This is an excellent article that is worth reading and then re-reading prior to any playtesting session. To be a great designer, these lessons need to be part of you.

    One of the hardest aspects of playtesting is accepting that not everyone will like your game. It’s impossible to please everyone, and as a result, you often won’t ever be 100% pleased with your own design. It’s easy to go on testing, tweaking, and iterating forever.

    As designers, we need to learn to get a sense of how much testing is enough. 5 playtests? Definitely not enough. 500? Probably too many (unless your game takes only a few minutes). Playtest too little, and it’s a crime against your players. Playtest too much, and it’s a crime against yourself: the game may begin to take over your life in a bad way, changes will make little progress, and you begin stealing time from yourself that you could be spending designing new games and letting the current one give the market a shot where you can learn even more from it. (Or you can decide it’s just not a good game, and let it die in a corner).

    It’s hard to know exactly when this sweet spot of “the right amount of testing” is, but a good start for first time designers is to recognize the difference among (1) playtesting just to say you did it, (2) playtesting to add considerable value to your game, and (3) over-playtesting as procrastination because you’re scared of putting it out there for the greater world to see. I’ve done all of these things, and #2 is the place to be.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>