SoundingBox will end operations on August 1, 2021. Read the announcement. [x]

# Competitive Tests

Competitive analysis doesn't have to be a dark art. With SoundingBox any team can discover opportunities for improvement or set strategic direction.

Competitive research is a strategic research practice. Traditionally, competitive analysis is conducted by someone who goes through the competition's online offerings and makes a big spreadsheet enumerating features. While not without value, this kind of analysis can't tell you much about how actual people experience competitors' digital offerings. It won't tell you how your look-and-feel compares, or how well different approaches to content or other aspects of experience work—relative to the competition.


To jump-start your competitive test, create a competitive test from our template. Competitive tests are made possible by the split test test architecture.

A SoundingBox competitive study makes your competitive landscape your laboratory. It gives you a framework for being able to identify how subtle aspects of experience design are working for real people in a competitive context. To put it another way: competitors' sites are out there, they're public, why not use them to figure out answers to tough questions?

  • How does our look-and-feel or brand message perform? Are competitors' better?
  • How engaging is our content compared to competitors?
  • Are there features or other interactions we should consider building out to enhance the experience that we haven't thought of?

# How it Works

A SoundingBox competitive test works by dividing your participants into groups, groups that are usually composed of the same demographic mix. The system automatically allocates each group to one competitor site. Each group completes the same task or set of tasks on each competitive site. And each group answers the same set of questions after doing the task or tasks on each site.

In a competitive test, participants aren't asked to detect differences between the designs on their own. One participant only interacts with one site. The numbers tell the story. Once complete, you can see the extent to which your site is more or less engaging than the competition. You can see the extent to which your content is more or less informative. You have complete control over the metrics you choose to quantify and compare.

Getting benchmarking numbers is just the beginning. Qualitative data helps flesh out the narrative. You can go back to your team and present empirical results: our site falls short of the competition in these ways. This feature resonated with people on competitor X. Should we build a feature like it? Our look-and-feel needs an update.

# Creating a Competitive Test

Create an account if you haven't already. Create a new study based on the competitive research template. A competitive test is a kind of split test. Split tests are like other SoundingBox studies in that they are comprised of tasks (activities we ask people to do) and questions (where we ask people about how they felt after doing their activity).

# Setting up Groups

Partitioning your responses into groups happens when you set up your tasks. Tasks in a split test have additional properties that allow you to enter multiple group URLs per task. On the backend, we take care of the rest. If you're asking screening questions, and choose to get an even mix of participants, the same combination of participants will interact with each task group URL, giving you the apples-to-apples comparison that you need.

# A Note on Sample Size

Since competitive tests usually involve asking people about their opinion about an experience, having an adequate sample size can be vital to making claims about your data. You can remain agile and not break the bank with around 30 participants per group or competitor. If you want greater certainty about your results, you're welcome to choose to make your sample size more substantial, and many customers do.

# Think-aloud and Competitive Tests

To be most meaningful we recommend including at least three competitors in your competitive test. If you have a lot of competitors, then including more will increase the value of your test. When the number of participants grows beyond 30 or 40 total, the importance of having participant audio begins to diminish, since you may not have the time or stamina to watch and listen to every session. Opting not to capture sound also reduces the per-response cost, making it easier to get to a sample size that you can generalize about.


Capturing participant audio can be turned off when you create your task in your study. This will reduce your cost by $5 per participant.

# Analyzing Results

Once your results are in, click on Analyze and load up your study by clicking on the study tile. You'll notice right off that we load each competitive site to the right of your study, and clicking on the competitor tile loads that competitor's data blocks, each of which summarizes the questions you've asked. At a glance, you can see which competitor did best by looking at a tile called Overall and toggling between the competitors.

Next try clicking on the mixer. Here you'll see a chart showing the same summary data for each competitor, with one competitor site for each dot. If you've iterated and have prior competitive tests you'd like to compare, load them and you'll see them in this view alongside your current study, showing you how much things have changed between iterations.

Read more about our analysis dashboard.

# Getting to the "Why"

All of this is just a starting point. You can see which competitor won overall, and which competitor won for each thing you've measured, but you still need to come up with some reasons why they won so you can tell the story to our colleagues. That's where the open-ended responses and the replays come in. Often participants will give you clues about their feelings by telling you about them in the open-ended (free text) responses you've asked them to provide. You can find other signals by replaying their interactions. Did they encounter usability problems, or react adversely in their open-ended responses?

You'll find replays in the Replay tab, and you'll find open-ended text responses in the Grid view. Remember clicking on any tile or data point in the dashboard will sort replays by that measure, making it easy to prioritize which responses to watch first.

# Design Strategies

With usability-style tests there are two general types of tasks that you want to consider: open tasks that ask people to explore with little prompting, or closed tasks which give people clear goals. Each has its value. For competitive tests, sometimes open tasks can be the most revealing since with an open task test, each site is allowed to define the experience. If a site prioritizes specific experiences over others, that will likely determine the outcome and contribute to its scores. With closed tasks, the researcher to some extent determines the outcome because you're choosing what people do. If one site happens to prioritize features that fall into the task you've defined, it will likely do better. The competitive test template has an example of each task type.

In a perfect world you would always start with an open task test to try to get the most "clean read" possible, and then, as needed, focus on specific closed tasks, if they are critical to your strategy.

Read more about open and closed tasks.