Valid research starts by finding the right people to participate. We call this screening. Screening allows you to require participants to answer one or more questions that you specify to any level of granularity you like.
Our panel is big, consisting of more than 500k people in the US and Canada with Europe and parts of Asia on the way. It's so big and comprehensive, we have no problem letting you require up to 10 screening questions that participants must answer correctly to participate in your study.
In practice most studies ask just one to three questions because most "targeting" can be boiled down to only a few questions. For example, if you're a credit card company you want to ensure that people have a credit card.
But before we delve into creating your screening questions, let's start by answering a question that should always be at the back of any researcher's mind.
How honest are people?
Our participants are honest, contentious people who genuinely want to help you get the feedback you need. That said, there are always people who try to game the system, and online survey marketplaces have a lot of this going on. Generally, we find dishonesty more rampant in markets where people are merely clicking through survey questionnaires—which is different from what we're doing.
Our approach different in that 1) people are incentivized to do good work because we can reject their work and 2) we collect signals that traditional online surveys don't (for example, we record and replay everything someone does, as well as their voice, if thinking aloud is chosen as an option). By the time a response reaches you, it will consist of a genuine, earnest attempt to do your tasks and answer your questions.
Writing a good screening question consists of a question and a list of answer choices. The question can either be a single or multi-select (multiple choice) style. And like all of our survey-style questions, you can choose from a list of pre-made questions or create your question from scratch.
Screening questions are created when you create your study. A screening question differs from a regular study question in one important respect. Each screening question item has a Screen-Out checkbox. Select this box if you want to screen people out who select this item.
Writing a good screening question
A good screening usually consists of a list of choices which don't reveal the intent of the question. So, for example, consider the difference between a yes/no question like "Do you have a credit card?" and a question that asks "Please tell us what financial products you currently have" which lists five to seven options, one of which is a credit card.
Structuring your questions this way can also help when presenting your results, because you've just minimized a natural reservation people will have.
What if I don't want to screen?
You don't have to screen, and some projects do not require it since the market for the product is pretty much anyone. If you have a product like this, you'll be happy to know these studies are the least expensive and fastest to run.
When you don't ask a screening question in your study, we still ask three basic demographic questions about gender, age, and technology use, so you'll always have some information about who each participant is.
The impact of screening on cost
As you probably can guess, the more screening questions you ask, the harder it is to find participants that match a given profile. The way we manage this is by increasing the incentive (the amount of money we pay the participant). The increased incentive improves the likelihood that people will try to qualify.
The simple formula is this: for every screening question you ask, we add an additional $5 onto the per-response charge. This allows us to pay the participant more and generally ensures your study will be a success.
The impact of screening on study completion time
Because we increase the participant incentive the more questions you ask, the impact of screening on study completion time is often negligible. Of course, it all depends on what you're screening for, the number of participants and so on, but generally most of our studies complete in a day or two, with longer times for larger samples.
Getting an even mix
When you're doing an experience2 test to compare one or more sites or prototypes head-to-head, you'll want to consider whether you want what we call an "even mix" of participants on each site or prototype. For a simple example, getting an even mix on gender will give you a roughly 50% mix of men and women on each site or prototype in your study that compares two or more sites or prototypes. For other standard questions such as income or age, the mix you will get reflects the distribution of these categories in the general population. So you'll get fewer high-income people for example.
Screening and other demographic questions
Sometimes you want to collect information about who people are (their demographics) without screening per se. If this is the case, we generally recommend including the question in the body of the study itself (in the Study portion of the create study wizard, rather than in the Screening section). That said, there are two exceptions to this.
- When you would like to get an even mix of participants on each site or prototype in an experience2 study
- When you want an even mix of participants in a basic usability test
Getting an even mix of people means we will screen the same blend of each demographic category into the test. When you're getting an even mix of people, screening questions can contain a combination of screen outs, non-screen outs, or no screen outs at all.
Why all this focus on "mixes"? Methodological sticklers will require it if you want to compare two groups, as you usually do with an experience2 test. It lets you say things like "we showed the same group of people version A and the same group of people version B, and 20% more people preferred version B over A."
To screen or not to screen
A good test is this: if you want the question to affect the allocation of participants within the test, make it a screening question. If it doesn't matter to the allocation and you want to know something about a participant, put it in the body of the study.