Your screener has a user experience too

Back

You’re recruiting players to help assess a game’s user experience.

Before they get there, they have to navigate yours.

Repeated questions. Long lists of titles. Instructions that need reading twice. Another page of profiling, with no indication of how much remains.

Some suitable players may decide they’ve had enough. Their decision to leave doesn’t, by itself, tell you whether they would have been valuable research participants.

Who are we losing along the way?

• At Gromotion, we’ve seen recruitment questionnaires taking around 15–20 minutes with dropout rates above 40%.
• That doesn’t establish that length alone caused the abandonment. Nor can we know how many people who left would have qualified.
• But it raises an important question: are we giving relevant players a reasonable opportunity to be considered?
• Completing a demanding application demonstrates persistence. It doesn’t establish gaming experience, suitability or an ability to contribute useful feedback.

More questions can introduce more exclusions

• A longer questionnaire doesn’t automatically make the audience harder to recruit. Adding more mandatory conditions can.
• A question about another game, a particular behaviour or a narrowly defined preference may introduce another exclusion. Across a screener, those conditions can accumulate until the audience is considerably more restrictive than the original brief suggests.
• Some distinctions are essential. Others may be useful context without needing to determine eligibility.
• Before programming an exclusion, ask what research purpose it serves.

Give screening and validation distinct jobs

• Screening should establish who can be considered and capture what is needed to progress them appropriately. Essential audience requirements, quotas, availability and substantial participation commitments belong early enough for applicants to make an informed decision.
• Validation then allows us to explore relevant experience, clarify ambiguity and assess consistency through recorded responses or a live conversation.
• Some questions are better suited to that exchange than another set of fixed choices.
• Moving a question to validation should improve the assessment. It should not postpone an obvious exclusion or make someone repeat everything they have already answered.

Test the application as carefully as the questions

• Complete the whole screener on a phone, following the routes an applicant might take.
• Look for repeated topics, confusing terminology, cumbersome answer lists and unnecessary effort. Where masking is needed, consider whether its extent is proportionate.
• Give an honest completion estimate. Check that progress indicators, where used, reflect the actual journey.
• Every question should have a clear job. The complete questionnaire should also be a reasonable experience.
• At Gromotion, we want suitable applicants to reach the point where their experience can be properly assessed. Thoughtful screening and human validation work together to support that.

Developing a gaming screener? Get in touch to discuss what needs establishing upfront and what would benefit from a conversation.