We’re trying to create the conditions for a truly representative sample in market research – want to help?

The tl;dr – we’re building a qualitative research recruitment panel that is designed to be inclusive and accessible from the ground up. If you or someone you know would like to join that panel, and be connected with opportunities to earn money for their opinions, we’d love for you — or them! — to sign up. The full URL is below, so you can just keep scrolling, or smash this button:

Representative samples aren’t representative.

One of the major complaints about qualitative research is its inherently small sample sizes. There is always a concern that there is some bias in the sample — some way in which this small group of people are not representative of the audience the brand is trying to reach, or the customers the product team is trying to serve. For many companies, the ‘answer’ to this ‘problem’ is to do quantitative research, or to do something that people like to call ‘quali-quant’ (a phrase I would like to banish from the marketing pantheon, and I hope you’ll support me in this one autocratic desire), which is, I guess, asking hundreds of people open-ended questions.

Except asking more people the same questions is no cure for the bias that underlies the entire market research industry. The panels, recruiters, and the screening methods we use are inherently un-representative, both by default, and then of course, by design. We exclude people from our samples all the time, in a variety of ways both implicit and explicit, and this is a far, far greater problem than sample size. Let’s start with some of the implicit ways we create un-representative samples.

All sample sources are opt-in.

Nobody is required to respond to a survey or an invitation to participate in qualitative research. The people who say yes to such an invitation are people who variously are curious about the topic or about research, or who have an opinion they’d like to share, or who want to be heard, or need the money, or just want the money, or think it sounds like fun, or want an excuse to be out of the house, or something to do with their spare time. These are sometimes talked about as illegitimate reasons to participate in research — but they are the most common reasons, and I’m not sure that there are any better reasons, or to be honest, why we as researchers care.

Next, let’s talk about the experience of opting in. Signing up to participate in qualitative research is usually the product of word of mouth — a friend of yours did a focus group, and told you they had a good time and made $100, so you should look into it. Or you were approached at a shopping center and asked if you’d like to take a survey. Or you were intercepted on a website to answer a few questions and win a sweepstakes. Or some brand you have an online account with entered you into their customer research database and you opted in to have your information shared with their research platform, like e-rewards or something similar. These methods assume a lot about the person who opts in.

But one of the things these sites assume above all the other assumptions is that, not only do you have money to spend, and friends to tell you about focus groups, but that you have access to transit, access to the internet, and that when you get to the place or the website where the research is being conducted, that the site is accessible to you, that you can use it.

A lot of research experiences are not accessible.

Survey websites may not be compatible with screen readers or other assistive technologies for the vision-impaired; they may not provide captions on video or audio stimuli for people who are deaf or hard of hearing; they may not work with assistive technologies for people with limited mobility. In-person research may have similar problems — lack of ramps, assisted doors, parking, bathroom stalls are not un-common; room set-ups that are hard for people to physically navigate if they have mobility, vision or hearing problems, methodologies that don’t take into account non-neuronormative learning styles or disabilities. To say nothing of the challenges of getting to a focus group facility located far away from a bus stop, or a website that is not accessible via a mobile device.

We exclude people before we even start.

And then the explicit exclusion begins.

We make assumptions in our recruiting. There may be initial ‘random selection’ for the invitations to participate in research, but from there, we have to take the random selection and look for those participants who are representative of the audience we’re trying to reach. So we use screeners to make sure we both have enough people who are representative of the audience, and to screen out people who are not representative.

It’s typical to screen out people for a variety of reasons, however, that may have nothing to do with whether they are representative of the actual audience. We frequently screen out people who don’t have a high school diploma or equivalent, for example, even though people without high school educations can still have jobs, drive cars, use the internet, and buy consumer goods — I think we assume there might be some trouble with literacy or reliability. We often screen out people who are unemployed or on disability — some assume that these participants are ‘only in it for the money’ or that ‘all they’ll want to talk about is their disability’ (actual reasons people I’ve worked with have given).

But to be honest, I’ve never really had a colleague or boss explain why we screen them out, only that we do. Some of it, I suspect, is simply that brand managers rarely imagine their customer base to include people of lower socio-economic status, so we focus on the middle class and affluent, the educated, the employed, and the able-bodied. Another part of it is that the people who work in the research industry are predominantly white, college educated, affluent and able-bodied — and some have some pretty unfortunate beliefs about people who are not like them, even if they’re not wholly conscious of those beliefs.

These exclusions – implicit and explicit – matter.

Who are we excluding when we set these screening parameters?

• The CDC estimates 61 million Americans [1] live with disabilities; over a quarter [2] of the US adult population lives with multiple chronic conditions. About 8 million people in the US receive disability benefits from SSDI [3].
• Just under ten million people in the US lost their jobs [4] during the pandemic last year; 6.9 million people are currently unemployed [5] according to the Bureau of Labor Statistics.
• As much as 45% of Americans do not have access to mass transit [6]; around 9% of US households do not currently own a car (18% of Black households do not have a car). [7]
• About 5% of US adults [8] — or over 10 million people — have attained less than a high school diploma, according to Census data.
• About 4.5 million people [9] are currently under some form of community supervision (e.g., probation or parole); another 2 million are incarcerated. [10]

These are not trivial numbers, and they represent real people, real experiences. We’re in the business of randomly selecting representative samples to have high confidence in our data and analysis. So if, just as a starting point, your ‘representative sample’ is not 26% disabled, it is by definition unrepresentative.

If we aren’t designing for these (often intersecting) communities, we’re narrowing the aperture too much — we’re omitting literally millions of Americans who, no matter how affluent and educated they are and no matter how much they spend, can not participate in our research even if we let them. And a lot of the time, we don’t let them. A lot of the time, researchers don’t want them.

Well, we want them. We want all of us.

Which is why we’ve decided to build a qualitative research recruitment panel [11] specifically for greater inclusion and accessibility.

We’d love for you to sign up or share the link with your communities: https://8ye75qu0dfx.typeform.com/inclusive-panel?typeform-source=www.thedifferenceengine.co

We’re building a research participant community of diverse people whose perspectives and experiences are under-represented in market and product research. We want to help center different voices that are often left out of the conversation about the way we spend our time and money and effort.

We also believe people should be compensated for their time and expertise — even simply being an expert about their own life and experiences — and want to ensure that people are paid fairly. As part of this community, all our participants are also co-creators — telling us what a fair rate for their time is, what topics they don’t want to discuss, and giving us feedback on how to make the experience of the community more inclusive and accessible as we grow.

This idea was a long time in the making as we’ve worked to make our recruiting and sexual orientation dimensions, but the pandemic really did accelerate our thinking as remote research techniques continued to increase our interactions with the disability community, the formerly incarcerated, the un- or under-employed. They are consumers of our clients’ products and services, and their perspectives are valuable.

But we know that the industry makes it hard to participate — so in addition to building our own panel we’re working to ensure our digital tools and methods meet Web Content Accessibility Guidelines, to vet partners who also provide tools that are compatible with assistive technologies, and to — when we return to in-person research — are accessible to participants with disabilities and/or chronic conditions. Ashley’s been researching and writing [12] a lot about how to improve our processes to be more inclusive, and we’ll continue to share that writing with you.

That’s it. That’s the pitch. Let us know what you think, and if you’d like to get involved. That’s enough from me, let’s see what Ashley and Nic have for us this month — I went on for such a long time they decided to keep it mercifully brief:

From Ashley:

In this video, Liz Jackson, the founder of The Disabled List shares “how the things that disabled people radically fight for become the things that are empathetically done for us”. [13] It’s an insightful perspective into design and the power of creating ‘with’ rather than ‘for’. Also, #comicsanstakeover

From Nic:

This is an OpEd detailing experiences of young people starting careers while never meeting their coworkers, supervisors, and the C Suite; and the impact felt by a lack of pro-social work culture. [14]

URLs:

[1] CDC on rate of disabilities: https://www.cdc.gov/ncbddd/disabilityandhealth/infographic-disability-impacts-all.html

[2] CDC on prevalence of multiple chronic conditions: https://www.cdc.gov/pcd/issues/2020/20_0130.htm

[3] The Center on Budget and Policy Priorities on rate of Americans on disability benefits: https://www.cbpp.org/research/social-security/social-security-disability-insurance-0

[4] Pew Research on jobs lost during the pandemic: https://www.pewresearch.org/fact-tank/2021/04/15/fewer-jobs-have-been-lost-in-the-eu-than-in-the-u-s-during-the-covid-19-downturn/

[5] Bureau of Labor Statistics November 2021 unemployment numbers: https://www.bls.gov/news.release/pdf/empsit.pdf

[6] American Society of Civil Engineers’ 2021 Report Card for America’s Infrastructure: https://infrastructurereportcard.org/cat-item/transit/

[7] The National Equity Atlas’s data on car access; the Atlas is a product of PolicyLink and the USC Equity Research Institute: https://nationalequityatlas.org/indicators/Car_access#/?geo=01000000000000000

[8] The National Center for Education Statistics on education attainment rates: https://nces.ed.gov/fastfacts/display.asp?id=27

[9] The US Department of Justice’s Bureau of Justice Statistics in the Office of Justice Programs data on rates of probation and parole in the US in 2019: https://bjs.ojp.gov/sites/g/files/xyckuh236/files/media/document/ppus19.pdf

[10] Pew Research on rates of incarceration in the US: https://www.pewresearch.org/fact-tank/2021/08/16/americas-incarceration-rate-lowest-since-1995/

[11] Our page with more detail on the panel, including a link to sign up: https://www.thedifferenceengine.co/research-panel

[12] Ashley Derrington’s post on our blog about why we’re talking about disability and inclusion: https://medium.com/the-difference-engine-llc/inclusive-accessible-research-why-are-we-having-this-conversation-1fd9acff1e4

[13] Liz Jackson on inclusive design speaking to AIGA: https://www.aiga.org/inspiration/talks/liz-jackson-honoring-the-friction-of-disability

[14] Anne Helen Peterson and Charlie Warzel on starting a career remotely in the pandemic: https://www.nytimes.com/2021/11/22/opinion/remote-work-gen-z.html?referringSource=articleShare