A note about a thing you don’t need to worry about and a technology you don’t need to use.

I recently stumbled upon an article in Axios about how the $15 minimum wage is unpopular with swing voters. It suggested that President Biden might abandon including the proposal in stimulus because it would face too much resistance, and that swing voters’ response in research Axios commissioned might explain why.

Now, as I’m reading this, it’s just one sentence at a time, so I’m surprised to see the name Schlesinger — we use them frequently for qualitative research recruiting, and I suppose I was expecting a polling firm, like Ipsos or Quinnipiac or something — but the research was, indeed, focus groups “on governance in the Biden era”. The groups were conducted on February 9. Which means the Biden era was about 19 days old.

Seven people comprised each group. There were two groups in total. The article used these phrases:

• “A striking 13 out of 14 participants…”
• “Only five members of the group…”

And here I begin to scream silently at the screen. But because anger gives my life meaning, I clicked on the link to the study itself. Here I discover that the participants are asked a series of “discrete questions” using Perception Analyzer, that is, dial testing tools. The purpose? To “eliminate groupthink”. I spiral into despair.

The concern about groupthink when it comes to focus groups is ever present. I am frequently asked what new tips and tricks and tools we have to avoid groupthink (my response: “Nope, we still pretty much just talk to people”).

But here’s what you should know about groupthink. While the term was coined in 1952 by sociologist William H. Whyte Jr., the main research into it was conducted by research psychologist Irving Janis, who studied, among other cases, the Bay of Pigs. I took a political science course in college called Decision Making — the Bay of Pigs is used in these courses as the archetypal example of groupthink.

So, what are the conditions that create groupthink? Janis says groupthink describes excessive consensus-seeking among high-prestige, tightly knit decision-making groups. The consensus-seeking becomes excessive because the members of the group value being members of the group more than anything else — causing them to deindividuate, going along to stay part of the in group, suppressing dissent and following the leader. Not only do they deindividuate, but they also believe strongly in the inherent goodness of the in group, and critically, in the inherent malevolence of out groups.

If this sounds like a focus group to you, then you are watching focus groups designed by sociopaths. Seven strangers picked to be part of a focus group about politics based on being swing voters in swing counties is insufficient to create the excessive consensus-seeking required. Focus groups are generally not high-prestige, nor are their members tightly knit — most of the time they’ve never met before, and in the case of the groups in the Axios article, they weren’t even in the same room! They have no sense of being allies, much less having enemies — they know nothing about other groups, even if there are any.

Focus groups are not deliberative bodies. They don’t gather to make decisions. The stakes of their opinions are typically quite low — they aren’t going to have do anything other than articulate their opinions. Skilled moderators create space for them to do so without blowback from others in the room. Traditional, formal style focus groups barely give time for respondents to build rapport, much less close ties leading to the formation of an in group.

Experienced moderators take great pains to create space for dissent and individuation instead of pushing participants towards consensus, and they likewise try to guide the conversation from one subject to another without leading witnesses. Typically the problem we’re trying to solve is not too much consensus, but not enough people speaking up with their own points of view. They’re not agreeing with each other or trying to ‘please’ the moderator; when ‘conflict’ arises, people decide to keep quiet precisely because the stakes are so low. It doesn’t really matter, they think, if they speak up. They understand this group correctly to be ephemeral, temporary — so there is no point in excessive conformity, nor in excessive disagreement. That’s the actual problem moderators try to solve. But this is not groupthink.

Now, before you go, I don’t want you to miss this opportunity to, well, shrug in indifference about dial tests.

Little Annie, or the Perception Analyzer

What is a dial test? Chances are you’ve seen one on CNN during a presidential debate or State of the Union speech. They’re the squiggly lines indicating how a group of people are responding in real time to the speaker. They’ve been around since the 1930s, and they are… fine.

The very first version of a dial test — also known as the Perception Analyzer or Program Analyzer (or back in the ’30s, Little Annie) was invented by, essentially, an ad sales guy at CBS called Frank Stanton and a social science researcher named Paul Lazarsfeld.

The earliest machine audience measurement systems for ratings had already been developed, but Stanton wanted a machine that could do qualitative measurements of audience response during a program. He thought it could be charted using a kymograph, and decided to partner up with Paul Lazarsfeld, who had conducted some “real time response” research in his early academic years in Vienna with people listening to classical music, to build just such a device.

Now, the first thing you should know is, Lazarsfeld’s early experiments with real time response were not well documented.

“There is, for example, no evidence of how many “listening” studies he actually conducted, nor did he publish any findings based on them.”

Levy, Mark R. “The Lazarsfeld-Stanton Program Analyzer: An Historical Note”, Journal of Communication, Vol. 32(4) — Dec 1, 1982.

The device itself was — and still is — based on some underlying assumptions that, as near as I can tell, have never been scientifically proven (though, in fairness, they don’t appear to have been soundly disproven either). The assumptions include, again from Levy’s 1982 article, “first, that measurements of like-dislike could serve as meaningful indicators of the entire range of subjective reactions to the stimulus; second, that judgments of liking-disliking were made more or less constantly throughout exposure; third, that experimental subjected evaluated “parts” of the stimulus, rather than reaching holistic judgments about it; and fourth, that the stimulus itself, and not other factors such as the setting of exposure, was the major influence conditioning the audience experience.”

In other words — sure, fine, whatever. But really, who knows?

Dial tests were big for awhile on Madison Avenue and at the television networks, and then faded for a time in favor of other methods of measurement, occasionally making a resurgence, especially in the field of public opinion research and political polling, where it continues to thrive.

In 2007, dial tests made a big comeback during the presidential election season, in part due to heavy promotion by Frank Luntz, with backup from a couple of professors of communication at Southern Methodist University, Dan Schill and Rita Kirk. They were big proponents of the technique, particularly as a way of boosting ratings. In 2009, they wrote a chapter in a collection entitled Real-Time-Response Measurement in the Social Sciences, in which they tried to put to rest objections to the methodology by saying:

“One issue that should be put to rest is the notion that RTR is an inherently unsound methodology. . . . Ignoring the large body of research finding strong reliability and validity of the RTR methodology (Baggaley 1987; Biocca et al. 1994; Boyd & Hughes 1992; Fenwick & Rice 1991; Hughes 1992; Hughes & Lennox 1990; Maier et al 2007; Hallonquist & Peatman 1947; Hallonquist & Suchman 1979; Pham et al 1993; Schwerin 1940), these critics mistake the real time response reaction of the dial focus group with a large scale public opinion poll which relies on equal probability random sampling to estimate the attitudes of a larger sample. (Schill & Kirk, 2009, p. 168)”

Mitchell, Gordon R. “Public Opinion, Thinly Sliced and Served Hot”, International Journal of Communication 9(2015), 21–45.

In other words, if you want to be able to predict what a group of people thinks about something, for instance, independent voters in swing counties, dial testing is perhaps not the best tool. It measures, just like it says on the box, real time responses of the dial focus group. The degree to which this group of seven people, or hell, even TWO groups of seven people, are representative of a larger group is severely limited.

For a more extensive inquiry into the validity and reliability of real time response measurement, in the field of music evaluation, this piece is great (JSTOR link, so paywall or institutional login needed) and essentially says, “It depends”.

So, fine — go ahead and use consumer response technology in your Zoom focus groups if you really feel you must. But do not think it’s projectable to larger groups — and do not mistake the twisting of dials on a group Zoom call among strangers, with the way people actually react to bare stimuli out in their communities and on their feeds, or amongst their family, friends, colleagues and peers.

If you made it this far, thank you — your reward is that I can tell you why I silently scream when someone worries about groupthink in qualitative research. Worrying about groupthink tells me that you don’t really know what groupthink is, and that you don’t really know how focus groups work. It might also tell you why dial testing causes me to sigh.

But probably my greatest source of outrage was simply that Axios wrote an article about a technology that can’t be projected onto large populations being used to solve a problem that probably didn’t exist, among 14 people in Zoom-based focus groups about a nineteen day old “Biden era”.

What are we even doing any more, I ask you.