"So, Whaddya Think?" as Methodology
Abstracts of long-ago poll-based studies I can neither confirm nor cite
My training in economics gives me a particular distrust of and disdain for social science research that seeks to ascertain objective reality by asking people, “So, whaddya think?”
Years ago, a left-of-center American doctor I know vacationed in a European country and told me that his faith in government-provided healthcare was reinforced by conversations he had while traveling: “Folks I spoke with seemed very pleased with their healthcare system.” I pointed out that all his interlocutors were fellow guests in the upper-tier inns where he stayed—not members of lower socioeconomic castes whose healthcare experiences might be very different. I did add that it’s entirely possible that poorer citizens might still have proven just as enthusiastic, not because the healthcare system was actually good, but rather because people in that country tend to respond happily to pollsters on any question. I told him that if the Canadian healthcare system began using rusty lawnmower blades as tongue depressors, Canadians are so relentlessly enthusiastic that they would likely tell pollsters that they were extremely pleased by this development because Canadian healthcare providers only use the very best rusty lawnmower blades and because every Canadian has an equal right to rusty lawnmower blades in their mouths.
Over the decades, I’ve accumulated scads of illustrative stories that I’ve used to support my skepticism about polling and surveys. Unfortunately, I have not always been fastidious about recording the sources behind the stories, so some of these accounts are now faintly recalled anecdotes with long-lost bibliographic citations. Following are five such examples from what I might dub the Journal of Vaguely Recalled Social Research (Volume 1).
Take these accounts with a grain of salt. If any of you care to play sleuth and hunt for my long-lost sources, I’d be delighted for you to post whatever citations you come up with in the comments section.
JAPAN, MINNESOTA, AND MATH
Educational researchers polled parents in Minnesota and Japan as to their opinions of their children’s mathematics educations. Parents in Minnesota were generally ecstatic about the quality of instruction in their schools. The teachers were excellent, and the curricula were well-planned. Parents in Japan were just the opposite. Their children’s math courses, they said, were terrible. The quality of teachers was poor and the teaching strategies ineffective. As a final step, the researchers compared the standardized math scores of students in both places and found that the students in Japan’s terrible math programs scored considerably higher than those in Minnesota’s delightful programs.
ON DYING AND ETHNICITY
Sociologists examined terminally ill patients of differing ethnicities to compare differences in their coping mechanisms. Patients of British descent strove to maintain quiet stoicism, accepting their fates in quiet dignity. Italian patients wept uncontrollably. Irish patients repeatedly asked what they had done to cause God to visit such fates upon them. Jewish patients relentlessly offered detailed medical advice to the doctors treating them.
AMERICAN DREAM VERSUS CANADIAN DREAM
Political scientists examined the differing causes of national pride between Americans and Canadians. Americans tended to cite abstractions: “I love freedom of speech.” “I want to live where one can worship according to his own beliefs.” “I like our work ethic and the ability to make something of oneself with creativity and hard work.” “I like the freedom that comes with broad horizons and the ability to move about freely.” Canadians, in contrast, tended to cite their favorite government agencies: “I like our healthcare system.” “I admire the Mounties.” “I respect our social safety net programs.” “I like the cleanliness of our railway system.” In Annie Proulx’s novel, The Shipping News, a Newfoundland newspaper editor, displeased by his province’s decision to join the Canadian Confederation in 1949, says:
“There is more government in Canada than any other place in the world. Almost half the population works for the government and the other half is worked on.”
THE RELATIVE IMPORTANCE OF BREAST CANCER AND BUNIONS
Medical researchers examined the notes of an 18th century German physician whose patient was a woman who had breast cancer and also a bunion on her foot. She wasn’t too concerned about the cancer but was deeply worried about the bunion. The cancer, she knew, would kill her before too much time passed. The bunion hurt and interfered with her work. She had little concern about the cancer because it had little practical impact on her daily life and because she knew that the doctor could do nothing to alter its course. She was deeply worried about the bunion because it was something that a competent doctor might be able to relieve, thereby providing her with immediate relief and improved capacity to go about her daily chores. This case offers some insights into the so-called “worried well” in our own time.
THE RESPLENDENCE OF GOVERNMENT HEALTHCARE
Policy researchers in the United Kingdom asked people for their opinions of the National Health Service, and the results were overwhelmingly positive. One respondent had a disfiguring ailment that made it deeply embarrassing for her to go out in public. So, she had mostly cloistered at home for a number of years while waiting for a surgical date to correct the disfigurement. She was, nevertheless, highly enthusiastic about Britain’s healthcare system, because, “Were it not for the NHS, I wouldn’t be able to get corrective surgery.”
Enjoy these recollections and use them to your heart’s content. Just know that I can’t confirm the details or provide you with the sources.
PRESIDENT LANDON
Here is perhaps the most infamous methodological error in survey history. In 1936, Literary Digest’s well-regarded political poll predicted a large victory for Republican Alf Landon over the incumbent Democrat, Franklin Roosevelt. Polling was limited to those who had telephones, automobiles, and time on their hands. It did not occur to the researchers that in the midst of the Great Depression, such people were likely to be considerably more conservative and Republican than people who lacked telephones, cars, and free time. The embarrassment of the survey led to the magazine’s swift demise.
I've always wondered why surveys are gaining so much credibility in healthcare "research." Surveys are so open to bias. But then so is much of research just by the decision to exclude certain subjects over others. We need to realize that all research contains bias. And don't even bring up the bias in AI generative text....
I was a resident physician in a large Anglophone hospital in Montreal forty years ago. We used to joke about the different reactions to death in different ethnic groups. The stereotypes I remember is that the Italians would punch holes in the wall and that the English Quebecers would apologize to you for the inconvenience caused by their relative's death.