Calling BS in a data-driven world

By Patricia B. Mirasol

“The world is awash with bullshit, and we’re drowning in it,” is how Carl Bergstrom and Jevin West open their book,  Calling BS: The Art of Skepticism in a Data-Driven World. Having a good bullshit detector during the COVID-19 pandemic is a matter of life and death, the professors said during a webinar aiming to arm people with the skills to identify and challenge quantitative garbage. The fellow professors also teach a course on the subject at the University of Washington. 

Bullshit—or BS—is defined as language, statistical figures, and other forms of presentations intended to impress or overwhelm the audience, with a blatant disregard for truth and logical coherence. “Calling BS” is an utterance in which one publicly spurns something objectionable.

Mr. West said that while humans are good at spotting BS words, numbers are a lot harder. “They seem precise, objective, and scientific. Some numbers are so great they seem to create divine authority.”

FAULTY QUANTITATIVE DATA
Because the COVID-19 landscape is fraught with gaps in information, it has given rise to organized disinformation campaigns, disingenuous federal messaging, fake studies and astroturfing, dreadful preprints, equally dreadful peer-reviewed work, sloppy reporting, and cherry-picking, the co-authors enumerated.

Disinformation can arise because numbers aren’t presented in a way that allows for meaningful comparisons. One simple way to check is to look at the axes of graphs. Does the dependent variable axis of a bar chart go all the way to zero? It should. Bar graphs emphasize the absolute magnitude of values associated with each category. Line graphs, on the other hand, do not need to include zero, because the emphasis is on the change in the dependent variable (usually the y value) as the independent variable (usually the x value) changes.

“Mathiness”—or formulas and expressions that may look and feel like math but disregard the logical coherence and formal rigor of actual math – propagates BS. Few people understand advanced mathematics, so the use of equations is often wrongly equated with rigor.

Selection bias is another element that contributes to disinformation. Selection bias is an error with the methodologies behind recruiting and retaining participants in studies, or analyzing the data obtained. It makes the results a less reliable reflection of the target population. 

EXAMPLES OF INACCURACY
During the webinar, the professors shared examples of how BS proliferates. When the Centers for Disease Control and Prevention released guidance stating that a period of beyond three months is necessary for reinfection to be possible among COVID-19 survivors, news outlets reported those three months as the maximum time period COVID-19 survivors were protected from reinfection. According to Mr. Bergstrom, these news outlets confused necessity and sufficiency. He advised tracking the information back to the original source—in this case, the CDC—to verify reported claims.

Another instance mentioned in the discussion was the skepticism surrounding climate change being the cause of the wildfires in the US. “If the fires in Oregon and Washington are ‘climate change,’ then why do the fires stop at the Canadian border?,” asked one. Several professors explained that while the wildfires affecting the US do not stop at the border, they do stop being tracked by US data at that point.

CLEANING UP THE INFORMATION ENVIRONMENT
People don’t need to have technical expertise, nor know how an algorithm works, to call out problems with data. A Ph.D. is not a requirement to find the truth, said Mr. Bergstrom. “You can spot bullshit by looking carefully at what goes in and what comes out,” he said. Ask: are the numbers or results too good or too dramatic to be true? Is the claim comparing like with like? Is it confirming a personal bias? 

Mr. Bergstrom advised seeking out multiple sources to get the entire picture. “Always triangulate,” he said. “Ask: ‘Who’s telling me this? How do they know this? What are they selling?’” 

Calling BS isn’t about showing how smart you are, said Mr. West. It’s about making the community smarter. “Do it in a way that’s right and concise. Do it in a way that doesn’t attack character.”