Measuring “iffy” news on social media in time for 2018 election

NewsWhip, a social media engagement tracking firm, collects URLs on hundreds of thousands of sites every day and then gathers information on which of those sites have engagements on Facebook and Twitter. 

Iffy Quotient queries NewsWhip for the top 5,000 most popular URLs on the two social media platforms. Then the U-M tool checks to see if those domain names have been flagged by Media Bias/Fact Check, an independent site that classifies various sources based on their reliability and bias.

The U-M tool divides the URLs into three categories based on the Media Bias/Fact Check lists: “Iffy,” if the site is on the Questionable Sources or Conspiracy lists; “OK,” if the site is on any other list, such as Left-Bias, Right-Bias or Satire; “Unknown,” if not on any list.

Here’s what else the analysis found:

On both Facebook and Twitter, the Iffy Quotient approximately doubled from January to November 2016.

The Iffy Quotient was higher at Facebook than Twitter in 2016 and into 2017.

The Facebook Iffy Quotient has declined steadily since early 2017 and has now returned to its early 2016 levels.

The Twitter Iffy Quotient has not declined much and is still nearly twice its level in early 2016.

The Facebook and Twitter Iffy coefficients were roughly comparable through most of 2018 but Facebook’s is now somewhat lower. The contrast between Facebook and Twitter is even more pronounced in an engagement-weighted version of the Iffy Quotient, which we can think of as a rough proxy for the fraction of total user attention. 

In 2016 the Iffy sites’ share of attention was about twice as high on Facebook as Twitter; now it is 50 percent higher on Twitter.

“By contrast with the current environment of accountability by ‘gotcha’ examples of bad outcomes, the Iffy Quotient tells us something about the overall performance of the platforms,” said Paul Resnick, founder and acting director of the center, and associate dean for research at the School of Information. “The platforms can track metrics internally with their own data, but hesitate to report them externally. By publishing continuously, we can provide accountability when things get worse and credibility for claims of progress.” 

Resnick said reporters and members of the public who would like to monitor social media for the upcoming election and beyond can sign up to receive alerts when the Iffy Quotient changes.

The team plans to add other sources and additional social platforms to the tool. When that happens registered users will also receive an alert. 

CSMR leaders said there are some limitations to the tool and study. Iffy sites sometimes publish reliable information but the metric treats all their URLs as Iffy. NewsWhip also may miss some popular URLs, especially from newer fly-by-night sites. Media Bias/Fact Check may not have gotten around to listing newer Iffy sites. And NewsWhip’s URLs include some that are not related to news or public affairs; the team plans to automatically filter those out in a future version of the tool. 

Because of these limitations, the report authors caution against over-interpreting the Iffy Quotient.

“Trends in the Iffy Quotient over time are meaningful,” Resnick said. “The absolute number…not so much.”