Update from First Draft was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.

As researchers of disinformation, we inherit terms and concepts from platforms that can shape the way disinformation is understood and detected.

But often they reflect the needs of policy communications more than high-quality, independent research.

Coordinated Inauthentic Behavior (CIB) is a case in point — a term devised by Facebook that has shaped our understanding of misinformation, but one that has been criticized for its ambiguity.

For robust research, we need our own concepts and detection methods — ones that are transparent, precisely defined and can be reproduced by other researchers.

we need our own concepts and detection methods — ones that are transparent, precisely defined and can be reproduced by other researchers.

What features of a community should define it as “coordinated?” How can we compare degrees of coordination, or examples of coordination from different contexts and events?

We set out one way to do this: quantitative indicators. In an investigation of coordinated online activity observed during the run-up to the US 2020 election, we used a set of quantitative indicators to precisely define and detect coordination. We show how it can lead to a better approach.

Quantitative indicators for measuring coordination

Platform-defined metrics such as CIB are not designed for independent research. And arguably they shouldn’t be: They exist to support platforms’ policies and their communication, which often requires definitions to be flexible.

It is therefore up to disinformation and social media manipulation experts to put forward independent frameworks for assessing online coordination, as organizations such as EU Disinfo Lab have begun to do. These frameworks should outline specific criteria that can be measured empirically.

This is where quantitative indicators are helpful. Detection models that use explicit quantitative benchmarks are not only more likely to identify coordination with greater accuracy, but also make research methodologies more transparent than an unexplained qualitative assessment. And they provide findings that are reproducible by others.

An example of a quantitative indicator would be something like this:

• When approximately X% of all retweets are identical, a community is defined as extremely coordinated

Methods and findings that are reproducible may attract greater participation and constructive collaboration in relation to the analysis of online coordination, encouraging the development of more widely adopted definitions and measurements.

Transparent and quantitative measures can also provide the foundation for difficult, qualitative judgements about whether coordination matters, and what to do about it. With precise measurements, the degree of coordination can be compared across communities and events (for example, elections) to inform action.

quantitative measures can also provide the foundation for difficult, qualitative judgements about whether coordination matters, and what to do about it

Relative measurements of the degree of coordination among users (for example, the degree of coordination within a community, rather than an absolute number of coordinated users) can be particularly helpful: They can help to uncover the extent to which a group of actors may be sophisticated, well-resourced and dedicated, even if the group is smaller in number.

In turn, this can contribute to a better theoretical understanding of the multifaceted role and impact of coordination in online information.

Case study: Quantitative indicators for coordinated communities on Twitter in lead-up to the US 2020 elections

Leading up to the US 2020 elections, there were many communities coordinating online, to varying extents and with a variety of goals. To uncover what kinds of coordination were taking place, researchers at the Institute of Informatics and Telematics (IIT-CNR) in Pisa adopted precisely the kind of quantitative metrics we have been considering so far.

The goal was to map different communities on a continuous scale and identify those coordinating most intensively and sophisticatedly.

Their method works by detecting coordinated communities on Twitter, based on the extent to which large sets of users repeatedly share (retweet) the same tweets across an extended period of time. Given that coordination is not clear-cut, numerical indicators estimate the extent of coordination for each community detected as “coordinated.”

The quantitative indicators that the team used were:

• When approximately 30–50% of all retweets are identical, a community is likely to be mildly coordinated
• When approximately 90% of all retweets are identical, a community is likely to be extremely coordinated

The model also takes into account the likelihood for a tweet to be retweeted in a coordinated manner. If a tweet has received many thousands of shares, the likelihood of two users both sharing that tweet is high and in turn, the likelihood that they coordinated to share the tweet is low. As a result, common shares of highly engaged-with tweets weigh less than shares of low-engagement tweets when it comes to the final calculation of the coordination scores.

The model was applied to 70 million tweets shared in the run-up to the US election. It could detect both mildly coordinated groups of users as well as extremely coordinated ones.

These quantitative measures were applied to identify groups of users that were coordinating their activity to consistently share the same US 2020-related tweets between October 3 and December 3, 2020. These groups of users or “communities” were then labeled based on the types of hashtags that featured most frequently in the tweets they commonly shared. We then represented these communities visually through a network visualization graph (see Figure 1).

Among other findings, the model enabled the discovery of a small yet highly coordinated network of users that immediately stood out from others.

As shown in Figure 1, two clusters of coordinated users are significantly bigger than the others. The largest one, which appears in blue, is composed of users who supported Donald Trump.

The second-biggest cluster, in orange, is also composed of Trump supporters, but these users shared many conspiracy theories as opposed to generic pro-Trump or pro-Republican tweets. For example, an analysis of the tweets shared by this community revealed widespread support for QAnon and the “Stop The Steal” narrative.

We can see in Figure 1 that most communities appear to be sharply separated from one another, indicating there might be strong coordination within a community, but little coordination (i.e., little sharing of the same tweets) among different communities.

We can also see that one small community, highlighted in purple, appears to be the most densely concentrated. It lies between both pro-Republican groups, yet is also closely linked to the large pro-Trump group.

These unusual characteristics, revealed by the quantitative measurements of the detection model, prompted us to conduct a deep dive into the community’s individual user profiles, the types of messages they were promoting and the sources of the tweets they were sharing most frequently.

This analysis revealed that unlike other traditional, hyper-partisan groups of users who coordinated their online activity to push pro-conservative or pro-democrat messages in the run-up to the election, this network used the electoral debate as an opportunity to generate support for a political cause seemingly far removed from US domestic politics: the independence of Biafra, a small former secessionist state that was reintegrated into Nigeria in 1970.

This pro-Biafran community repeatedly shared tweets that contained pro-independence messages alongside generic US 2020-related hashtags and generic pro-Trump hashtags such as #MAGA and #KAG2020.

Since Trump’s endorsement of Brexit in 2016, Biafran separatists have considered Trump a supporter of their cause and his presidency an opportunity to attract international support for renewed Biafran independence.

Setting standards for a consistent approach

The detection of this pro-Biafran community illustrates how models such as IIT-CNR’s (or others such as the University of Urbino’s CooRnet) can be used to identify highly coordinated communities.

There are three key points we draw from our reflections:

1. Relative indicators can be extremely useful. They help enhance transparency and can point to the extent to which these communities may be well-resourced, dedicated and sophisticated in their behavior. Moreover, methodologies that support fine-grained analyses also make it possible to identify small coordinated networks that might otherwise go unnoticed.
2. Methodologies that rely on quantitative indicators can help standardize how we measure, and by extension understand, coordination. This could allow rigorous comparisons between communities involved in a debate, and between analyses of coordination at play in multiple contexts, such as elections in different years or countries. We need more research to fully explore how this could work.
3. Far from being the be-all and end-all, quantitative indicators provide a starting point from which additional qualitative analyses can be carried out. It is only through further investigation that the most critical questions concerning authenticity, harmfulness and legitimacy can be answered.

The original version of this post incorrectly referred to EU Disinfo Lab as EU vs Disinfo. This has been corrected.

How to improve our analysis of ‘coordinated inauthentic behavior’ was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.

As misinformation researchers, we spend a lot of time thinking about online advertising. We dig through ad libraries, monitor platforms’ announcements, and publish investigations into how disinformation agents are bending the rules.

We rely on social media platforms to give us information to do this. But the experience of working within platforms’ parameters has left us with a question: Can transparency be a trap?

In 2017, Facebook announced it was building a searchable archive of US federal election- related ads that would include some spending and targeting data. Various iterations culminated in the Ad Library, which set the standard for ad transparency. Later, Google also began sharing some information about political ads with researchers. Snapchat did the same, and Twitter eventually opted to get rid of political advertising altogether. By setting policy on it, social platforms have demonstrated they know transparency matters when it comes to political advertising. But they’re also able to control the terms of that transparency. Here are eight big questions that arose when we began scrutinizing the current landscape for advertising transparency.

1. What is obscured by the platforms’ definitions?

What counts as “political” and how is that decided? Election and media law in the US generally defines political ads as those purchased by or on behalf of a candidate for public office, or those relating to a matter of national importance; most major social media platforms use a similar definition.

Facebook calls these “social issue” ads, and defines them as ads messaging about anything “heavily debated, [that] may influence the outcome of an election or result in/relate to existing or proposed legislation.” But who determines what is “heavily debated” or what messaging has the power to influence an election? Advertisements promoting ultrasound services may appear apolitical to most, but if they’re paid for by an anti-abortion organization, they may warrant further scrutiny. On Twitter, political issue ads are banned in the United States, including those from climate advocacy groups. On the other hand, oil companies such as ExxonMobil have been allowed to run ads on the platform. Given the room for interpretation as to what is and isn’t “political,” is the distinction really useful? Should political issue-related ads, such as ads about climate change, count as “political”? And who makes that determination?

As part of a stated effort to protect the US election’s integrity, Facebook did not allow new political ads to run on its platform from October 27, 2020 to March 4, 2021 (with a brief exception made for political ads targeting Georgia’s Senate runoff election in January). But ads about vaccines, ads about election fraud and ads from politically motivated groups including Prager U, the self-described “leading conservative nonprofit,” all ran during this time. Because of the norms established by the platforms, ads deemed non-political are not held to the same transparency standards, so they remain visible to the public, with less scrutiny from researchers. When platforms aren’t thoughtful with their definitions, powerful issue lobbies are able to exploit loopholes to promote their message.

2. Who gets to access and interpret the transparency data?

There are barriers to entry for every mechanism of transparency the platforms have provided us. A researcher looking to explore Snapchat’s political ads archive must be able to run and interpret a csv file. Facebook provides more data to researchers with the advanced skills to access their API. There is also no standardization across the platforms’ databases, making meaningful cross-platform comparisons difficult. So while platforms are increasingly giving researchers access to data, should it only be trained researchers who can scrutinize how social media is used to target communities? How could we open this up for all interested people?

The platforms also fully control what data they make public, and how, and it’s not always particularly useful. For example, Facebook provides impression data for political ads, but it is given in ranges. So an ad could be listed as having garnered <1000 impressions, but there’s no way to know if this means 998 impressions or none. Many advocacy organizations have called for more granular data, which platforms could conceivably provide in a standardized format that allows comparison, or in a user-friendly public interface.

3. Can we be confident that pro-transparency measures are effective?

It is crucial to verify whether nominal pro-transparency measures are having a positive effect. For example, many platforms provide some kind of label that indicates who paid for a political ad. This is an effort to increase transparency, but do the labels being used accomplish that? Facebook has been criticized for its lax advertiser verification requirements that allow advertisers to hide their identity behind shell pages. In this example, Students for Life, an anti-abortion advocacy group, is running ads through a page innocuously called “standingwithyou.org.”

4. Can we rely on these measures being enforced?

Are the tools built by the platforms suitable to deliver on their stated transparency goals? Researchers at the Online Political Transparency Project were surprised to see that ads containing Joe Biden’s name and image were not being picked up as “political” by Facebook’s AI. They were only able to determine this through setting up their own Ad Observer browser extension. How can we know that the tools offered by platforms are working as they are meant to? Platforms could provide more transparency around the methodology used to create these tools, so researchers could audit them for potential issues or errors.

5. Will they be evenly enforced?

A January 2021 study from Privacy International suggested that heightened transparency standards are unevenly applied around the world — authors dubbed this the “transparency divide.” The 2020 US presidential election saw unprecedented measures taken by the platforms that far outweighed their efforts elsewhere. Facebook, for example, publicized what was at the time its largest effort to date to, it said, protect the election’s integrity. At the same time, in India’s Bihar state, with a population of around 104 million people, a critical election for the state legislature garnered no blog posts or announcements from Facebook about protecting its integrity. Facebook and Twitter treated the rampant misinformation during these two elections differently, labeling more misleading posts in the US than it did in India. Transparency measures must meet equal standards globally and be subject to the same levels of enforcement.

6. Is the data reliable?

Researchers have consistently reported errors in the data provided as part of transparency efforts. For example, during the 2019 election in the UK, thousands of ads went missing from the Facebook ad archive because of an error. Similar complaints were made about Google’s ad archive in the US in 2019. What mechanisms are in place to ensure the data we’re getting is reliable?

There is good reason to be skeptical. In 2019, Facebook agreed to pay $40 million to settle a lawsuit alleging it had concealed inaccuracies in its video view metrics that led to a massive and misguided industry shift. Media outlets laid off print staffers in favor of investing in video content based on incorrect information. Why should we take Facebook’s data at face value now? Without independent oversight, there is no reason researchers should consider the data from platforms to be reliable.

7. How does transparency direct our attention?

A new tool for transparency auditing is an exciting thing for researchers, and so it is only right that it should become the subject of academic and journalistic research. But what is being missed when we focus on a particular type of information because of the transparency measures behind it?

Take, for example, how the increased access to information around ads marked by social media platforms as “political” has meant that less attention is paid to non-political or commercial advertising. Facebook has given researchers unprecedented access to advertising data around the 2020 US election, possibly the most scrutinized campaign to date. What about elections where that level of oversight was not in place? This concept is neatly captured as a “feature bias” by our colleague Tommy Shane. The features to which we already have access influence our perspective and, therefore, what we study.

8. What’s transparency for?

Kate Dommett, a lecturer at the UK’s University of Sheffield who studies digital campaigning, wrote in Policy and Internet about calls for more transparency in her field of study in the UK. She found that “despite using common terminology, calls for transparency focus on the disclosure of very different types of information.” Some organizations were calling for financial transparency, others for transparency around targeting data, and only some considered the specifics of how this information would be presented.

Dommett’s research illustrates the pitfalls of demanding transparency for its own sake. When researchers and advocates aren’t specific enough about the outcomes desired, platforms are able to provide an incomplete form of “transparency” as a fig leaf that blunts the political will for positive change. Take, for example, calls for transparency in political spending. If the desired outcome is to monitor the spread of particular messages, and social media companies only offer ad spending data, and not information about impressions and engagement, there are gaps we must seek to fill. Transparency is a tool, not an end in itself; we must reflect carefully on what we want to achieve when we call for it. If we don’t, we’ll keep falling into the trap of false transparency.

We need to know more about political ads. But can transparency be a trap? was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’re a reporter, getting your queries right really matters.

In 2015, Daniel Victor of The New York Times was searching for witnesses to an incident on a plane involving a female passenger and a Hasidic Jewish man who didn’t feel comfortable sitting next to her. Victor found that querying for “hasidic” and “flight” on social media brought up a lot of people talking about the incident, but not people who were actually there.

But then he discovered something. There were three words that could identify genuine eyewitness accounts: “me,” “my” and “I.”

“Most people relating a personal experience — [also known as] good sources — will use [them],” Victor explained. “Most people observing from afar — aka, useless sources — won’t.”

For anyone researching social media, skillful query design is critical. Get it wrong and you won’t find what you’re looking for. Get it right and you can discover surprising things that others are missing.

Rumor cues

In this post, we introduce “rumor cues” to describe an approach to query design that, like first-person pronouns, can be a powerful but overlooked entry point into online conversation.

The term builds on insights from research into how rumors spread, and is designed to help reporters and researchers find truth-seeking behaviors online that contain, or are vulnerable to, misinformation.

To explore them, let’s look at a rumor that spread in the early phase of the pandemic, claiming Washington state would go into lockdown.

In this post there are no hashtags, no conspiracy watchwords, no dog whistles. There isn’t even the word “covid” or “coronavirus.” So how would you find it?

One answer is the word “grapevine.” Another is “heard.” Both introduce the rumor by referring to sources of information.

These kinds of verbal cues typically accompany rumors; researchers have found that others do too, such as “apparently,” “reportedly,” “really?” and “is this true?”

What these words have in common is that they relate to truth-seeking — discussing, evidencing, persuading or questioning what’s true. Like the first-person pronouns “me,” “my” and “I,” words related to truth-seeking — which we’re calling “rumor cues” — can help to monitor misinformation.

Tracking rumor cues is especially important at this particular moment, when networked rumoring can drive life-threatening misinformation, and when routine searches for information online are being weaponized by conspiracy theorists. Both are major social vulnerabilities connected to truth-seeking that we need to identify and better understand.

In this post, we show how reporters and researchers can use rumor cues for three purposes: identifying rumor in real time, newsgathering with shadow queries and understanding the rhetoric of conspiracy theorists.

Identifying rumor

One of the great challenges in tackling misinformation is identifying rumors before they spread. This can help to address deficits: demand for information — often through rumors or unanswered questions — that is not met with adequate supply, creating a vacuum for misinformation.

Crisis researchers have discovered useful insights to help with this. Zhe Zhao and her colleagues, for example, found that expressions such as “Is this true?” “Really?” and “What?” were common in online rumors following the Boston marathon bombings; similarly, Kate Starbird and her team found that the words “apparently,” “reportedly,” “alleged” and other forms of “expressed uncertainty” were commonly associated with rumors.

Other examples might include researchers call “non-specific authority references,” such as “experts” or “doctors,” which are more associated with misinformation. Linguistics research also indicates that terms like “disguised”, “hiding, “show”, “exposes” and “uncovers” feature prominently in discussion of disinformation.

Combining words like “covid” or “lockdown” with rumor cues can help us to sift through enormous numbers of posts to find the rumor in the haystack, and support interventions from credible voices.

Newsgathering with shadow queries

Rumor cues can also be used for news monitoring with shadow queries: slightly inflected search queries, such as “coronavirus truth” instead of “coronavirus facts.” Topic keywords like “covid” can also be combined with Rumor cues, such as “won’t cover this” or “mainstream view,” and words such as “hidden,” “suppressing” and “concealed” to locate unverified counter-narratives.

Shadow queries like these can reveal distinct truth-seeking networks. Working with researchers at King’s College London, we found that two seemingly similar queries, #covidfacts and #covidtruth, uncovered two very different hashtag networks: #covidfacts was linked to fact-checking hashtags such as #factchecking, #factsmatter and #debunking, while #covidtruth was linked with conspiracy hashtags like #dontbelievethehype, #covidhoax19 and #wakeup. The minor inflection of the query with different Rumor cues revealed two separate networks that both sought to establish the truth, but in very different ways and with very different claims.

Analyzing conspiracy rhetoric

Another case for rumor cues is the exploration of conspiracy rhetoric. Researchers have warned for some time that the search for accurate information online is being weaponized by conspiracy theorists in ways that are frightfully difficult to counteract.

To understand the rhetoric that powers this manipulation, I worked with a team of researchers to examine 600,000 conspiracy-related Instagram posts by filtering for rumor cues.

We looked for words such as “truth,” “research,” “evidence” and “trust.” One phrase was particularly common: “trust your.” We found that it was often followed by bodily references — trust your “gut,” “body,” “eyes,” “heart,” “immune system,” “instincts” and “intuition” — indicating a private, embodied form of knowledge, quite a different authority than experts.

Another rumor cue we explored were Bible references, which were often used to evidence and support narratives. We found that Revelations 13:16–17, which makes reference to “the mark of the beast,” was particularly common. According to BibleRef.com, a possible knowledge source for Christians, this may refer to “implanted computer chips or other technology,” echoing a major conspiracy narrative about vaccines.

These alternative authorities to institutional expertise — the body and the Bible — can be uncovered and explored with rumor cues, and can yield insights about how to create effective counter-messaging.

What we mean by ‘rumor cues’

Rumor cues, which we also describe with the more technical term “epistemic keywords,” are any words that can be used to query online spaces or datasets for truth-seeking behaviors, such as rumor or conspiracy theory. These might be words or phrases related to knowing, discussing, evidencing, persuading or questioning what’s true. More precisely, they are queryable traces of epistemic activity in online spaces.

Rumor cues are not a silver bullet. But they can be another tool in your kit when looking for misinformation online.

Thanks to the research team at the Digital Methods Initiative Winter School 2021 for their support in testing out this concept. Thanks also to students at King’s College London, by Jonathan Gray, for their experimentation with epistemic keywords in relation to Covid-19 conspiracy theories in winter 2020 as part of an engaged research-led teaching, with input and support from researchers on the infodemic project and the Public Data Lab.

Finding misinformation with ‘rumor cues’ was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.

Until very recently, mRNA (messenger ribonucleic Acid) vaccine technology and vaccine-derived poliovirus (VDPV) were still considered highly specialized, niche topics. But following the recent Pfizer and Moderna announcements, as well as the subsequent VDPV outbreaks in central Africa, these topics have suddenly risen to prominence. And their rise has been accompanied by worryingly high levels of mis- and disinformation.

We call these situations “data deficits”: where high levels of demand for information about a specific topic are not adequately matched by a supply of credible information. Unlike data voids, where search engine queries turn up little to no results, deficits are situations in which much information exists but it is misleading, confusing, false or even harmful.

These deficits are not the result of deliberate actions from bad actors. In fact, they typically occur when quality information providers are unaware of the demand for information on a given topic or are unable to provide the information in an effective, compelling manner.

However, bad actors can step in and exploit these deficits, filling them with content meant to deceive or that fits their agenda.

So how do certain data deficits get exploited by disinformation actors? How can reporters, policymakers and civil society spot them before that happens?

How data deficits can be exploited

Last summer, First Draft revealed the presence of multiple vaccine-related data deficits through our analysis of the online vaccine information ecosystem, including mRNA- and VDPV-related ones.

Since then, we found that several online messages published by sources identified as “key players in [foreign actors’] disinformation and propaganda ecosystem” exploited these deficits by incorporating them into wider disinformation and conspiratorial narratives. Their apparent aim was to undermine trust in people and institutions connected to the vaccines.

These messages were then disseminated throughout the online information space thanks to a combination of laundering techniques, which are outlined below. Such techniques are frequently employed as part of disinformation campaigns to influence public discourse while obscuring the intentions and identities of the actors involved. Articles mentioned mRNA and VDPV to amplify the narratives that a) US and Western Covid-19 vaccines more broadly are unsafe “experiments” and b) Bill Gates and the institutions connected to him and the vaccines they produce are untrustworthy. These messages were then subsequently:

1. Duplicated and translated in multiple languages by a loose and/or concentrated network of purported news websites and blogs that regularly syndicate each others’ content. The resulting multiplicity of reports artificially enhances the noteworthiness — and, by extension, the credibility — of the messages, thereby exploiting the bandwagon fallacy. The overall audience reached is also maximized as a result.
2. Slightly modified to obfuscate the source and reduce traceability. Small changes are made to the headline and text content, and different pictures are used as the articles’ preview images. Some images can also be added into these messages, many of which tend to be more graphic and emotionally-charged. The sources quoted at the end of the article may also be changed.
3. Spread across multiple platforms, including through simultaneous sharing “spurts” on Facebook Groups to reach a wide range of target communities within an extremely short amount of time. Some links were posted to as many as six Groups in under 40 seconds.
4. Artificially amplified by accounts that exhibit a high number of indicators of inauthenticity.

How to spot data deficits

So how do we identify data deficits before they’re exploited? Here we offer a set of qualitative indicators that can help inform which deficits need to be addressed first. These indicators build on the quantitative indicators and research methods that First Draft has previously used to identify data deficits. If addressed proactively and with quality, accessible information, these deficits may be less likely to be exploited by malicious actors.

Novelty

Is the subject new or previously unknown to a wider audience? This may mean quality information is less likely to exist or to have been disseminated in a compelling and accessible manner. Conversely, the production and distribution of bad quality but equally compelling information is a far more expedient process and may therefore benefit from a first-mover advantage.

Technical complexity

Is the topic characterized by highly-specific information whose comprehension may only come naturally to experts in the field? In this case, easily accessible information may be particularly difficult to produce. On the other hand, messages that simplify the topic in a misleading manner and incorporate it within already-popular narratives are likely to resonate with receptive audiences.

Alignment with pre-existing narratives

Does the subject demonstrate a clear potential to fit into pre-existing, long-standing disinformation narratives? If so, it may be easy to instrumentalize these topics as part of wider misleading messages aimed at exploiting fears, eroding trust and increasing polarization. For example, novel vaccine technologies (such as mRNA ones) could be used to stoke up fears over the safety of vaccines, and thereby bolster narratives portraying all vaccines as untrustworthy.

Political saliency and emotive dimension

Does the data deficit clearly fall within an emotionally-charged issue or wider topic with high political or geopolitical stakes? If so, the incentive for bad actors to exploit the data deficit with disinformation narratives may be high. The vast body of literature on information operations suggests that opportunities to sow discord, undermine democratic processes and amplify emotive tensions are more likely to be exploited by malicious actors.

Legitimate questioning

Is the topic the subject of high levels of legitimate questioning? If so, misleading explanations that address natural concerns may appeal to mainstream communities and thus reach a larger audience. Of course, this heightens incentives for malicious actors as their window of opportunity to bear a greater influence widens. While it may be difficult to distinguish legitimate from illegitimate questioning based on debunked misinformation tropes, one tangible indicator of high levels of questioning can be found using certain pre-emptive research techniques. By searching for social media posts containing interrogative phrases such as “is this true?”, “really?” and “what?” and then clustering these posts based on the similarity of the rest of their content, topics subjected to questioning and rumoring can be identified early on.

We can identify threats ahead of time

We must continuously undertake pre-emptive research that can inform proactive messaging aimed at competing with wider narratives, as opposed to just with individual pieces of content. By collecting and analyzing social media data that falls within the “middle” of social web activity — between verified media outlets or influencer accounts and those found on 4chan messaging boards, private Facebook Groups or other semi-closed anonymous online spaces — with a particular focus on identifying influential narratives, we can identify emerging data deficits early on. Qualitative indicators of data deficits can be used to prioritize responses and in turn maximize impact. The cases of mRNA technology and VDPV suggest the pre-emptive identification of key vulnerabilities is possible.

Qualitative indicators of data deficits can be used to prioritize responses and in turn maximize impact.

The upwards trend in Covid-19 vaccine misinformation and state-linked disinformation is poised to persist. Now is the time to forge greater collaboration mechanisms involving research and monitoring organizations, media outlets, subject-matter experts, platforms and policymakers to ensure data deficits can be identified early and filled with accessible, evidence-based information. Doing so can prevent the successful spread of harmful disinformation narratives.

Identifying ‘data deficits’ can pre-empt the spread of disinformation was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.

“The ocean twilight zone is a layer of water that stretches around the globe … below the ocean surface, just beyond the reach of sunlight … the twilight zone is cold and its light is dim, but with flashes of bioluminescence — light produced by living organisms. The region teems with life.”

— Heidi M. Sosik, 2018

There’s been a lot of debate recently about “Facebook’s Top 10,” a Twitter account that lists “the top-performing link posts by U.S. Facebook pages in the last 24 hours,” managed by The New York Times’ Kevin Roose.

Given that conservative pages tend to dominate the results, the lists have been used to argue that Facebook is biased in favor of conservatives. Facebook, in turn, has pushed back, arguing that engagement doesn’t equal reach.

Irrespective of this argument, “Facebook’s Top 10” points to wider issues about what we see and don’t see in misinformation research. And they go beyond what data we can access, and which metrics we look at.

The top-performing link posts by U.S. Facebook pages in the last 24 hours are from:1. Franklin Graham2. Donald J. Trump3. The Dodo4. Newsmax5. Donald J. Trump6. Donald J. Trump7. Fox News8. Fox News9. The White House10. Donald J. Trump

How do analytics dashboards shape what we see online? What if, by focusing on posts with the greatest engagement, we are missing the things bubbling underneath? Could we be looking in the wrong places and missing real harm, simply because our tools make some things harder to investigate and study?

These were questions our researchers grappled with in our recent research into vaccine misinformation.

To find a solution, we took inspiration from the history of marine biology.

The twilight zone

In 2004, marine biologist Richard Pyle delivered a talk about his research into the ocean “twilight zone.” Pyle discovered that researchers had been focusing on the very top layer of the ocean’s depths. “[We] know a lot about that part up near the top. The reason we know so much about it is scuba divers can very easily go down there and access it.”

The problem was that one can only scuba 200 meters deep. Biologists were well aware of this, and so used submersible vehicles to go deeper. But this created another problem, as Pyle explains: “If you’re going to spend $30,000 a day to use one of these things and it’s capable of going 2,000 feet … you’re going to go way down deep.”

What Pyle discovered was a middle “twilight zone” — so named because of the limited sunlight that pierces to that level — that researchers had neglected because it was easier to look at the surface, and more enticing to go down deep.

This twilight zone, once registered, became a huge source of discovery for ocean biologists, at one point leading to discoveries of seven new species for every hour spent in that region.

Misinformation’s twilight zone

There are a number of lessons here for social media research. We tend to study the accounts with the largest number of followers, the ones responsible for huge engagement metrics. We see network graphs of trending hashtags, dumps of scraped social media data shared by researchers trying to look for evidence of “coordinated inauthentic activity.”

Or we see qualitative researchers lurking in private Facebook Groups, Discord servers or 8kun boards, trying to spot disinformation campaigns before they make their way onto more popular social media platforms.

Both are valuable, but it’s not sufficient for understanding the ecosystem as a whole.

The ocean’s twilight zone is, first and foremost, a reminder that our understanding of misinformation online is severely lacking because of limited data: platforms deny access; ethical guidelines prevent researchers from entering or reporting on certain spaces online.

But more importantly, this maritime comparison is a reminder that our technology can draw us toward seeing some things and not others. CrowdTangle and Twitter’s API are not passive databases that we access, but products with affordances that influence our activity. Some features exist, others do not, and this affects what we see.

the interests of platforms are baked into not just the data they share, but the features they allow for querying it

And critically, the interests of platforms are baked into not just the data they share, but the features they allow for querying it. For example, on CrowdTangle you cannot filter for labeled or fact-checked posts.

Beyond hard limitations such as these, we also need to consider friction — where accessing certain metrics or items is simply made much harder than others. This includes ranked lists that draw us toward the most engaging posts and away from those in the middle zone.

More work is needed to surface feature biases, because we might be missing a critical part of the picture without realizing it.

The problem of feature bias has been raised before. Richard Rogers, a key figure in the development of digital methods, has observed that social media platforms can lead researchers to focus on “vanity metrics” such as engagement scores, rather than “voice, concern, commitment, positioning and alignment.”

But more work is needed to surface feature biases, because we might be missing a critical part of the picture without realizing it.

Applying this to our research

Engaging with the concept of the twilight zone led our researchers Rory Smith and Seb Cubbon to take two critical methodological decisions in their research into vaccine misinformation.

The first, and most fundamental, was to focus on how narratives were evolving and competing rather than on highly engaged posts. The units of analysis in analytics dashboards are individual posts, but narratives are much more powerful than individual pieces of misinformation, shape how people think and can’t be simply debunked.

They also chose to exclude posts from verified accounts as a way of accessing “the middle” of social media activity. The most engaged-with posts were generally from official, often pro-vaccine accounts, such as professional media outlets. Filtering out verified accounts cut through the noise and found more of the anti-vaccine discourse bubbling underneath.

But this was only feasible because there was a feature to filter out verified accounts; otherwise, it would have been very costly to manually exclude them at scale. The filter illustrates our dependence on not just data, but features, and how this affects what we do and don’t see.

In the end, searching for the twilight zone is not a fixed process or location, but a reminder and an endeavor: to think outside the logic of analytics dashboards, and, where we can, look for the neglected parts of the ecosystem.

Searching for the misinformation ‘twilight zone’ was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Covid-19 is neither the first nor the last health emergency we will face. My fellow scientists estimate that we will face a pandemic or health emergency at least once every five years from here on. There is a chance that this is the optimistic scenario. The reality could be far worse.”

Sally Davies, former Chief Medical Officer for England, writing for The Guardian

What could we do better next time?

This is the question we need to be asking now. As we emerge from the immediate aftermath of the crisis, and the infodemic begins to stabilize into a new normal, we must take this opportunity to take a moment and reflect.

Where did we succeed in getting the right information to the right people at the right time? Where did we fail?

At First Draft, we have spent much of the summer tracking the infodemic and its numerous implications, including an analysis of 9,722 fact checks related to coronavirus between January and June.

What has emerged is a series of questions.

We lay them out here to spark conversations among reporters, fact checkers, platforms and researchers, focusing on how we get the right information to the right people at the right time during a pandemic.

1. How do we best respond to questions about origin, when limited information is available?

What do “The Simpsons,” Dettol and Dean Koontz’s novel “The Eyes of Darkness” have in common?

They were all thought to have predicted the coronavirus pandemic.

Though of varying concern, each speaks to a need for an origin story — an answer to the question: Where did this come from? How did it get here?

As Claire Wardle, First Draft’s US director, told the UK Parliament during a session on coronavirus misinformation: “It’s easy to dismiss conspiracies, but we have to understand why they’re taking hold.

“There isn’t a good origin story for the virus, and so this information vacuum is allowing misinformation to circulate.”

The problem is that during a pandemic, we don’t know the answers to these questions. Limited information, and possible government interference, mean fact checkers may struggle with limited evidence to provide a simple answer. In their place, casual, wild speculation and strategic disinformation will triumph.

“There isn’t a good origin story for the virus, and so this information vacuum is allowing misinformation to circulate.”

How can we best address the origin problem? In some cases, might it be better to credibly speculate about what might be the case, instead of failing to fill in the gaps?

2. How can we best support collective sensemaking, when at first all we can do is falsify claims?

In March, crisis informatics researcher Kate Starbird shared an explanation of a social process that follows crises called collective sensemaking: the endeavor to make sense of a crisis by filling in information gaps at an individual and group level. Importantly, it is driven by anxiety and often panic, leading some to taking life-threatening actions.

One of the problems of falsifying claims, such as rejecting claims over the origin of the virus or its alleged treatments, is that they do not fill the gaps in people’s understanding. On the contrary, they may stifle that process by rejecting the information being used to make sense of things.

By contrast, as researchers have noted in relation to other public health crises, rumors will often better serve people’s emotional needs than the accurate information available at the time.

Social media undoubtedly accelerates and amplifies rumor in these scenarios, but it may also be able to help. Just as researchers have explored how to use technology to facilitate distributed sensemaking in hospitals, can social media platforms develop features to safely support the natural and inevitable process of sensemaking following a crisis? How can reporters and fact checkers support?

3. How do information needs change over time, and how can they best be met?

The information needs during a pandemic change over time. Put simply, at first we want to know about origin, then about treatments, and finally attention turns toward public policy.

Connected to this trajectory is a transition in the source and format of misinformation: as the focus moves from online rumors to policy decisions, so too does the kind of misinformation from memes to statements from politicians. These require different skill sets and procedures to verify, and in some cases require more resources. As we noted in our analysis of 9,722 fact checks related to coronavirus, “As the months wore on, the topics that fact checkers addressed increasingly drew on complex political and social phenomena.”

We need a more nuanced understanding of these trajectories, something crisis informatics researchers have already begun to explore.

As the months wore on, the topics that fact checkers addressed increasingly drew on complex political and social phenomena.

We also need to understand whether constraints (such as skills and resources) and incentives (such as Facebook’s payments for fact checks) disrupt fact checkers’ ability to focus on their missions, how this dynamic plays out through the phases of a pandemic, and if the public need can be better served with preparation or emergency support.

4. Which online communities use fact checks during a pandemic, and for what purposes?

It is possible to measure which fact checks in the dataset received the highest number of interactions — the sum of reactions, comments and shares on posts in public groups and pages — using data from CrowdTangle, a Facebook-owned data analytics tool.

But these interactions only tell us that people interacted in some way.

Many questions remain: Who were they, and why were they interacting? Was it in the way fact checkers would want? And what would that even be?

5. What kinds of translation procedures would help critical information to reach communities around the world?

As Poynter has remarked in relation to a viral rumor, “This is a perfect illustration of what ‘infodemic’ is… Just like viruses, misinformation knows no borders, especially during such a crisis.”

But while viruses and viral misinformation might not acknowledge borders, credible information does.

Over half (54.3 per cent) of the fact checks we studied were in either Spanish or English. This breakdown to some extent reflects the relative number of speakers globally, with the notable exception of Chinese, which accounted for fewer than 1 per cent of the fact checks.

However, almost half (54) of 109 major languages were not represented at all, amounting to tens of millions of speakers.

The fact checkers we studied were not the only source of credible information during the pandemic. But questions remain.

Were some people unable to access fact checks in their spoken languages? What was the impact of this? Can emergency human translation support help communities in the immediate aftermath of an outbreak?

6. What elements of the Covid-19 experience can we draw on to tell better stories next time?

As a society, we know more than we did in January. We know what it’s like to gradually learn of a pandemic, for the numbers to be confusing, what “R” means (or at least that it matters), and that, in the end, the origin of a virus may just be a market, as was originally suspected.

Inevitably, the coronavirus will likely be used as a biological and social benchmark for comparison with future outbreaks. How can we best anticipate this?

How can we draw on the new understanding of pandemics and infodemics? Are there numbers, concepts, metaphors, visuals or tools we should re-use and build upon? What worked that we should replicate?

Equally, how will the experience of coronavirus be used against efforts to achieve calm and effective action? Can we explore risks through simulations, or “red teaming”?

— —

These questions will take some time to answer, and will require people with different skills to do it. They are also not exhaustive; they are a starting point for the process of reflection that we must now confront.

But I hope they will contribute to a thorough examination of what happened, and what we need to do next.

The questions we need to ask before the next infodemic was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is part of First Draft’s new Footnotes publication. Read more about what we want to achieve and inspire with this new body of work.

When we think about disinformation, we tend to focus on narratives.

5G causes coronavirus. Bill Gates is trying to depopulate the planet. We’re being controlled by lizards.

But while narratives are concerning and compelling, there is another way of thinking about online disinformation. All narratives, no matter how bizarre, are an expression of something that underlies them: a way of knowing the world.

Contrary to claims that we live in a post-truth era, research suggests that people engaging with disinformation care deeply about the truth. William Dance, a disinformation researcher who specializes in linguistics, has found that people engaging with disinformation are more likely to use words related to the truth, such as disingenuous; nonsense; false; charade; deception; concealed, disguised, hiding, show; find; reveals; exposes; uncovers.

People engaging with false news stories are not disinterested in truth, but are hyper-concerned with it — especially the idea that it’s being hidden.

Contrary to claims that we live in a post-truth era, research suggests that people engaging with disinformation care deeply about the truth.

Because they can seem bizarre, misinformed narratives can sometimes lead others to assume their proponents are simply irrational or disinterested in truth. It can also distract from the ways of knowing that lead people to conspiracy narratives. Not everyone is interested in accounts of the world based on institutionalized processes and the perspectives of experts.

Some people may value different methods, rely on different evidence, value different qualifications, speak in different vernaculars, pursue different logics, and meet different needs. In the words of tech journalist and author Cory Doctorow, “we’re not living through a crisis about what is true, we’re living through a crisis about how we know whether something is true.” Part of that crisis stems from not understanding other ways that people know, and why.

The challenge we face is that we won’t know if trends exist, or if vulnerabilities are growing, unless we ask the right questions. What are the different ways people seek knowledge online? What assumptions underpin them? How are they changing? Are they being manipulated?

We also need to find out how to answer these questions. We need to find a route from the abstract questions of knowledge to the measurable traces of online behaviors.

During the pandemic, we’ve been experimenting with ways to do exactly that. In the midst of the pandemic, we looked at search-engine results using different keywords related to knowing about coronavirus: “facts” and “truth.”

On Google, searching for “coronavirus facts” gives you a full overview of official statistics and visualizations. That’s not the case for “coronavirus truth.” There you’ll get results referring to cover-ups and reports that China has questions to answer about the Wuhan lab — one of the major early conspiracy theories about the origin of the virus.

On Bing, we saw something similar but more pronounced. Where searches for “facts” yielded stories from official sources and fact checkers, one of the top results linked to “truth” is a website called The Dark Truth, which claims the coronavirus is a Chinese bioweapon. Its suggested stories for “coronavirus truth” prompted people to search for terms such as “the real truth about coronavirus,” “the truth behind the coronavirus,” “coronavirus truth hidden” and “coronavirus truth conspiracy theory.” (Since our screenshots were taken, Bing has updated its results for “coronavirus truth” to display data visualizations, but The Dark Truth still appeared as of July, and the related searches were broadly the same.)

These results offer a glimpse into different ways of knowing online, and how platforms are responding with algorithms’ understanding (and production) of the different connotations of “facts” and “truth.”

Even though “facts” and “truth” might feel the same, they appear to engage with different ideas about information: for example, what we know versus what they’re not telling us; official statistics versus unofficial discovery.

When we’ve applied this same principle to the emerging threat of vaccine hesitancy, we see a similar pattern. “Vaccine facts” leads to official sources promoting pro-vaccine messages, while “vaccine truth” leads to anti-vaccination books and resources.

Even though “facts” and “truth” might feel the same, they appear to engage with different ideas about information: for example, what we know versus what they’re not telling us; official statistics versus unofficial discovery.

These results are snapshots of different knowledge-seeking behaviors, which consider a single entry point — “facts” and “truth.” But as this and other research suggests, there is much more to examine about ways of knowing, and how this plays into disinformation.

What might a framework of digital ways of knowing look like? Using facts and truth as illustrative concepts, we’ve proposed several dimensions (scale, process, causal logic, qualifications, method, tone and vernacular) as a starting point for considering different assumptions about what “knowing” might involve and to what narratives it might lead. Such an approach is designed to avoid the judgmental framing that conspiracy theories easily fall into. It is intended to show that certain ways of knowing, like lived experience, have their own validity that might not be recognized by the paradigm of fact checks.

Attention to ways of knowing as well as narratives can help us to ask different kinds of questions. How are ways of knowing changing? How might they be manipulated? How should reporters, educators and platforms respond?

If we fail to ask these questions, there is a risk that we won’t account for — or respect — the different assumptions people make when seeking knowledge. We may fail to speak across divides and ignore how other people’s needs from information can differ from our own.

We might also fail to understand how certain ways of knowing, such as media literacy, can be manipulated and weaponized. We know that some people are more likely to seek alternative, all-explaining narratives — those with low social status, victims of discrimination or people who feel politically powerless. As well as witnessing the rise in 5G conspiracy theories, we may also be experiencing the rise of certain ways of knowing and their manipulation, especially in the context of a resistance to institutions and elites.

Donald Trump’s 2020 campaign has begun to engage with the idea of “truth over facts” with its campaign website thetruthoverfacts.com, which mocks a series of gaffes by Democratic candidate Joe Biden. Though the website is satirical, it primes the idea of the truth being something more fundamental — and Trumpian — than Biden’s misremembered facts.

At First Draft, we plan to develop techniques for monitoring and analyzing these behaviors in the coming months. We want to speak to others interested in this line of research as we experiment with new techniques. If you are interested in the study of online ways of knowing, or have something to tell us that we can use, we want to hear from you. Please comment below or get in touch on Twitter.

The difference between the facts and the truth was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the summer of 2015, First Draft published its first Medium post. The website came slightly later, and with it the flurry of articles, guides, videos, resources, studies, courses and quizzes that have come to define the organisation’s work at the forefront of understanding information disorder.

On the one hand, the topics that inspired the creation of the First Draft coalition five years ago are as relevant as ever. The three tags accompanying that first post are ‘verification,’ ‘UGC’ and ‘eyewitness media.’ There is no doubt that eyewitness footage continues to shape history, as the devastating video of George Floyd’s death demonstrated. And as the global protests continue, videos and images swirl incessantly with associated claims and counter-claims, requiring journalists to apply strict verification protocols in order to document what was happening on the ground.

What’s new are the tactics of media manipulation, the sheer volume of lies being pushed so brazenly by official sources, and the scale of our polluted information environment. 2015 seems quaint in comparison. The terms misinformation, disinformation and coordinated inauthentic activity all feel thoroughly inadequate to describe this moment. And similar patterns are being seen around the world, from expected places such as Brazil, the Philippines and India, as well as countries where the tactics feel less familiar — the UK, France, Australia and Canada, for example.

If the agents of disinformation borrow tactics and techniques from each other, which they do, then so must we.

It’s been three years since I created the seven types of m/disinformation and spent a summer with Hossein Derakhshan writing the report “Information Disorder: Toward an Interdisciplinary Framework for Research and Policy making.” The world looks very different now; the frameworks we’re using, even the terminology and research we’re drawing from, all feel hopelessly out of date. None of us is able to keep up with the speed at which things are moving.

In June, a new piece of research was published that analyzed the impact of the “10 Tips on how to Spot False News” that Facebook rolled out in newsfeeds across 14 countries in April 2017. I’m so glad we have some empirical measure of the impact of that initiative, but it’s over three years after the fact. Do we have to wait until 2023 to find out the impact of Twitter’s new manipulated media labels?

We really hope you will join the growing community of people thinking deeply about these issues so the early ideas published here can be taken apart and made even stronger

The Harvard Misinformation Review, which I helped set up, is doing an amazing job of speeding up the process of getting peer-reviewed research out much more quickly. It’s been an incredible addition to this emerging field.

And in that same vein, we decided that we wanted to create a space for our own staff and guest contributors to test out some early thinking and ideas. So we’ve decided to create Footnotes, a dedicated online space for people to publish new ideas, preliminary research findings, and innovative methodologies. To open up some of our own inspirations and processes to the wider community in the hope that they can be picked apart and borrowed from, driving the conversation ever forwards. If the agents of disinformation borrow tactics and techniques from each other, which they do, then so must we.

We want to encourage people to share early ideas so there is an opportunity to pull them apart and build on them. We hope it might be a place to work through research questions and design, before the process of starting to collect and analyze data. We want it to be a place to suggest, critique and compare notes on ideas and concepts.

We hope the blog will be useful for an expert audience including researchers, disinformation beat reporters, fact checkers and policy makers. Initially we aim to publish once per month, but maybe that will increase with guest contributors.

Our first piece will be out later this week, in which our head of policy and impact, Tommy Shane, shares some early thinking about how different ways of knowing — such as the differences between searching for ‘facts’ and ‘truth’ — may be fuelling misinformation online.

We hope Footnotes is useful, and we really hope you will join the growing community of people thinking deeply about these issues so the early ideas published here can be taken apart and made even stronger.

Introducing Footnotes: A place for ideas was originally published in First Draft Footnotes on Medium, where people are continuing the conversation by highlighting and responding to this story.