These are excerpts of an article originally published in Social Media + Society. Read the full article here.

By Tarleton Gillespie Senior principal researcher at Microsoft Research New England Affiliated Associate Professor in the Department of Communication and Department of Information Science at Cornell University

Introduction

It is easy to assume that, in the current debates about how social media platforms moderate problematic content, that content moderation is content removal: deleting content and suspending users. Even the journalists, critics, policymakers, industry stakeholders, and academics who know better spend an inordinate amount of time focused on removal. Debating whether a platform should permanently ban the sitting president of the United States is a potent way to interrogate its influence. The First Amendment implications of removal lure journalists, pundits, law scholars, even social scientists. Accusations of “censorship” by critics have the most traction when content has been completely deleted or a user has been permanently banned.

But while removal may be the most visible response, it is by no means the only remedy available. Besides being able to (1) remove content and users, platforms can also (2) implement age barriers, geo-blocking, or temporary holds, to keep problematic content away from some users some of the time; (3) append fact-checking labels and interstitial warnings, to alert users to problematic content before or as they encounter it; (4) impose demonetization and punitive strike systems, as disincentives for producing problematic content; (5) and provide counter-speech and model preferred norms, to raise the overall level of discourse (Goldman, 2021). Some of these strategies are not as visible or controversial as removal; others may go unnoticed because they are imposed by different product teams, intervene at different points in the platform, are more difficult for users to identify in practice, or get justified in different ways.

This essay focuses on yet another type of remedy: when platforms (6) reduce the visibility or reach of problematic content. Many social media platforms have quietly begun to identify content that they deem not quite bad enough to remove. The offending content remains on the site, still available if a user can find it directly. But the platform limits the conditions under which it circulates: whether or how it is offered up as a recommendation or search result, in an algorithmically generated feed, or “up next” in users’ queues. This is not a new policy or technique, exactly, though it is being deployed to new ends, for an expanding list of reasons. Reducing the visibility of risky, misleading, or salacious content is now used by many platforms to curb any content they deem to nearly violate their rules, across nearly all the traditional Trust & Safety concerns.

YouTube and Facebook, which have most clearly acknowledged these techniques, call them “borderline content” policies. I’m reluctant to adopt that term: using “borderline” as a pejorative has quiet resonances with assumptions too often made about both geographic borders and “borderline” mental health conditions, in ways I don’t want to reify. Critics have called this “shadowbanning”[1] or “suppression”[2] – I find these terms more compelling, though I worry, along with Cotter (2021), that they may inadvertently help platforms dodge the very criticism being levelled at them. I will use reduction to encompass all of these.

That there isn’t yet a settled industry term is telling. Platforms are, understandably, wary of being scrutinized for these policies — either for being interventionist and biased, or opaque and unaccountable. Some platforms haven’t acknowledged them publicly at all. Those that have are circumspect about it. It’s not that reduction techniques are hidden entirely, but platforms benefit from letting them linger quietly in the shadow of removal policies. So, despite their widespread use, reduction policies remain largely absent from news coverage, debate, policymaking, and even much of the scholarly conversations about content moderation and platform governance.

My aim is not simply to ask whether these techniques are good or bad, or suggest how they could be improved or regulated. I want to argue that reduction techniques should be included within a broadened definition of “content moderation,” not adjacent to it. Content moderation and algorithmic recommendation are by and large treated as different platform functions, handled by different product teams, and scrutinized by different critics according to different measures and concerns (Caplan, 2019). But if we remain analytically agnostic about these distinctions, we might instead see both content moderation and algorithmic recommendation as governance – by different means, deployed in different ways, with different justifications.

First, I will define reduction techniques and identify how they are implemented by several of the major social media platforms. Then I will examine how the platforms explain and justify these interventions. I will explain how reduction works and offer a typology of reduction techniques. I will then make a case for why reduction must be understood as part of a broader content moderation project, and how doing so changes how we think about content moderation and platform governance…

“Borderline Content”

YouTube announced its “borderline content” policy in a January 2019 post, though the practice had already been in place for a few months or more. This followed on the heels of nearly two years of blistering criticism, not only for hosting problematic content, but for amplifying it with its recommendation system. Ex-YouTube designer Guillaume Chaillot specifically blamed the recommendation algorithm for the glut of misinformation, warning that “fiction is outperforming reality;”[3] Zeynep Tufekci, commenting on a Wall Street Journal exposé of conspiracy videos, called YouTube “the great radicalizer” for its tendency to recommend increasingly extreme videos, regardless of the topic.[4]

It is worth lingering for a moment on how YouTube framed this new intervention. The title of the post, “Continuing our work to improve recommendations on YouTube,”[5] gives no indication that this is in any way a Trust & Safety concern. The post begins with a lineage of adjustments YouTube had already made, including limiting “clickbaity videos with misleading titles and descriptions,” and “getting too many similar recommendations,” part of the “hundreds of changes to improve the quality of recommendations for users” YouTube had made just in the past year. Then the new policy is introduced as if it isn’t new at all:

Harms are presented as falling along a spectrum: “content that comes close to—but doesn’t quite cross the line of—violating our Community Guidelines.” In this spatial understanding of harm lies a “borderline,” just left of the existing prohibitions (Maddox and Malson, 2020).

Notice that the problems identified are traditionally moderation issues — misinformation and conspiracy – but the tactic is drawn from recommendation quality: clickbait, repetitive suggestions, and so on. While discussions of online harm tend to highlight victims and perpetrators, quality interventions tend to treat users as consumers, and are more concerned about customer satisfaction than the public interest.

YouTube then quickly assures the reader that:

Reduction techniques depend on this kind of demarcation, between what is hosted and what is recommended, between archive and algorithm. Reduction only pertains to where content is offered up, suggested; not where it lives in the archive (Gillespie, 2016).[6]

YouTube later incorporated this strategy into a broader approach to governance that the company called “The Four Rs”: remove, raise, reward, and reduce. By late 2019, YouTube PR could assert that the effort to “reduce” borderline content and harmful misinformation was working: “The result is a 70% average drop in watch time of this content coming from non-subscribed recommendations in the U.S.”[7] – though as Lewis noted, such claims are nearly impossible to confirm, especially given that the category itself is defined by YouTube’s efforts to identify it.[8]

Facebook and Instagram had already announced their reduction policy in May 2018[9], after also enduring two years of criticism for amplifying misinformation.[10] The terminology and justifications are similar to YouTube’s: “There are other types of problematic content that, although they don’t violate our policies, are still misleading or harmful and that our community has told us they don’t want to see on Facebook — things like clickbait or sensationalism. When we find examples of this kind of content, we reduce its spread in News Feed using ranking…”[11]

Facebook later published a detailed list of what pages, groups, or events they will not recommend[12] and an exhaustive “Content Distribution Guidelines” indicating what they will “demote” in the News Feed.[13] Facebook’s list includes three categories of concern, two of which echo YouTube’s: borderline content and harmful misinformation, but also low-quality junk. Facebook will not recommend “content that users broadly tell us they dislike,” meaning clickbait, “engagement bait,” contest giveaways, and links to deceptive or malicious sites; or “content that is associated with low-quality publishing,” including unoriginal or repurposed content, news that’s unclear about its provenance, or, content enjoying a surge of engagement on Facebook that’s unmatched on the wider web. Including this in their reduction strategy, just like when YouTube frames theirs in the legacy of clickbait, further confirms that this is the aims of content moderation using the tools developed in the pursuit of recommendation quality.

In July 2021, Instagram introduced “sensitive content control,” which allowed the user to adjust the degree to which sensitive content should be filtered out of the “explore” recommendations the platform offers.[14] While the announcement emphasized the agency users were being given, the fact that users could now “allow,” “limit (default),” or “limit even more” sensitive content revealed that such content already was, by default, being reduced. The company did not immediately specify what counted as sensitive; Instagram head Adam Mosseri later indicated that the intervention focuses on “sexually suggestive, firearm, and drug-related content”[15] and was separate from parallel efforts to reduce misinformation and self-harm.

Twitter, LinkedIn, and TikTok have similar reduction strategies already in place, though they’ve been less vocal about them. In a July 2020 promise to better address COVID-19 vaccine misinformation, Twitter asserted that “Tweets that are labeled under this expanded guidance will have reduced visibility across the service… however, anyone following the account will still be able to see the Tweet and Retweet.”[16] They used similar language to describe their approach to election misinformation in 2020 and 2021.[17] LinkedIn has acknowledged that, in response to community guidelines violations, they may “limit the visibility of certain content, or remove it entirely.”[18] That language first appeared in LinkedIn’s “Professional Community Policies” when they were substantially rewritten in December 2019.[19] TikTok has been even more coy, perhaps because their widely praised recommendation engine is so central to their service. But in the community guidelines, the company acknowledges that “For some content – such as spam, videos under review, or videos that could be considered upsetting or depict things that may be shocking to a general audience – we may reduce discoverability, including by redirecting search results or limiting distribution in the For You feed.”[20]

Depending on how we broaden the definition, we can find other platforms engaged in reduction strategies, that share at least a family resemblance. Tumblr, which once took a more permissive approach to sexual content (Tiidenberg et al., 2021), used to limit the circulation of explicit content by refusing to serve up search results to queries it judged explicit. Users could post pornographic images, could even tag them as “#porn” -- but if a user searched for “#porn” no results would be returned. Hashtag blocking has its problems (Gerrard, 2018; Pilipets & Paasonen, 2022; Sybert, 2021). But like reduction, it similarly demarcates between hosting content and offering it up in search results or recommendations.

Reddit’s “quarantine” policy, though structured differently (Chandrasekharan et al., 2022; Copland, 2020; DeCook, 2022), belongs as well. Reduction is just one part of the Reddit quarantine, but the effect is similar. For users who don’t already subscribe to a quarantined subreddit, no posts from within that subreddit will appear on the Reddit front page or be returned in search results or other site-wide recommendations. In other words, it’s there if you go looking for it, but it won’t be circulated broadly. Reddit’s reasoning will sound familiar: “to prevent its content from being accidentally viewed by those who do not knowingly wish to do so, or viewed without appropriate context.”[21] It’s worth noting that Reddit has been more explicit and transparent about their quarantines than YouTube and Facebook have about reduction. Still, Reddit users outside the quarantined subreddit may not know why some things aren’t bubbling up on their front page anymore...

Conclusion: Moderation by Other Means

Major platforms, including Facebook, YouTube, Instagram, Twitter, Tumblr, TikTok, LinkedIn, and Reddit have added reduction to their content moderation techniques. They use machine learning classifiers to identify content that is misleading enough, risky enough, problematic enough to warrant reducing its visibility, by demoting or excluding it from the algorithmic rankings and recommendations – while not going so far as to remove it. They distinguish different aspects of their platforms to ascribe different logics of responsibility and justify different interventions: passive hosting versus active recommending, what users asked for versus what they were presented with. Reduction policies are explained as either responsible oversight of algorithmic systems, or unavoidable responses to the human impulse toward the sensational. Or they are not explained publicly at all.

The way YouTube, Facebook, and others have developed and legitimated their reduction policies reveals a great deal about how content moderation now works. This is content moderation by other means, conducted sometimes by other parts of the companies, deployed in different parts of the platform. In order to address growing concerns about harm (and user speech), these platforms are turning to mechanisms designed traditionally to manage quality (and consumer satisfaction). But, because platforms have until recently been circumspect about these strategies, and because the public debate has focused so strongly on removal, reduction is rarely included in discussions of content moderation, or of the power of platforms over public discourse. These new strategies should remind us that platforms have long managed content they want less of by making adjustments to their recommendations, rankings, or search results: duplicates, spam, clickbait, engagement bait (Hallinan, 2021), NSFW content, bots, coordinated inauthentic content (McGregor, 2020). Yet we tend not to think of any of these as content moderation – even though someone’s speech is certainly being constrained, so as to prevent users from encountering what they either don’t or shouldn’t want.

Whether it is selecting outand selecting for, through policy or through design, with whatever justification – all of it “moderates” not only what any one user is likely to see, but what society is likely to attend to, take seriously, struggle with, and value. Reducing news content so as to improve the “organic reach” of posts from your friends and family is a form of moderation (Cobbe & Singh, 2019). When Mark Zuckerberg, after the January 6 insurrection at the U.S. Capitol, announced that Facebook would begin testing ways to show less political content in the newsfeed,[22] that is moderation too. Whether a platform intervenes at a single post, or all posts that include a single term, or a machine learning classifier’s best guess of which content falls on the wrong side of a rule – or the reduction of an entire category, so as to decrease the likelihood of polarizing, hateful, or misleading content – that’s all moderation (Carmi, 2021). Platforms intervene in the circulation of information, culture, and political expression by removing, reducing, personalizing, rewarding, and elevating; these are overlapping and cumulative strategies, both in practice and in effect, and they must be examined together. As Carmi (2020) put it, “The separation between signal and noise in this context is complicated, as what constitutes a disturbance is decided by multiple actors, and is not restricted to those who create the medium. What needs to be filtered constantly changes because what is considered to be an interference to the business model is also constantly in flux” (p.186).  

If reduction is a form of content moderation, then it must be included in the ongoing debates about platform responsibility. Does it benefit the public, or undermine it, when platforms regularly and quietly reduce what they deem to be misinformation, conspiracy, and “borderline content” violations? We do not know the impact of reduction techniques. We do not know whether that impact differs when what’s being reduced is white nationalism, junk news links, explicit sex work, or users struggling with the impulse to harm themselves (Gerrard, 2020). We do not know whether we can trust platforms to engage in these reduction practices thoughtfully, in ways that produce a robust but fairer public sphere. We have little access to who is making these policies and distinctions, and according to what criteria.

These are excerpts of an article originally published in Social Media + Society. Read the full article here.

Notes