GUEST ESSAY
I Was the Head of Trust and Safety at Twitter. This Is What Could Become of It.
Nov. 18, 2022
Justin Lane/EPA, via Shutterstock
By Yoel Roth
Mr. Roth is a former head of trust and safety at Twitter.
This month, I chose to leave my position leading trust and safety at Elon Musk’s Twitter.
My teams were responsible for drafting Twitter’s rules and figuring out how to apply them consistently to hundreds of millions of tweets per day. In my more than seven years at the company, we exposed government-backed troll farms meddling in elections, introduced tools for contextualizing dangerous misinformation and, yes, banned President Donald Trump from the service. The Cornell professor Tarleton Gillespie called teams like mine the “custodians of the internet.” The work of online sanitation is unrelenting and contentious.
Enter Mr. Musk.
In a news release announcing his agreement to acquire the company, Mr. Musk laid out a simple thesis: “Free speech is the bedrock of a functioning democracy, and Twitter is the digital town square where matters vital to the future of humanity are debated.” He said he planned to revitalize Twitter by eliminating spam and drastically altering its policies to remove only illegal speech.
Since the deal closed on Oct. 27, many of the changes made by Mr. Musk and his team have been sudden and alarming for employees and users alike, including rapid-fire layoffs and an ill-fated foray into reinventing Twitter’s verification system. A wave of employee resignations caused the hashtag #RIPTwitter to trend on the site on Thursday — not for the first time — alongside questions about whether a skeleton crew of remaining staff members can keep the service, now 16 years old, afloat.
And yet when it comes to content moderation, much has stayed the same since Mr. Musk’s acquisition. Twitter’s rules continue to ban a wide range of lawful but awful speech. Mr. Musk has insisted publicly that the company’s practices and policies are unchanged. Are we just in the early days — or has the self-declared free speech absolutist had a change of heart?
The truth is that even Elon Musk’s brand of radical transformation has unavoidable limits.
Advertisers have played the most direct role thus far in moderating Mr. Musk’s free speech ambitions. As long as 90 percent of the company’s revenue comes from ads (as was the case when Mr. Musk bought the company), Twitter has little choice but to operate in a way that won’t imperil the revenue streams that keep the lights on. This has already proved to be challenging.
Almost immediately upon the acquisition’s close, a wave of racist and antisemitic trolling emerged on Twitter. Wary marketers, including those at General Mills, Audi and Pfizer, slowed down or paused ad spending on the platform, kicking off a crisis within the company to protect precious ad revenue.
In response, Mr. Musk empowered my team to move more aggressively to remove hate speech across the platform — censoring more content, not less. Our actions worked: Before my departure, I shared data about Twitter’s enforcement of hateful conduct, showing that by some measures, Twitter was actually safer under Mr. Musk than it was before.
Marketers have not shied away from using the power of the purse: In the days following Mr. Musk’s acquisition, the Global Alliance for Responsible Media, a key ad industry trade group, published an open call to Twitter to adhere to existing commitments to “brand safety.” It’s perhaps for this reason that Mr. Musk has said he wants to move away from ads as Twitter’s primary revenue source: His ability to make decisions unilaterally about the site’s future is constrained by a marketing industry he neither controls nor has managed to win over.
But even if Mr. Musk is able to free Twitter from the influence of powerful advertisers, his path to unfettered speech is still not clear. Twitter remains bound by the laws and regulations of the countries in which it operates. Amid the spike in racial slurs on Twitter in the days after the acquisition, the European Union’s chief platform regulator posted on the site to remind Mr. Musk that in Europe, an unmoderated free-for-all won’t fly. In the United States, members of Congress and the Federal Trade Commission have raised concerns about the company’s recent actions. And outside the United States and the European Union, the situation becomes even more complex: Mr. Musk’s principle of keying Twitter’s policies on local laws could push the company to censor speech it was loath to restrict in the past, including political dissent.
Regulators have significant tools at their disposal to enforce their will on Twitter and on Mr. Musk. Penalties for noncompliance with Europe’s Digital Services Act could total as much as 6 percent of the company’s annual revenue. In the United States, the F.T.C. has shown an increasing willingness to exact significant fines for noncompliance with its orders (like a blockbuster $5 billion fine imposed on Facebook in 2019). In other key markets for Twitter, such as India, in-country staff members work with the looming threat of personal intimidation and arrest if their employers fail to comply with local directives. Even a Musk-led Twitter will struggle to shrug off these constraints.
There is one more source of power on the web — one that most people don’t think much about but may be the most significant check on unrestrained speech on the mainstream internet: the app stores operated by Google and Apple.
While Twitter has been publicly tight-lipped about how many people use the company’s mobile apps (rather than visit Twitter on a web browser), its 2021 annual report didn’t mince words: The company’s release of new products “is dependent upon and can be impacted by digital storefront operators” that decide the guidelines and enforce them, it reads. “Such review processes can be difficult to predict, and certain decisions may harm our business.”
“May harm our business” is an understatement. Failure to adhere to Apple’s and Google’s guidelines would be catastrophic, risking Twitter’s expulsion from their app stores and making it more difficult for billions of potential users to get Twitter’s services. This gives Apple and Google enormous power to shape the decisions Twitter makes.
Apple’s guidelines for developers are reasonable and plainly stated: They emphasize creating “a safe experience for users” and stress the importance of protecting children. The guidelines quote Justice Potter Stewart’s “I know it when I see it” quip, saying the company will ban apps that are “over the line.”
In practice, the enforcement of these rules is fraught.
In my time at Twitter, representatives of the app stores regularly raised concerns about content available on our platform. On one occasion, a member of an app review team contacted Twitter, saying with consternation that he had searched for “#boobs” in the Twitter app and was presented with … exactly what you’d expect. Another time, on the eve of a major feature release, a reviewer sent screenshots of several days-old tweets containing an English-language racial slur, asking Twitter representatives whether they should be permitted to appear on the service.
Reviewers hint that app approval could be delayed or perhaps even withheld entirely if issues are not resolved to their satisfaction — although the standards for resolution are often implied. Even as they appear to be driven largely by manual checks and anecdotes, these review procedures have the power to derail company plans and trigger all-hands-on-deck crises for weeks or months at a time.
Whose values are these companies defending when they enforce their policies? While the wide array of often conflicting global laws no doubt plays a part, the most direct explanation is that platform policies are shaped by the preferences of a small group of predominantly American tech executives. Steve Jobs didn’t believe porn should be allowed in the App Store, and so it isn’t allowed. Stripped bare, the decisions have a dismaying lack of legitimacy.
It’s this very lack of legitimacy that Mr. Musk, correctly, points to when he calls for greater free speech and for the establishment of a “content moderation council” to guide the company’s policies — an idea Google and Apple would be right to borrow for the governance of their app stores. But even as he criticizes the capriciousness of platform policies, he perpetuates the same lack of legitimacy through his impulsive changes and tweet-length pronouncements about Twitter’s rules. In appointing himself “chief twit,” Mr. Musk has made clear that at the end of the day, he’ll be the one calling the shots.
It was for this reason that I chose to leave the company: A Twitter whose policies are defined by edict has little need for a trust and safety function dedicated to its principled development.
So where will Twitter go from here? Some of the company’s decisions in the weeks and months to come, like the near certainty of allowing Mr. Trump’s account back on the service, will have an immediate, perceptible impact. But to truly understand the shape of Twitter going forward, I’d encourage looking not just at the choices the company makes but also at how Mr. Musk makes them. Should the moderation council materialize, will it represent more than just the loudest, predominantly American voices complaining about censorship — including, critically, the approximately 80 percent of Twitter users who reside outside the United States? Will the company continue to invest in features like Community Notes, which brings Twitter users into the work of platform governance? Will Mr. Musk’s tweets announcing policy changes become less frequent and abrupt?
In the longer term, the moderating influences of advertisers, regulators and, most critically of all, app stores may be welcome for those of us hoping to avoid an escalation in the volume of dangerous speech online. Twitter will have to balance its new owner’s goals against the practical realities of life on Apple’s and Google’s internet — no easy task for the employees who have chosen to remain. And as I departed the company, the calls from the app review teams had already begun.
By Yoel Roth
Mr. Roth is a former head of trust and safety at Twitter.
This month, I chose to leave my position leading trust and safety at Elon Musk’s Twitter.
My teams were responsible for drafting Twitter’s rules and figuring out how to apply them consistently to hundreds of millions of tweets per day. In my more than seven years at the company, we exposed government-backed troll farms meddling in elections, introduced tools for contextualizing dangerous misinformation and, yes, banned President Donald Trump from the service. The Cornell professor Tarleton Gillespie called teams like mine the “custodians of the internet.” The work of online sanitation is unrelenting and contentious.
Enter Mr. Musk.
In a news release announcing his agreement to acquire the company, Mr. Musk laid out a simple thesis: “Free speech is the bedrock of a functioning democracy, and Twitter is the digital town square where matters vital to the future of humanity are debated.” He said he planned to revitalize Twitter by eliminating spam and drastically altering its policies to remove only illegal speech.
Since the deal closed on Oct. 27, many of the changes made by Mr. Musk and his team have been sudden and alarming for employees and users alike, including rapid-fire layoffs and an ill-fated foray into reinventing Twitter’s verification system. A wave of employee resignations caused the hashtag #RIPTwitter to trend on the site on Thursday — not for the first time — alongside questions about whether a skeleton crew of remaining staff members can keep the service, now 16 years old, afloat.
And yet when it comes to content moderation, much has stayed the same since Mr. Musk’s acquisition. Twitter’s rules continue to ban a wide range of lawful but awful speech. Mr. Musk has insisted publicly that the company’s practices and policies are unchanged. Are we just in the early days — or has the self-declared free speech absolutist had a change of heart?
The truth is that even Elon Musk’s brand of radical transformation has unavoidable limits.
Advertisers have played the most direct role thus far in moderating Mr. Musk’s free speech ambitions. As long as 90 percent of the company’s revenue comes from ads (as was the case when Mr. Musk bought the company), Twitter has little choice but to operate in a way that won’t imperil the revenue streams that keep the lights on. This has already proved to be challenging.
Almost immediately upon the acquisition’s close, a wave of racist and antisemitic trolling emerged on Twitter. Wary marketers, including those at General Mills, Audi and Pfizer, slowed down or paused ad spending on the platform, kicking off a crisis within the company to protect precious ad revenue.
In response, Mr. Musk empowered my team to move more aggressively to remove hate speech across the platform — censoring more content, not less. Our actions worked: Before my departure, I shared data about Twitter’s enforcement of hateful conduct, showing that by some measures, Twitter was actually safer under Mr. Musk than it was before.
Marketers have not shied away from using the power of the purse: In the days following Mr. Musk’s acquisition, the Global Alliance for Responsible Media, a key ad industry trade group, published an open call to Twitter to adhere to existing commitments to “brand safety.” It’s perhaps for this reason that Mr. Musk has said he wants to move away from ads as Twitter’s primary revenue source: His ability to make decisions unilaterally about the site’s future is constrained by a marketing industry he neither controls nor has managed to win over.
But even if Mr. Musk is able to free Twitter from the influence of powerful advertisers, his path to unfettered speech is still not clear. Twitter remains bound by the laws and regulations of the countries in which it operates. Amid the spike in racial slurs on Twitter in the days after the acquisition, the European Union’s chief platform regulator posted on the site to remind Mr. Musk that in Europe, an unmoderated free-for-all won’t fly. In the United States, members of Congress and the Federal Trade Commission have raised concerns about the company’s recent actions. And outside the United States and the European Union, the situation becomes even more complex: Mr. Musk’s principle of keying Twitter’s policies on local laws could push the company to censor speech it was loath to restrict in the past, including political dissent.
Regulators have significant tools at their disposal to enforce their will on Twitter and on Mr. Musk. Penalties for noncompliance with Europe’s Digital Services Act could total as much as 6 percent of the company’s annual revenue. In the United States, the F.T.C. has shown an increasing willingness to exact significant fines for noncompliance with its orders (like a blockbuster $5 billion fine imposed on Facebook in 2019). In other key markets for Twitter, such as India, in-country staff members work with the looming threat of personal intimidation and arrest if their employers fail to comply with local directives. Even a Musk-led Twitter will struggle to shrug off these constraints.
There is one more source of power on the web — one that most people don’t think much about but may be the most significant check on unrestrained speech on the mainstream internet: the app stores operated by Google and Apple.
While Twitter has been publicly tight-lipped about how many people use the company’s mobile apps (rather than visit Twitter on a web browser), its 2021 annual report didn’t mince words: The company’s release of new products “is dependent upon and can be impacted by digital storefront operators” that decide the guidelines and enforce them, it reads. “Such review processes can be difficult to predict, and certain decisions may harm our business.”
“May harm our business” is an understatement. Failure to adhere to Apple’s and Google’s guidelines would be catastrophic, risking Twitter’s expulsion from their app stores and making it more difficult for billions of potential users to get Twitter’s services. This gives Apple and Google enormous power to shape the decisions Twitter makes.
Apple’s guidelines for developers are reasonable and plainly stated: They emphasize creating “a safe experience for users” and stress the importance of protecting children. The guidelines quote Justice Potter Stewart’s “I know it when I see it” quip, saying the company will ban apps that are “over the line.”
In practice, the enforcement of these rules is fraught.
In my time at Twitter, representatives of the app stores regularly raised concerns about content available on our platform. On one occasion, a member of an app review team contacted Twitter, saying with consternation that he had searched for “#boobs” in the Twitter app and was presented with … exactly what you’d expect. Another time, on the eve of a major feature release, a reviewer sent screenshots of several days-old tweets containing an English-language racial slur, asking Twitter representatives whether they should be permitted to appear on the service.
Reviewers hint that app approval could be delayed or perhaps even withheld entirely if issues are not resolved to their satisfaction — although the standards for resolution are often implied. Even as they appear to be driven largely by manual checks and anecdotes, these review procedures have the power to derail company plans and trigger all-hands-on-deck crises for weeks or months at a time.
Whose values are these companies defending when they enforce their policies? While the wide array of often conflicting global laws no doubt plays a part, the most direct explanation is that platform policies are shaped by the preferences of a small group of predominantly American tech executives. Steve Jobs didn’t believe porn should be allowed in the App Store, and so it isn’t allowed. Stripped bare, the decisions have a dismaying lack of legitimacy.
It’s this very lack of legitimacy that Mr. Musk, correctly, points to when he calls for greater free speech and for the establishment of a “content moderation council” to guide the company’s policies — an idea Google and Apple would be right to borrow for the governance of their app stores. But even as he criticizes the capriciousness of platform policies, he perpetuates the same lack of legitimacy through his impulsive changes and tweet-length pronouncements about Twitter’s rules. In appointing himself “chief twit,” Mr. Musk has made clear that at the end of the day, he’ll be the one calling the shots.
It was for this reason that I chose to leave the company: A Twitter whose policies are defined by edict has little need for a trust and safety function dedicated to its principled development.
So where will Twitter go from here? Some of the company’s decisions in the weeks and months to come, like the near certainty of allowing Mr. Trump’s account back on the service, will have an immediate, perceptible impact. But to truly understand the shape of Twitter going forward, I’d encourage looking not just at the choices the company makes but also at how Mr. Musk makes them. Should the moderation council materialize, will it represent more than just the loudest, predominantly American voices complaining about censorship — including, critically, the approximately 80 percent of Twitter users who reside outside the United States? Will the company continue to invest in features like Community Notes, which brings Twitter users into the work of platform governance? Will Mr. Musk’s tweets announcing policy changes become less frequent and abrupt?
In the longer term, the moderating influences of advertisers, regulators and, most critically of all, app stores may be welcome for those of us hoping to avoid an escalation in the volume of dangerous speech online. Twitter will have to balance its new owner’s goals against the practical realities of life on Apple’s and Google’s internet — no easy task for the employees who have chosen to remain. And as I departed the company, the calls from the app review teams had already begun.
More on Twitter
Resignations Roil Twitter as Elon Musk Tries Persuading Some Workers to Stay
Nov. 17, 2022
Opinion | Farhad Manjoo
Elon Musk Has No Idea What He’s Doing at Twitter
Nov. 10, 2022
Two Weeks of Chaos: Inside Elon Musk’s Takeover of Twitter
Nov. 11, 2022
Yoel Roth (@yoyoel) was the head of trust and safety at Twitter, where he spent seven years directing the company’s policy and enforcement work on abuse, election security and anti-spam issues.
A version of this article appears in print on Nov. 19, 2022, Section A, Page 23 of the New York edition with the headline: I Led the Trust and Safety Teams at Twitter. Here’s What Could Happen Next..
No comments:
Post a Comment