synonymous

Denise made a thread on Twitter about OTW's failure with regard to Trust and Safety here. It has been compiled into a post on Dreamwidth for easier reading here, and cloned here for documentation purposes.

A compilation of my Twitter thread about the OTW's absolute failure at Trust and Safety

(The following is a compilation of a thread I made this morning to my Twitter account in one place for easy reading. The original thread begins here. It has been lightly edited for formatting and to recombine several ideas I had to split over several tweets.) I've bolded what I think are the most important conclusions.

Warning: This entire entry discusses many cases of egregious abuse, including distribution of CSAM (child sex abuse material) and the systemic failure of the organization to respond to CSAM in accordance with the law.

I will continue to edit the compilation if I add to the thread on Twitter. (Last edited 6/15 4PM EST)

*

In today's "horrifying discoveries about how badly AO3 is diverging from basic, fundamental Trust and Safety best practices" discoveries: suspending an account doesn't make that account's content invisible, PAC has to review every individual piece of it: https://archiveofourown.org/comments/660613498. This is, to put it extremely mildly, an absolutely fucking bananapants paradigm that is placing an unconscionable workload burden on the team *and* yet another potential vector of massive liability for the organization as a whole re: material unprotected by §230 such as CSAM.

At this point I will say it plainly: OTW's Legal committee is fundamentally and egregiously unqualified to be setting policy and process for Trust and Safety work, and their doing so while remaining actively ignorant of best practices is an existential threat to the organization.

My personal receipts: during the period in May of 2022 when OTW volunteers were being emailed CSAM images, several people I know who had received those emails reached out to me for advice before OTW made the situation public.

Several of those emails included threats to expose OTW volunteer information, including emails, pseuds, and wallet names when people had made them available or where they were discoverable, to groups such as Kiwi Farms, 4chan, and QAnon. These groups very, very frequently use SWATting tactics, in which they utilize TDD relay and other phone spoofing tactics to appear to be coming from a local phone number and pretend to be a resident at someone's address in the middle of a violent and dangerous situation. The police, believing the inhabitants of the address to be under threat, dispatch the SWAT team to invade the home. I will forego a wider explanation; you can find them in lots of places.

The absolute first thing anyone should do if someone threatens you with these groups is to call your local police department and make them aware that there is an elevated risk of SWAT attempts to your address. It's variably effective: some PDs have a process, some don't.

The risk to you from the police in calling the non-emergency line and asking if they have a process for flagging your address as a SWATting risk is almost nothing, and it is significantly lower than the risk of actually being SWATted.

When the OTW did finally make a public announcement about the incident, I confirmed with the people who I'd spoken to that they had not provided their volunteers with any information on personal safety, including but not limited to advising vols about anti-SWAT protection. So I did it myself, in this thread: (Link to Twitter thread from 2022) The "in the last 14 years" was because at the time there was no publicly available information about what vector the email attacker had used to gather their database of OTW volunteers, and with SWAT risk, it is better to be over-cautious than under-cautious.

I also reached out via website contact form, as you can see me saying in that thread, to offer my help with a) seeing if there were any technical measures that could be taken to quarantine any future email attacks so they wouldn't be sent to volunteers and b) placing them in contact with someone I know on the board at NCMEC, the NGO that handles online CSAM, to see if NCMEC could offer them any additional help in either identifying the source of the images being emailed or with any further technical blocking measures. (I also wanted to make sure I shared the tip about using Tetris to reduce the impact of visually traumatizing material, because it's not as much of a miracle cure as pop science has it but there is some benefit. Everyone I'd talked to was, understandably, very shaken by it.)

Shortly after I reached out, Rebecca Tushnet of OTW's Legal committee emailed me and asked for a phone call to discuss the matter. I believed that phone call would be to discuss the actions OTW had taken with law enforcement to see if I had any additional suggestions. Instead, Rebecca spent half an hour using extremely high-pressure tactics to demand I remove that thread I linked to above. The following tweets are her talking points paraphrased from the contemporaneous notes I made memorializing the conversation immediately after:

1) There was no SWAT risk to OTW volunteers, because OTW doesn't keep address information about volunteers and therefore there is no way the attacker could have discovered it no matter what access they had to OTW systems;
2) The existence of that thread was frightening volunteers and making the OTW look bad;
3) No one should be advising anyone to call the police because many OTW volunteers are marginalized people and it is dangerous for them to call the police.

My response to those points were:

1) While my initial rapid-response thread didn't make it explicitly clear, by the time Rebecca called me I had added the followup six tweets explaining that the risk was not from the records OTW had on volunteers but in the fact that unless someone is very well trained in OPSEC, it is trivially easy to find the physical address of any individual online based on nothing more than their pseud, their social media, and offhanded details they've given about their life. I can do it. A lot of people can do it. Kiwi Farms, one of the three groups the attacker had explicitly threatened to involve in the incident, does it over their morning coffee as a fun and exciting warmup to start the day.

2) It is, in fact, a reasonable and rational response for someone to be frightened when someone commits multiple international felonies in emailing you CSAM, plus threatening to dox you to people who think SWATting someone is a fun and invigorating method of debate. And, in fact, the organization's complete stonewalling in failing to provide any form of incident response to individual volunteers -- both the ones who had been exposed to CSAM and the ones who were terrified that they would be exposed to CSAM in a future wave -- was the source of significantly more fear, uncertainty, and upset than any amount of practical risk mitigation advice I could provide. The fact I was advising OTW volunteers to treat the incident as gravely as I did was because it was, in fact, a very serious incident.

3) It is completely and objectively legitimate for marginalized people to fear interactions with the police. This is why it is MORE important for marginalized people at elevated risk of SWATting to proactively contact the police to alert them to the elevated SWAT risk. If it is dangerous for a marginalized person to call the police to warn them they're at elevated risk of SWATting, it is SIGNIFICANTLY more dangerous for those marginalized people to have the cops kick down their door with guns out ready to shoot them.

I told Rebecca that I would not be deleting that thread. I said I'd be happy to add any clarifications about point 1 above that she felt I hadn't adequately covered in my addendum, but for her to pressure me to delete it was inappropriate, particuarly given that it was the only safety advice the OTW volunteers had received up until that point. I offered to put her in contact with one of the organizations I know of that does incident response to help craft an organizational response with a safety toolkit.

Her response to that was to ignore it and continue her pressure tactics, which I finally cut off with "Rebecca, it would be a violation of my professional ethics to delete that thread, and quite frankly, for you to be pressuring me this hard to do so is a violation of yours."

There was a very long pause and she said "well, I guess we'll just have to agree to disagree, then."

To the best of the knowledge available to me, from discussion with OTW volunteers, the organization still has provided zero resources, advice, or safety information to volunteers who were emailed CSAM in May of 2022.

James, the OTW's lead SRE, did contact me after that conversation to follow up on my offer to provide resources for steps the organization could take to prevent future attacks. Unfortunately, I discovered at that time the advice I was going to give on quarantining and filtering email at the organization level was not going to be helpful, because the org volunteers did not do org business through an org email server but through their personal emails.

At that point I returned to my advice from the original thread and suggested the organization distribute instructions on how to disable auto-loading of images in the most common email clients and webmail systems. I offered to help compose that advice. I did not receive a response to that offer. To the best of the knowledge available to me, from discussion with OTW volunteers, the org provided no resources to volunteers on how to disable images in email clients and webmail systems in case of future attack.

To the best of the knowledge available to me, from discussion with OTW volunteers, the org did not advise volunteers who had received the CSAM email payload to abandon their email addresses or offer any help in doing so.

This tweet is conjecture informed by the OTW Board and Legal response to the recent revelations about this incident: I believe the reason they did not treat this incident with the severity it deserved because they (falsely) believed they already knew who did it.

The source for the conjecture in the previous tweet is the message written by the OTW Board and Legal committee and sent to volunteers on 5/30/23 that is documented (along with followup) here: https://synonymous.dreamwidth.org/2244.html

I believe any reasonable reader would interpret this message as an explicit accusation that the person whistleblowing about OTW's egregious mishandling of Trust and Safety was the person responsible for the emailed CSAM attacks.

I believe any reasonable reader would interpret the addition of the "maybe their account was compromised" to that message is a deliberate and calculated figleaf included solely to provide the organization a defense against defamation charges. Particularly in the context of the contemporaneous actions taken by the OTW at the time, in which they neither informed the whistleblower of any suspicion of account compromise nor required any account resecuring steps to restore access months later, this figleaf is obvious.

I believe any reasonable reader would interpret "After May 7, no more attack emails were sent to anyone. This is another coincidence we cannot explain" as an explicit accusation, particularly at the end of a paragraph about disabling the whistleblower's account.

To the best of the knowledge available to me, from discussion with OTW volunteers, this is the only followup statement volunteers have been given about the incident.

As it happens, I do know the person who is being accused here -- she was a volunteer on the LJ abuse team quite some time ago, although we haven't been in contact for like 15 years -- and I don't believe she's capable of having done something this heinous.

That having been said, even if you don't know her -- hell, even if she were the attacker, which I absolutely don't believe -- there is no universe in which it's acceptable to make this public statement about an incident that involved the commission of ~900 separate felonies.

If the person you're accusing isn't the attacker, you've just defamed them to your entire organization. If the person you're accusing is the attacker, you've just let them know they are under suspicion and interfered with any law enforcement investigation of the incident.

That portion of the message sent to the entirety of the OTW volunteer base was written by a member of the OTW Legal committee. I do not know the individual person who authored it.

Whoever authored that message is a fundamental existential threat to the organization, because it displays judgement so bad that, in my opinion, it approaches the level of professional malpractice.

The whistleblower has made additional statements about the directives PAC (the AO3 T&S team) were given regarding sexually graphic visual content embedded in works posted to AO3 where the visual content was of uncertain provenance and the models were unidentified.

These statements were made on the DW anonymous discussion community fail_fandomanon, but I have confirmed with her she was the author of this anonymous comment. Content warning: CSAM, child sex abuse

On professional training, knowledge, and belief, the scenario in item 2 of that comment, however terrible or obvious the Photoshop job is, qualifies under the third definition of "child pornography" as given in 18 USC §2256(8)(C).

There is absolutely no scenario in which a PAC volunteer should be forced to examine an image that has been "created, adapted, or modified to appear that an identifiable minor is engaging in sexually explicit conduct" to determine if it's a policy violation. If the image is an identifiable minor, and the image is of them engaging in sexually explicit conduct, it is considered "child pornography" as the term is used in 18 USC §2252A and is governed by 18 USC 2258(A). 18 USC 2258(A) covers the obligations of a user-generated content website to report certain content to NCMEC.

Let me be absolutely clear: I have no information that leads me to believe OTW Legal failed in their legally mandated duty to report content to NCMEC. From inspecting the AO3 code I do, however, believe the AO3 software is technically incapable of meeting the OTW's legal obligations under 18 USC 2258(A)(h) regarding preservation of information, specifically subsections (2) and (3).

As an example: reading the code around orphaned works makes it clear information is not retained for the 90 day period required by law. Until very recently, the only information retained about an orphaned work was certain IP information. (As of April (verifiable in the code and the public OTW bug tracker at https://otwarchive.atlassian.net/browse/AO3-5521), the userid of the user who posted an orphaned work is now retained for 72 hours, but 72 hours is still less than 90 days.)

This creates a scenario in which someone can orphan a work that contains content reportable under 18 USC 2258(A) past such time as OTW has a legal duty under 18 USC 2258(A)(h) to retain all information about that communication, and the AO3 software will destroy that information.

You now all have enough information to understand the full extent of my horror in discovering the information I started this thread with: the fact suspending an AO3 account doesn't take down and preserve account content means it is impossible to comply with 18 USC 2258(A)(h).

18 USC 2252(A) is not the only category of content that must be handled as specified in 18 USC 2258(A). A user-generated content website is legally required to also take down and preserve:

1). content that "advertises, promotes, presents, distributes" images or video of a minor engaged in sexually explicit conduct available for sale, anywhere else on the internet (18 USC 2252A)

2). content that "distributes, offers, sends, or provides", to a minor, images or video of a minor engaged in sexually explicit conduct, "for purposes of inducing or persuading a minor to participate in any activity that is illegal" (also 18 USC 2252A). This covers, for instance, someone sending CSAM to a minor and saying "now send me a picture of you back".

3) evidence of someone "produc[ing] with intent to distribute [...] by any means, including a computer, in or affecting interstate or foreign commerce, child pornography that is an adapted or modified depiction of an identifiable minor" (still on 18 USC 2252A). This means the act of someone Photoshopping the head of an identifiable minor on the body of a porn performer is itself a reportable act under 18 USC 2258(A) independent of the existence of the resulting image.

4) someone attempting to buy, sell, or transfer custody of a minor for the purposes of creating a visual depiction of sexually explicit conduct, or assisting others in doing the same (18 USC 2251A)

There are several other scenarios that involve mandated reporting and preservation, but those are ones that are less likely (IMO) to have ever come up on AO3. But I can easily imagine content on AO3 that meets all four of the above scenarios. (If I'd been thinking before I started this thread, I would have gone and found some examples -- I am reasonably sure I could turn them up in a few hours of searching -- but I was too angry and I started it without doing the prior research, heh.)

The fact the AO3 archive software makes it technically and factually impossible for OTW the organization to comply with their legally mandated reporting and preservation obligations under 18 USC 2258(A) is an existential threat to the organization.

Any lawyer representing the org since 2008 who failed to recognize that OTW is technically and factually incapable of complying with their legally mandated reporting and preservation obligations under 18 USC 2258(A) is incompetent to be advising the organization.

The fact OTW is technically and factually incapable of complying with their legally mandated reporting and preservation obligations under 18 USC 2258(A) means the organization is not covered by the liability shield of 18 USC 2258(B).

On professional training, knowledge, and belief, the implementation details about how to comply with the reporting and preservation requirements of 18 USC 2258(A) are so widely available and well-known in the Trust and Safety field that a failure to employ them rises to the "reckless disregard" standard set forth in 18 USC 2258B(b)(2)(B). These are not obscure details. There is no room for good-faith disagreement in implementations. This is absolute black-letter law.

The examples I've given in this thread of how an AO3 user could destroy material the OTW has an affirmative obligation to preserve under 2258(A)(h) are only a sample. From reading the code, there are a number of others. I am only providing a very high-level overview.

My concern is that if I continue to provide examples of how the OTW is technically incapable of meeting its obligations regarding CSAM on the archive, it will make those tactics more widely used. I have this concern because PAC, the AO3's Trust and Safety team, historically has been unable to act in a timely fashion to handle novel abuse that is happening on AO3 because they are not permitted to handle novel scenarios without permission from Legal.

I know of no other user-generated content site where the Trust and Safety team is forbidden to act on novel scenarios without permission from the organization's lawyers. You can easily verify how far a deviation this is from industry standard by going to the job board of the TSPA, the Trust and Safety Professional Association, and checking the job listings for a selection of listings for "the person who sets the T&S policies".

I didn't check today, but the last time I checked was last week, and at the time I verified not one of the T&S policy lead jobs were even marked as "JD preferred", let alone required the applicant to be a practicing lawyer.

This is because the idea that a T&S team cannot react to novel scenarios without permission from an organization's lawyers is, to put it frankly, absolutely fucking bugfuck. Lawyers should not set enforcement policies. The Trust and Safety team should.

The fact OTW prohibits PAC from responding to novel scenarios without direction from OTW Legal is not only so far outside industry standard that any experienced T&S person would boggle, it is the cause of every single delay AO3 is experiencing in effectively handling abuse.

In my considered professional opinion, as someone who has been doing precisely this work in a very similar environment since 2002, the OTW's approach to Trust and Safety is incompetent, dangerous, and an existential threat to the organization and to the archive.

I want to be very clear this is not the fault of members of PAC, the people doing the actual T&S work. The fault lies solely with the OTW Legal committee, who are out of their depths and over their heads, and with the OTW Board who has allowed them to control the organization.

To address what seems to be a common objection to criticism inside the org, let me also be clear: none of what I'm talking about is the ridiculous fucking bullshit some people call abuse, like "the wrong character tops" or "these characters grew up together so it's incest".

I stand by this thread I made in 2020 about how calls for AO3 to arbitrate whether or not a fictional work contains racist depictions is the wrong ask and how implementing that system would lead to more issues for marginalized people, not fewer: (link to Twitter thread of June 18, 2020)

In addition to the commentary I made at the time, I will add, in light of what has been revealed in the last few months, that asking PAC to determine whether or not a particular work contains depictions of racism is impossible. The organizational dysfunction that is preventing PAC from handling such basic, black-letter scenarios as "an image created, adapted, or modified to appear that an identifiable minor is engaging in sexually explicit conduct" cannot ever effectively moderate for racist content.

When I talk about PAC being prevented from handling novel scenarios, I am talking about conduct, not content. I appreciate the work of the EndOTWRacism campaign to raise awareness of the OTW's organizational dysfunction. I firmly disagree with the foundational work their campaign was built upon and many of the arguments they have made. [note from synonymous: link here is my own, as Denise has expounded upon this talking point in the linked post, but it is not relevant to the goals of this journal]

Also, while I was composing all this, the official account for AO3 Policy and Abuse confirmed in the comments to their news post regarding this that they do not, in fact, comply with the recordkeeping and preservation requirements of 18 USC 2258(A)(h). https://archiveofourown.org/comments/660837025

I do not blame the PAC staffer who wrote that comment for not knowing the law requires them to disable access to but preserve CSAM posted to the archive. I blame Legal for not knowing federal fucking law.

Later addition:

wait a fucking minute I just noticed something else later in the thread of that comment I linked to in the threadstarter that's completely fucked up, I was too busy boggling over the other shit

-- okay sorry I was checking a cite on the part of this I was fuzzier on, but: the comment later in the thread I was referring to is https://archiveofourown.org/comments/660618574 and the issue is this: "We weren't allowed to delete content uploaded by an Age-Barred Individual, but only suspend them until they turned 13."

This means AO3 is knowingly violating COPPA by retaining, at the very least, screen name and email address of users under the age of 13. Fanworks uploaded by "age-barred individuals" are not necessarily "personal information" under COPPA (although they could contain it) but there's no question they're retaining, at minimum, screen name and email address (and probably IP information, too). I was doublechecking GDPR requirements for age-gating but I gave up; I think this violates those as well, though.

The penalty for violations of COPPA are civil; it is a fine up to $43,280 per instance of personally identifying information knowingly retained about a US user under the age of 13. GDPR fines are extremely discretionary but the statute provides for "20m EUR or up to 4% of the total worldwide annual turnover of the preceding financial year, whichever is higher".

Later later addition:

Addendum: someone on the DW copy of this reminded me the OTW is not one of the categories of nonprofit the FTC Act applies to, so please disregard the statements about COPPA above: AO3 is not subject to the law. (It is still a problem their process and their docs don't agree.)

(I am so used to dealing with the rule on COPPA as "there are enough cases where it can apply to nonprofits that it's better to be safe than sorry" that I forgot to check the specifics for OTW in particular!)

Update 1: Additional commentary from azarias on the AO3 post

(link)

I recommend everyone concerned about the OTW's handling of CSEM read this twitter thread. Experienced trust and safety professional Rahaeli has shared how Rebecca Tushnet, OTW Legal Committee member and former head of Content, attempted to pressure her into removing safety advice for volunteers who had been exposed to CSEM and threatened with doxxing.

Rebecca's concern seemed to be preserving the image of the OTW. In addition to being a leading member of the org and a distinguished law professor, Rebecca is also the person most responsible - but not solely responsible - for mandating PAC's negligent handling of CSEM and other severely abusive content. Her inappropriate exercise of power over PAC has endangered volunteers, Archive users, and the OTW itself, and I'm shocked but unsurprised to learn that she's gone even further than I knew.

The Board of Directors, past and present, is also responsible for enabling Rebecca Tushnet's behavior, as are at least some of her colleagues on the Legal Committee, especially Legal Chair Betsy Rosenblatt.

(link)

I am not a legal expert and cannot confirm her interpretation of the relevant laws. I can confirm that she's right about the Archive software's inability to comply with many of the requirements she lists. Furthermore, PAC did not have any internal policies whatsoever on data retention in CSEM cases while I was a member. We also did repeatedly rule that photoshopped pornographic images of real and identifiable minors were allowed on the website, per guidance from the Legal committee.

I asked for more specific guidance on topics such as data retention numerous times. To the best of my knowledge, those guidance requests were forwarded to Legal. I did not receive any answers. Because I was not allowed to communicate with Legal directly, I can't say for sure if they were never answered, or if they were answered but a communications breakdown occurred before I could receive them.

Update 2: Relevant FFA commentary and addendums on aspects of the thread

(link)

I know this is a small detail, but "the org volunteers did not do org business through an org email server but through their personal emails" is stunning in its own way.

It's so basic? Even the little anime con I volunteered for in 2006, that had thirty-something volunteers, had an email server that *forwarded* mail to the appropriate volunteers' personal inboxes.

...

With a centralized email server, admins can set up whitelisting and filtering to do things like automatically not load images, filter out unknown addresses, flag email coming from an external server (meaning nonny@random-throwaway.com can't get through, nonny@transformativeworks.org can), and quarantine all email for a period of time.

Modern email providers like gmail can also do a decent chunk of that for you. But each volunteer would have to know about it, and do it themselves, before the spammer got to them. And then of course there's the issue of your personal email being linked to other stuff you do online, which makes it easier to connect you to the org, easier to doxx for your work in the org, and so on.

So, yes, they endangered volunteers by not doing this. Setting up an email server for 1000 people is not more difficult or expensive than running AO3. It's inexcusable that they didn't have one as of 2022.

We have actually switched over to centralized stuff partly as a result of that. They have not required all volunteers whose AO3 accounts are inextricably linked to being able to do their work (at least tag wrangling, I don't know about anyone else) to switch their regular AO3 accounts to have the official volunteer email, so my kudos emails can still get to me-as-fan not me-as-volunteer, but all the official org stuff is supposed to route through our spiffy new transformativeworks.org emails.

(link) azarias (similar to their commentary here):

We had no data retention policies regarding CSEM whatsoever, and I can confirm AO3 software was incapable of fulfilling some of the requirements Denise cites.

I am not a legal expert and can't confirm Denise's interpretation of the relevant laws, however PAC did repeatedly ask Legal for guidance on these topics and did not receive any. Nor are any Legal committee members experts in the relevant areas of law, to my knowledge.