A peek into Reddit's anti-spam internals

5 years ago, back when I still used Reddit, something unusual happened. My app of choice, Relay for reddit, was bombarding me with a bunch of weird notifications about removed spam. Getting these notifications wasn’t unusual in and of itself - I was a moderator of a few fairly small subreddits that’d from time to time get posts automatically removed for spam. However, when I went to actually look at the removed spam, I saw something I was never meant to see. I saw Reddit’s anti-spam internals. so that's about it. Removed: spamurai (*Removing potential spam content from unproved user*: comment `t1_pupp13` (0.7294469 perspective spam) by u/GoodBoyBacon (0.06 days old, spammy: 11, hosted: false, -1 karma, 4 reports, org: `ComcastCable`, email: gmail.com) in r/GoodBoysOnly (guest) posting nil from `oauth.reddit.com` via `nil` from UA: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36 Edg/95.0.1020.30, RHS: oc:ac:kT:lw:bV:aX:af:a6:l5:y3:aT:m9:pt:f3:hZ:az:aR:aQ, LANG: en-US,en,q=0.9, TLS: j7bXVc3l/qer8FRj2aEiqOrx1ro=DDZ0TViWlY5HYgOPw1SZqDxwiO8= - referrer: https://www.reddit.com/, thumbnail: `` - ) • GoodBoyBacon • 1 points • 27 min You see u/BadGuy67?

He's the same guy as https://www.reddit.com/r/ReallyBadGuys/comments/qw3rt1/if_ur_a_bad_guy_post_here_please/ Removed: Reddit (shadowban applied on 10-27-2021) • GoodBoyBacon • 0 points • 1 hr I'm not the same guy as that other guy please read my comment How Reddit moderation works So Reddit is a site comprising of smaller sub-communities, which are called subreddits. For example, /r/mylittlepony is a subreddit for fans of My Little Pony. These subreddits can be created by anyone, and they are moderated by a group of community moderators appointed by the creator of the subreddit. If we go1 on /r/mylittlepony we can see the list of moderators on the sidebar: MODERATORS MESSAGE THE MODS message sent. Orschmann optimistic_outcome Chinch335 IllusionOf_Integrity spokesthebrony TheeLinker Lankygit Raging_Mouse Searchbar_Trixie gbeaudette ...and ...and These moderators can remove posts, ban users, manage modmail etc, but they are just normal Reddit users. If you’re a moderator you can see who removed a post or comment: Removed: rebane2001 • ExampleUser • 1 points • 1 hr I'm breaking the rules 😈 This includes the automod - a rules-based moderation system: Removed: AutoModerator • ExampleUser • 1 points • 1 hr bad word But then you’ll sometimes also see the mysterious “Auto”: Removed: Auto • ExampleUser • 1 points • 1 hr hi This is what happens when something gets caught in Reddit’s mysterious spam filters, or when Reddit’s sitewide admins remove something manually. In the moderator log, they’ll show up as “reddit” and “Anti-Evil Operations”: ExampleSubreddit: moderation log filter by action: filter by moderator: You cannot be reasoned with.

13 days ago Anti-Evil Operations removed link "[ Removed by Reddit ]" by EvilPoster 27 days ago Anti-Evil Operations removed link "[ Removed by Reddit ]" by Rule_Breaker1337 1 month ago reddit removed link "buy my shirt" by xXx_ShirtSeller_xXx 1 month ago reddit removed link "sexy ladies" by SpamBot_00018341 2 months ago Anti-Evil Operations removed link "[ Removed by Reddit ]" by PuppyGirlHater These sitewide spam removals is what the rest of this post is going to be about. Oopsie What happened to me back in 2021 was that due to some kind of an error on Reddit’s side, the usual Removed: Auto text had been replaced with the actual removal reason. Why this happened to me I do not know - it returned back to normal after an hour or so. All I was left with was a bunch of screenshots I managed to take while this stuff was still going on. But that doesn’t mean we can’t speculate! Up until 2017, Reddit’s source code was publicly available. Of course, a lot has changed since then, but we can still analyze the archived code and hypothesize what might be happening. The function responsible for moderator removals is POST_remove: def POST_remove(self, thing, spam): """Remove a link, comment, or modmail message.""" ... admintools.spam(thing, auto=False, moderator_banned=not c.user_is_admin, banner=c.user.name, train_spam=train_spam) We can see it calls admintools.spam with a few arguments, notably: moderator_banned, which marks whether something was removed by a moderator or an admin, and banner, which notes down the username of whoever did the ban action.

Poking around a bit more, we find the get_mod_attributes function: # Comments added by me for the blogpost def get_mod_attributes(item): data = {} # If user is logged in and a moderator if c.user_is_loggedin and item.can_ban: data["num_reports"] = item.reported data["report_reasons"] = Report.get_reasons(item) ban_info = getattr(item, "ban_info", {}) # If post was removed if item._spam: data["approved_by"] = None # If post was removed by a mod if ban_info.get('moderator_banned'): # Show the banner name data["banned_by"] = ban_info.get("banner") else: # else, if post was removed by an admin # Hide the banner name data["banned_by"] = True else: data["approved_by"] = ban_info.get("unbanner") data["banned_by"] = None else: data["num_reports"] = None data["report_reasons"] = None data["approved_by"] = None data["banned_by"] = None return data This is the part of the API that actually returns us the information about removals - the banner in ban_info is the red text I was seeing Relay. And it seems like it will only get returned if the removal was by a moderator, not an admin. But where does that Auto text come from? Reddit’s API only returns an actual username, or True. Turns out that it’s actually coming from Relay2 itself: // reddit/news/oauth/reddit/model/base/RedditLinkComment.java if (this.bannedBy.equalsIgnoreCase("true")) { this.bannedBy = "Auto"; } else if (this.bannedBy.equalsIgnoreCase("null")) { this.bannedBy = ""; } Okay, that explains that. But where am I getting these internal messages from? Well, it seems like Reddit is re-using the banner field for internal removal reasons: def POST_submit(self, form, jquery, url, selftext, kind, title, sr, extension, sendreplies, resubmit): """Submit a link to a subreddit.""" ...

if not is_self: ban = is_banned_domain(url) if ban: g.stats.simple_event('spam.domainban.link_url') admintools.spam(l, banner = "domain (%s)" % ban.banmsg) hooks.get_hook('banned_domain.submit').call(item=l, url=url, ban=ban) The above code snippet runs whenever a new link is posted. It checks whether the domain is spam, and if it is it removes it with the banner set to “domain (REASON)”. We can see it in action with this removed post for example: 1I_EAT_PONIES in MyLittleOutOfContext Conga! Removed: domain (banned as an experiment to see what happens with tubmlr spam ring. - em 5/31/12) • 0 Comments • 24.media.tumblr.com • 9 yrs Seems like em was playing around with auto-removing all tubmlr [sic] links on Reddit in 2012? Anyways, it seems like Reddit is stuffing its internal spam removal reasons in the banner field, but making it so that only sitewide admins can see them. And something in a codepath similar to get_mod_attributes was broken for a couple hours, allowing me to see those reasons. Let’s take a look at the kinds of reasons I managed to get a glimpse of! domain (2012 - present) The first category is the domain removals, as shown earlier. Nearly all of these are just Removed: domain (spam), though I did find this gem in there: 1presafur in MyLittleOutOfContext Just register and look for me here* h9OI5WUQZPL Removed: domain (le sexxxxy sex spam) • 0 Comments • www.example.com • 5 yrs Perhaps I’m just childish, but I find the idea of someone going le sexxxxy sex spam while working on a spamfilter rather amusing. Reddit probably had some issues with Tumblr spam, because in addition to the tubmlr removal we saw earlier there was also this: 1JackofH3art in MyLittleOutOfContext It hurts so good.

Removed: domain (ban - 11/12/12 mg ) • NSFW • 0 Comments • bartl3by.tumblr.com • 8 yrs I’m quite certain that this removal was targeted at Tumblr in general, and not the specific blog linked, since bartl3by.tumblr.com seems to be a legitimate (although somewhat perverse) blog. I believe domain removals are the only type of anti-spam we can actually see in the public Reddit source code. Though, even that is partially hidden. spammit (2012 - present) The next category is spammit, which somehow analyses a post and gives it a percentage rating: 1Kyderra in MyLittleOutOfContext I'm very fondling of you Removed: spammit(72.98% spammy) • 0 Comments • dashie.mylittlefacewhen.com • 8 yrs Yes, there’s no space between spammit and the parenthesis. The percentages of removed posts were generally fairly high, with the lowest one being 39.71% spammy and highest one 98.19%. That being said, spammit doesn’t seem like a very accurate anti-spam measure for the subs I moderate because it seemed to hit a lot of legitimate Imgur posts with a 70-98% spammy rating. bans (2016 - present) Next, we have post removals for banned users. 1kaitlynwwrettin in MyLittleOutOfContext Cost Reduction & Cost Saving Consultants | Puppygirl Consulting Removed: banned user • 0 Comments • www.example.com • 3 yrs Some of them are marked with just a Removed: banned user, though others get a fancy Removed: Reddit (banall performed). 1KerryVinebt403 in MyLittleOutOfContext casino online Removed: Reddit (banall performed) • 0 Comments • example.com • 3 yrs The posts I saw being removed like this were all very obvious spam. Mostly just ads for all kinds of services. I suspect this is the admins seeing an obvious spambot and just nuking it from orbit.

shadowbans (2016 - present) It’s known that Reddit shadowbans its users. A shadowban is a silent ban where seemingly nothing happens to your account and you’re still able to post/comment, but nobody else will be able to see your posts and comments. In fact, there’s even a subreddit for checking whether you’re shadowbanned. But now we can actually see what a shadowban looks like to admins: 1pickertramontana in trixiemasterrace Blonde Teen Takes A Massive Meow In Her Bark Removed: Reddit (shadowban applied on 11-06-2019) • 0 Comments • self.trixiemasterrace • 1 yr I’m not going to share the specific conversation here, but there was a really funny comments thread going on where a person was blaming mods for removing all of their comments while in reality being shadowbanned by Reddit. spamurai (2020 - present) Now we get to the most interesting part of the entire spam filter thing. Unlike spammit, spamurai is a system that does have some public references to it. According to slide #28, Reddit uses Minsky for “ML”, and Spamurai for “Rules”. I’m not sure how this is calculated into the removal reasons, so I’m just going to ignore it and assume everything is spamurai. First up, there seems to be some sort of a spamurai subsystem called echelon. It seems to remove certain keywords, such as the EqG elsagate spam seen below, and lewd (OF? Snapchat?) stuff like puppyvids.69. 1mypham71375 in Pony_irl Equestria Girls Princess Animation Series - Twilight Sparkle Cutie Mark ... Removed: spamurai (echelon: Equestria Girls Princess Animation Series) • 1 Comments • youtube.com • 2 yrs Then, there’s some targeted removals, such as this one that targets clothing spam. Removed: spamurai (approval required on hyperlink comment from high spam score account (suspected shirt affiliate spam)) • Adventurous-ties • 1 points • 5 months Dog-Women consulting company will open the Ukrainian pharmaceutical market for you!