The naughty username checking system used by Twitch (opens in new tab)

> to make sure random base64 didn't contain profanity

I would have said "why bother" until this happened to us.

A customer rang us up in a fury because some demo/ random data that we generated happened to have the word "penis" in it. They were convinced we must have put it there because we thought he was a cock. It was very difficult to defuse the situation.

When it comes to censoring randomly generated strings, I like simply to omit vowels from the alphabet. Usually I'll omit some of the more obvious lookalikes too, e.g. [1 0 v].

It's a simple solution. Sure, it is still possible for something to slip through that looks similar to something bad. But the potential to strongly offend is greatly reduced.

It’s so laughable that we care about whether a generated string contains some temporally relevant profanity. We truly are still barbarians, and will be viewed as such by history.

We did a similar thing at Groupon after a customer’s coupon code contained an F bomb.

Hashids (https://hashids.org/#how-does-it-work) have a pretty clever trick for this. They’re able to encode multiple IDs to a single obfuscated hash, which works by reserving some characters from the alphabet to use as a separator between each encoded value. That guarantees that whatever characters you choose to be separators are never next to each other in the output. By default their separators are (lower + upper case) “c, s, f, h, u, i, t”

It worked surprisingly well when we used it.

>the first one that popped up during the demo was a big ole F bomb

And this, ladies and gentlemen, is what it would show BEFORE the filter... but after (runs the code again, and prays it works) ... NO PROFANITY!

In my help desk days I took a call from an irate person ranting about how we were telling her to "get a male sex change". Eventually I figured out she had become upset with "msexchange" showing up in the address!

How statistically likely is this? Can't you just regenerate until no you have a suitable short URL. Aside from performance, this is as random as you can be. Or generate the characters one by one and backtrack, this requires less random days. Or regenerate the unwanted substring.

Fascinating! I guess an easy solution is to inject non alpha characters into any generated string. I imagine a constraint was that you wanted them to be easy to type?

People should just be taught (I would consider this is an essential skill which should be put in basic school curriculum to psychologically prepare people for the life, many support/sales professionals learn it quickly but the rest of the people don't) words are just words and they don't have to take them seriously. No word is rude until you consider it rude. No word can actually harm you directly. It's you who is in charge of deciding what to feel whenever you hear/see a particular word. Sadly, most of the people just ignore this job. I personally enjoy being called by the derogatory nickname of my ethnicity because I just like how does it sound. First half of my life I didn't even know it was supposed to be insulting (as I didn't know actual racists exist and can be serious about hating/judging people by just their ethnicity).

It's getting particularly problematic for international sites. What constitutes offensive language in one country might be a perfectly normal name, or product, in another country that uses the same language, let alone in another language.

There's been a running joke on the UK Reddit subs recently about people getting short bans for using the term "faggot" as it's now on some automatic, site-wide blocklist. The bans are completely context free, so people are supposedly getting banned for discussing the food product, a kind of large meat-ball made primarily from minced offal. The same happens with "fag", meaning a cigarette.

That this system is so primitive doesn't surprise me at all. After all, people with a family name "Null" often get database errors -- despite even Oracle, which can't tell null and empty string apart, decidedly has IS NULL vs == "NULL" comparisons... https://www.bbc.com/future/article/20160325-the-names-that-b...

Its a similar problem to people having names that are considered "fake". I shudder what people actually named Harry Potter or James Bond have to go through regularly.

Also, I dont see any sense in actually enforcing any of those lists. I always think about the "Journey of Life" in The Grand Tour, which, frankly, is completely inoffensive in any other language than english https://www.youtube.com/watch?v=BLXe2WTYngQ

Seems like a rude name might give you some privacy advantages in the 21st century. To the price of being excluded by random services maybe.

Old math prof of mine was named Dr. Cock, and he wasn't the only one, the name was common. Students called him Dr. Octocock.

Random story I remember my ICT teacher telling us in school along the same lines as Scunthorpe. They installed some nonsense web blocking thing that schools like to have and a lot of teachers complained because it blocked weightwatchers.com

I only recently learned about Scunthorpe because the username I've been using for 15 years is now affected.

Georgyo has orgy right in the name. More and more places are refusing to accept it when signing up.

> We used to call this the "Scunthorpe problem"

Fwiw, it's well-known as such, after relatively high-profile incidents involving people with that address: https://en.wikipedia.org/wiki/Scunthorpe_problem

The city of Toppenish in the US had the same problem when the council turned on a generic filtering system for the city networks. Everything stopped working.

Haha, I call this the "Clbuttic" problem. But nowadays it can be solved with machine learning fairly easy https://moderationapi.com/blog/moderate-text-automatically-u...

If nobody ever implemented a rude word filter, nobody would ever need to implement a rude word filter.

This looks like legit, no-nonsense gets-the-job-done code that gets updated every time some jerk find a new way to be a jerk to others. It isn't great, but at least not over-engineered, and I'm not sure if Twitch account sign-up volumes and abuse are at the point where they should staff a project to do this more robustly / scalably

It seems you are assuming that software is usually written well, or as well as it can be. It's much more likely to be the opposite.

It also mostly checks for English naughty words and not much else. People can have fun in lots of other languages, so it would seem this is a small sample.

That list is a list of words chosen by William Knowles to taunt any NSA who may be listening.

It's not a list of words used by the NSA or any spies. https://attrition.org/misc/keywords.html

I can't find the article now but the developer who wrote these scripts said they were a singular effort from years ago before security was taken over by a more formal development team.

Someone in another thread mentioned that these might be part of corpus generation for an ML model. That would make more sense to me.

Seems plausible to me. One of the streamers on Twitch had an actual wooden board on the wall he lasered subscriber names onto, and some of the regulars had fun finding lewd usernames to gift subs to. There were quite a lot of them out there. It was kind of a running joke how much Twitch let through the cracks.

If valid, this means virtually every phone call, and every email (with clients) is flagged. :P

This must be more nuanced. Maybe, it's "additional algo processing when a word is hit", eg another layer before "involve human".

>the guy that streamed a talking banana on Twitch

Of all the ways I expected a talking banana to backfire, I didn't expect this one. Thanks for sharing

The list included mike hawk (phonetically similar to 'my...'), so they are interested in phonetics, apparently, even for these usernames. The banana streamer has their stuff set up better than twitch, then.

From the video: "It was a mix of racism and creativity"

That sums up the WoW Classic (and I'm sure many other gaming communities) a little too perfectly.

That was both terrible and amazing.

They figured that out on Ellis Island so yeah. Soundex.

That is oddly hilarious yet really dark at the same time. Thank you for sharing the story about the talking banana.

But now it's anti-Chinese if you can't say 那個 :v

is not even remotely complete, in italy we have two regions dedicated to the creation of blasphemies so advanced in ingenuity that two telephone books in regexp would not be enough to stop them

That one was my favorite too. Filter lists don't work well in a multi lingual world. https://www.reddit.com/r/Rainbow6/comments/a01w7q/got_banned...

that should actually be porcAmadonna

Italian

It's an endless arms race. I spend a lot of time studying unsavory people and a great deal of effort goes into the development of new dogwhistles that are designed to either provoke or connect with peers while maintaining deniability. You might like this paper on the evolutionary dynamics of covert social signaling: https://www.nature.com/articles/s41598-018-22926-1

At an online company I worked at we had various word filters for our forums, but new stuff was always popping up and getting through.

What worked in the end was having any newly created thread send a message containing the post title & body to a slack channel specifically for monitoring the forums. Employees and our forum moderators were in there, and any bad threads were nearly instantly deleted. Eventually the spammers mostly gave up. Hard to beat a dozen human brains :)

Better yet, my children's school had a parent support portal called Isis.

I have some fun emails whose subject line is "here are your ISIS family log-in details"

and this was right around the time the terrorist group was frequently in the news

Are people not familiar with the Egyptian goddess, though?

Honestly, what a great way to start _that_ conversation haha

ISIS was founded in 1999.

The Iraq War began in 2003.

Bush's administration ended in 2009.

ISIS gained power in 2014, 5 years into Obama's administration.

I feel like you could make an interesting game out of this. Given these rules, find the best "false negative," a realistic and inoffensive, but banned username. My best so far are "brownie_gurl" and "Megasthenes."

Sure, but...

The story we know was written by him. I wonder what the trolls would say, or the dead dragon, or the town that his actions helped destroy?

Is he a hero, or, just the guy who wrote it all down?

And just when something really important happens, he throws a powerful weapon(the one right) at his nephew and goes away to retire!

A life lead with riches (gold from the trolls), a ring granting him extremely long life and health, and yup.. off he goes, first sign of real trouble.

Poor Bilbo indeed!

So is Mike Hunt.

Poor Mike.

Agreed. If it's even close to a no no word it should be banned completely. No more riggers, naggers, poggers, biggas, lucks, bunts, minks, bikes, or trikes. This is a doubleplusgood plan.

People how hate naggers don't like levenshtein either

English only

Should optimize checks with a de-obfuscation function (attempt to expand non-AZ back to AZ, even if that then shoots permutations at banned word-runs).

It should probably also look more like a spam scoring system, where really obvious stuff is hard-trashed but borderline things are flagged for review / discussion.

I am also very disturbed that, as with most censorship, 'obviously bad' things such as terrorism/etc are co-mingled with 'is adult' as a negative check.

It seems reasonable for Twitch to have validated 'safe for minors' areas where names are filtered. Generic areas, where things are in the gray area and unchecked. Adult Only areas, where swears, profanity, maybe even some of the hateful things are allowed. Informed consumer choice.

It also includes a couple of Italian swearwords in the function `is_blasphemy`

Your best bet atm is to just look through reddit/hn comments/posts people make as they find stuff. The leak's too big for one person/team to quickly find all spicy stuff.

I feel bad for the Amazon employee who came into work one day with this project sitting on their desk.

Well, is a good idea actually.

You can if you’re a magic fire sprite:

http://www.threepanelsoul.com/comic/picked-out

420 == marijuana, blazeit - self explainatory.. not everyone is fine with it..

I see they got Mike Hawk, too. But Mike Litoris is still free to use his/their own name.

We had a legitimate Michael Hunt at our school. He went by Mike.

The word is bitch so the sanitised version would be B**

Why portray the bad word as 3 letters - it doesn't make sense.

Hello, what would be a better way to do it? In code? Regex?

Give me a few weeks

'Angry parents say gaming site Twitch promotes drug use, slow news day story at 10'

Boss: "Hey Bob! People started spamming one of our boards, we are currently busy doing other things and cannot deploy new client, can you make filter with these 3 words and deploy it ASAP?"

Bob: "Sure no problem if it's only temporary"

--few months later

Chief architect:

"People are spamming more and more, we should design new system for these 234 new bad words, I need team of 7, two backend guys, 5 frontend guys and 4 weeks. It will also require minor rewrite of few external components."

Boss: "geez we're in the middle of sprint right now, Bob can you add these 234 words to existing filter? Make sure it's in production before lunch, thanks" (checks watches) "I have to go now, meeting with customer, bye".

It is FUNCTIONS so it is code like any other. Even SQL is code (declarative).

The reason is: This can be updated very easily and out-of-band of other deploys of the main application code.

To make this work it has to be super agile to update. Probably this version we are looking at is an old version that happened to be put in git. The functions in prod probably have several additions since.

A more traditional one, older than these you linked: https://en.wikipedia.org/wiki/MediaWiki:Titleblacklist and https://meta.wikimedia.org/wiki/Title_blacklist

Language friend, this is a family site

Unexpected "The Good Place" :)

Or designer of the first commercial microprocessor, Intel's 4004 https://en.wikipedia.org/wiki/Federico_Faggin

Let's not jump to conclusions. We don't know where this specific filter was used, is it an automated blocker or just flagging for moderation, is it the only layer, or is it even the currently used one.

It only allows English characters to begin with. It doesn't even do Latin Extended characters like čćšđ, let alone non-Latin.

A bad words list is a regex. ;-)

But note: for the most part this isn't using regexs, and to the extent it does, it seems largely intended to make the maintainers' lives easier by avoiding having to represent (and maintain) all the permutations they are trying to match for.

What's sad though is that they're doing many, many passes through the pattern matcher, rather than just building a single big DFA from the whole list of patterns they want to match, which gets traversed in one pass.

safety-ml - I assume this is some attempt at training a machine learning algorithm to find bad names.