The weirdest bug I've seen yet (opens in new tab)

As I was reading this I was thinking to myself "I wonder if it is grammarly related" because I experienced a bug some time ago that presented itself in a similar way. It was impossible to reproduce but affecting lots of people internally within certain departments. Eventually we figured out the thing they had in common was that they had the Grammarly extension installed.

The other key thing was that the bug only appeared on our staging preview urls, not on the live website. It turned out it was because of a bad regex in the grammarly extension that caused the page to hang if the domain name was more than about 100 characters. Our staging domains were pretty long, I think they contained a few subdomains and had a job number or something in there.

This one is more crazy though if it is really caused by the desktop app - that's pretty scary!

Once my college professor was working on her research paper and told me she was struggling get text to stay underlined. Assuming a simple user error, I expected to help her out in 5 minutes.

Over three hours later, we discovered that the combo of her specific video card driver version along with her specific printer driver version would keep text from printing out underlined.

Even if you have crash reporting disabled there should be a .dmp generated somewhere in the user profile directory. Manually uploading that to a bug at https://crbug.com/new would allow a Chrome developer to debug it.

If you can't share the dump for similar reasons to why you have crash reporting disabled, you can build minidump_stackwalk from Chromium and use it to generate an unsymbolized stack trace that you can post to the bug. A Chrome developer can then symbolize it.

https://www.chromium.org/developers/decoding-crash-dumps/ has some more details.

What an interesting conflux of tech to create this bug. That's the web in 2023. I would love to know if it was a Chromium bug that got resolved, but navigating this is tough: https://bugs.chromium.org/p/chromium/issues/list?can=1&q=gif...

Also, I am fully here for Gusto posting this to say "wasn't our fault" and to throw some shade at Grammarly in the process

I randomly had the issue that after booting up Linux, I didn't have any sound. Turns out it was related to my Windows dual-boot setup!

When restarting from Windows, Windows doesn't shut down my realtek audio device, but only puts it to sleep and Linux fails to start it. Only solution is to always do a shutdown from Windows and then hitting the power button. The issue is still there: https://askubuntu.com/questions/1032543/no-sound-in-ubuntu-1...

I agree with the others here about the ending being a total letdown.

Unfortunately, with access to neither the Chrome source code nor the Grammarly source code, we can only guess.

Is this what the "open source" movement has created --- developers who are totally lost without source code and refuse to dig deeper? Of course the corporate interests who don't want us to know the truth, because their profit depends on it, would absolutely adore such propaganda...

I still remember a time when a lot of people would disassemble, understand, and patch programs without source --- and many of them weren't even career developers; it was just a way to get software to do what one wanted, and driven by that motivation, one would naturally learn enough to do so.

The article also touches on another point worth mentioning: the amount of complexity in the whole stack is insane. Seeing all the frameworks upon frameworks being name-dropped, I can't help but feel like a lot of this is self-inflicted.

When I removed loader-spinner.gif, the placeholder we display while the menu options load, the page stopped crashing.

Do menu options take long enough to load that they need a loading animation?

The weirdest one I saw was this: User claims that the wording of the info they enter into a certain form changes when they save. At first I suspected someone else editing the same form at the same time unbeknownst to each other but it wasn't the case according to the logs. And I saw the correct wording on my own computer.

Then I noticed in their screenshots that some of the menus had weird wording too. Turns out they had Chrome's "Translate this page" option on. Problem went away when we showed them how to properly switch languages in the app.

This is _so_ familiar!

I have seen accessibility tools in Chrome lead to this kind of issue in the past with a dropdown menu -- to the point where it could be replicated with a miniscule amount of HTML. The particular bug I hit 2 years ago was in Chromium-Edge, but the symptoms and cause were very similar.

Grammarly almost certainly leans on some of the accessibility tools in Chrome. These tools are somewhat different in the various Chromium flavors (Edge, Brave, Chrome, etc.).

Next time this happens, I recommend just letting people use a different browser. Firefox in particular has gotten much better at importing bookmarks, passwords, etc. from Chrome.

It was a Sign from the universe that it was time to make the switch. Who are we to reject Signs?

(Full disclosure: I'm an engineer on Firefox. But that has nothing whatsoever to do with my advice here, no siree Bob, not in the least.)

I love the corporate security policy that disables Chrome crash reporting for security reasons, but allows employees to install Grammarly.

Reminds me of the time back in 2010 when a piece of CSS on the Yahoo Search page would cause a complete desktop crash on Red Hat Linux: https://tech.bluesmoon.info/2010/04/can-website-crash-your-r...

To the author, did you ever consider contacting the Chrome dev team about this? They're pretty responsive to bug reports.

Grammarly is an ipso facto security hole that is extremely likely to be pwned or controlled outright by a number of state security services. It should be treated like malware.

If they can figure it out, they're sitting on potentially a very valuable exploit.

I gotta say, the ending was pretty disappointing. I was so hyped to learn why Grammarly crashes when Chrome loads a particular gif. What about the gif was different? What part of Grammarly made that happen? I hate to say it but this is modern web dev in a nutshell. Remove stuff until you find where the bug is, tweak it randomly until it's fixed, done. No root cause analysis or anything deep.

This is a great post anyway. Well written and still quite intriguing right up until the end. And it seems lots of comments on here seem to also know about this problem, so I think I can still satisfy my curiosity.

It is so satisfying to finally identify the root cause of such an obscure bug. Of course, along the way, you will shed tears. Sometimes literally.

I had a funny one a couple of weeks ago: Text on a button would sometimes not display. After much experimentation, it turned out that the text would display unless the character 'D' was present. It turned out that the sizing of the buttons was just barely large enough for the font-size selected, and apparently the 'D' in the selected font was microscopically taller than other characters.

FWIW I have to thank ChatGPT for helping solve that one. I explained the problem, and it gave me a list of things to try...

Shared library like libgif used by both Chrome and Grammarly but different versions?

Another favorite bug investigation of you're into this sort of thing: https://www.pagerduty.com/blog/the-discovery-of-apache-zooke...

For security reasons, your organization disables Chrome crash reports but allows the use of Grammarly, an app that essentially functions as a keylogger. Consented or not, it’s a keylogger.

https://support.grammarly.com/hc/en-us/articles/360003816032...

The weirdest bug I've seen yet was a concurrency race condition which led to 'spooky action at a distance'. Every time I tried to console.log() to observe some value to see the cause of the bug, the bug would go away.

But alas, I hadn't discovered some new quantum effect... It turns out to have been because the race condition was close within maybe 1 millisecond and adding the console.log() statement there meant that one part of the code would take a bit longer to execute and so the race condition would not occur.

The thing that struck me here is that Chrome crash reporting is disabled for security reasons, presumably not wanting to risk customer information being sent to a third party, but Grammerly is apparently fine? How is a tool that sends literally everything you type to a third party ok, but one that maybe sends something if the browser crashes isn't?

I've actually seen something like this in the wild myself. For awhile there were some GIFs that if placed in Discord, would cause it to crash for everyone who was looking at the chat.

Admins had a fun day when that was found!

I discovered a bug like this with GIFs too. It turns out some gifs can be encoded such that it is animated and a single frame long.

Can’t remember the exact event name now, but the browser fires a play event when gifs loop.

This particular gif was issuing too many of these play events that made the app super slow and freeze since it was doing some work in those handlers.

We had a bunch of crash reports and bug reports. None of those showed the actual gifs our customers were using. When we asked for the actual gif, we immediately spotted the problem.

The true bug in the photo is a "candy-striped leafhopper", Graphocephala coccinea, which is tiny but has very striking coloration.

The Larvae of leafhoppers are commonly known as spittlebugs, which create protective bubble nests while feeding on plant stems

What a cool mystery-solving post! I wish all technology writing were this clear and explanatory.

For another fun debugging tale, google: Mazda radio Seattle NPR bug

I'm guessing this is the webp bug.

The auto conversion to webp on the backend, signaled by chrome, resulted in a bad image that crashes the browser due to grammarly parsing of said bad webp.

Safari doesn't tell the server it does webp and so it downloads the actual gif, and doesn't crash.

Did they open a bug against Chrome with the image file? I feel like any crash on user-provided data is a big deal, always a correctness problem, but potentially a security problem. "We deleted the image so the problem is fixed for us" is OK (I wouldn't waste time writing a blog post about it personally), but I think that Chrome needs to fix this bug.

Is there a copy of the gif available? That’s interesting

I have a saying that isn't perfectly true but often will apply to "fixes" like this -

If you don't know why the fix worked, you may not have actually fixed it.

Discord suffered (suffers?) from a similar thing with gifs. It is or was common for people to post specifically crafted gifs in channels, anyone viewing the channel immediately had their client crash.

Discord client uses Electron, which is in turn Chromium.

Grammarly is awful. I remember adding custom grammarly attributes to sensitive password fields so that Grammarly would not read and store our users' passwords inside their service. So nasty.

This company would sure make a great asset for the FBI, NSA and CIA given that they're so interested in snooping in on foreign language speakers; which happen to be grammarly's main demographic. The thing is malware.

Reminds me of the story I read where the guy's car wouldn't start depending on what flavor of ice cream he picked and when investigated he was right. Some kind of evaporation/vacuum leak or something that was dependent on time and some flavors were farther away in the store and took more time to buy.

> The code for our main navigation bar has a fair amount of metaprogramming, and chasing down threads here was often more confusing than not.

One more point for Don't Be Clever. As Brian Kernighan put it: “Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.”

When I reached the "Stroke of Good Luck" title, I put down my phone and tried to think what it could be based on what I know before the reveal.

It was really hard to generate thoughts of the form "subsystem x causes the crash" and not of the form "try to do experiment x next", but here's what I came up with:

- a headache trying to think about it - if it's not an extension, it's likely another piece of software that has code running in the browser, like PDF plugins used to before Chrome started handling PDFs itself - the crash is in one of the browser subsystems, so either in the JS engine, the GPU code, the layout engine, the parser, some format plugin (video? pdf?) or the network code. Some format plugin is most likely of these to only crash for tech support and remote developers but not devs

Seems I was pretty close. After doing this exercise I would probaply have compared lists of installed software between staff who see the bug and those who don't, which would have solved it. My takeaway is that Zen debugging (just thinking what possible causes the seen behavior could have, without atking any new actions or measurements) is useful even when it seems completely useless and causes a headache.

Ah yes, the great fun of bug-hunting. Probably most people in working IT have some stories to tell about this.

One thing that comes to my mind happened in a company where they used a simple software to create medical reports. One clerk suddenly started creating "broken" PDF files. I debugged this for hours and could not find anything. At some point, I just copied the full raw text content that the clerk had written into one of those files out of the database into Sublime Text and I suddenly saw a small box appear between two characters of one word. It turned out that this clerk had created a Word document with text templates. Some of those texts were copied (at least partially) from other documents or websites. Apparently, either something went wrong while copying some text, or there was some control-character embedded for whatever reason in that text. I don't even remember what we did to prevent this from happening anymore, but one single "invisible" character can really cause a lot of trouble.

Another point is of course if you have strange side-effects just by combining specific software. Back in the day when Crysis 1 was new and popular, I was a little involved in the modding-scene. At some point, my CryEngine editor would refuse to start up. I reinstalled it and tried to debug it, but I could not find anything myself. So I turned to the forums and found a thread where a small group of people had the exact same problem. We all shared our setups and - basically by accident - found out that we all had Wacom tablets plugged-in to our computers. I don't remember if it was enough to unplug the devices or if we also had to uninstall the Wacom drivers, but one of those things fixed the issue and we could all continue to build beautiful maps in the CryEngine.

Btw, I am now suddenly constantly aware of my Grammarly plugin ...

I am using chromium to print web pages to pdf, and I have some images that will crash chrome's to pdf process, I found nothing wrong with this images, the metadata is fine (nothing weird in it). The other bad thing it does not reproduce n my dev machine only on the production server , so nothing I can do, in rare cases an image will always crash crhomium, I find it, open it and re-export it and then it works.

This reminds me of an old Chrome bug we ran up against involving the auto fill feature. All day, our software users are entering patient data, including address information. Chrome would grind to a halt on certain pages trying to provide all their previous entries as auto fill options to the form fields. Our sysadmins disabled the Chrome feature for all of our internal users and the issue disappeared.

   alert(banana++)

Should lock things up and not continue until OK is clicked.

If it is just you and your application you can just spray paint from the hip the alerts from top to bottom. After each line of html you can have one more <script>alert(banana++)</script>, in the middle of your css </style><script>alert(banana++)</script><style> etc

If there are uhh thousands(?) of people actively using the page you put just one alert some place in the middle.

Everyone will have to click on OK, the page crashes. You ask the crashee: Did it crash before or after the "make America great again" alert message. (call it something they would remember)

Now you know the issue is in the top or bottom half. You move the alert to half way the half with the issue in it. If you can get some sort of reasonably consistent crashing you will find it pretty quick even in production.

Hope this helps, or maybe it is a stupid idea and you could explain why.

Spoiler alert But if you read the article... ;@)

Improper animated GIF decoding is potentially a bug and potentially a security vulnerability. The reality is this leads to duck-tape workarounds and greater tech debt on the production side of the web. Sigh. There are standards, there are expectations, there quirks across N implementations, and MxN layers of duck tape in M consuming implementations. Just one implementation intentionally or unintentionally being different causes M headaches.

Also fun Chromium-derived browserisms:

- Updating while open is allowed and leads to silent and not-so-silent breakage.

- MSFT MDE causes unexpected breakage in fun ways every now and then, including the cause of a crash while dragging a tab on Windows.

- Some flavors of Chromium browsers are broken with IPv6 enabled, leading to an ERR_NETWORK_CHANGED on every nth page visit.

Hash collision?

I've run into something similar to this a few years ago where chrome randomly introduced a rendering bug that resulted in a similar crash and only affected our application in seemingly random fashion. I don't remember the full context but I think it was something involving layers & transparency.

We finally figured it out when a coworker couldn't replicate and we noticed they were a version of chrome behind. We were able to track down the specific commits in chrome that broke & fixed it in our case which was pretty cool to see at the time.

If you have any more time I recommend reading through the commit log and see if you can find the changes that broke/fixed this for you. I'd bet on another rendering bug.

Can you share the `loader-spinner.gif` ? Preferably over a service that does not recompress it.

> Using open-source Chromium instead of Chrome did not cause crashes, so we couldn’t see what Chrome code was failing either.

They don't address why they didn't just run Chromium. Or Firefox.

(This is potentially better than the 'solution' they much later ended up with, in which they probably only relieved a symptom of an underlying problem that can exhibit again, and in the meantime is a zero-day exploit waiting to happen. At least, with a different browser, there's a chance that the vulnerability doesn't actually exist, when it's known to exist in their Chrome configuration.)

tl;dr: a certain GIF would crash a Chrome tab, but only when the desktop version of Grammarly was installed. (Not a Chrome extension.)

That's insane!

Can anyone think by what possible mechanism the installation of Grammarly could affect whether a .gif file would crash Chrome?

The company seems to be on Macs since they report that the problem doesn't surface in Safari.

Is there some kind of dynamically-linked GIF decoding library used by macOS that Chrome relies upon, and Grammarly somehow installs one that takes precedence for all applications? I didn't think this would be possible -- I thought image decoding was done natively in the browser and not outsourced to the OS, for security reasons.

This reminds me of similar issues I've encountered as a software engineer. I first ran into this issue about a decade ago, albeit not because of Grammerly,but due to some specific gif causing web app to crash. Both times the gifs were animated. Happened years apart and at different companies.

I see some comments highlighting RAM, which could totally have been the issue. Totally looking forward to a follow up to this later down the road, I am sure this isn't going to be the last time we hear of this.

If Chrome is able to generate a crash-report, nothing would prevent you to intercept it.

> Unfortunately, with access to neither the Chrome source code [...]

I mean, it's still very possible to debug (especially with the fact in mind that Chromium is open-source, and for me has been a very useful source when reverse-engineering and debugging Chrome), but I understand why web developers would not be trained in reverse-engineering techniques.

We used to develop software that used the Windows Accessibility APIs (UI Automation). On certain versions of Excel with some files it would crash the client application with a null pointer exception once you try to read the window name/class. It would be interesting to see the cause of the crash e.g. a core dump/user-mode dump/event viewer log.

I can't remember the details anymore but the craziest bug I ever found was one that would crash the page whenever Chrome's Developer Tools were opened. I ended up having to use an embedded JS-based dev tool to figure out what was going on (I was a pretty junior developer and didn't have any better ideas at the time).

The main reason for tabs to crash is running out of RAM.

Never do you see the guy investigating memory usage, which is weird.

Honourable mention, “we can’t send emails more than 500 miles” https://web.mit.edu/jemorris/humor/500-miles

I feel the author might have missed out on a multi-thousand dollar bug bounty.

I wonder if they checked for exploit code in the image the they likely originally found somewhere on Google.

Ah, Chrome and slow spinners.

Python tests were taking ages on VSCode due to an SVG spinner:

https://bugs.chromium.org/p/chromium/issues/detail?id=103626...

https://github.com/microsoft/vscode-python/issues/9216

> Turbo is a gem we added to speed up our Rails application, but it turned out to be a red herring: it was only introduced after the bug had already been reported, and we learned that the engineer who introduced it had actually been experiencing these Chrome crashes for months prior to Turbo being introduced, and months prior to the bug being escalated to us.

lolol I can just imagine the engineer working on this: "huh, it keeps crashing, that's weird. Oh well, it was like that when I got here. ship it"

The blink debug logs would probably be pretty useful for the engineers involved...

oh and there's this one: https://web.mit.edu/jemorris/humor/500-miles

And that's yet another reason that we must have several different browser engines.

Maybe the name was odd `loader-spinner.gif`

I don't think I would believe myself if I found this being a specific gif. This is a great amount of coincidences in code to cause this.

Grammarly is an application that I don't get. The fact that people are installing, basically spyware, on their computers just to get grammar suggestions to make their writing more boring and add a spellchecker (which is already inside web browsers) is pretty astounding to me. The fact companies allow employees to have it, despite obvious security issues of sending everything one types to a saas, is even more wild.

Well, this is a fascinating murder mystery that establishes 3 compelling suspects - Grammarly, Chrome, and a gif - and then just ... ends, right before the big reveal.

Awww. I was really enjoying that; I like detective stories and rabbit-holes.

Then we get to the punchline: "Uh, we fixed the bug, but sorry folks; we didn't solve the puzzle". So I guess we'll never know why that particular anigif crashed Crome but only Chrome, and only if Grammarly was installed (or had been installed during the same session).

I hope Amy Lai lets us know if the story ever gets an ending!

> For security reasons, we do not have Chrome crash reporting enabled.

> We also confirmed with many of our affected users that they had Grammarly installed on their computers.

Ironic.

Why would they disable crash reporting for security reasons (which might actually help solve the root cause of their availability problem, which they never did solve)... yet run Grammarly (which I'd guess, security-wise, is less trustworthy than Google, in how they secure data themselves once they've inevitably stolen it from the customer)?

Extremely disappointing that they seem to have neither investigated nor enabled others to investigate (e.g. by filing a bug against Chrome).

This smells like a potential security vulnerability.

Such a tease. At least upload the gif so people can poke at it!

How could you write all this and not post the gif so we can try it?

Left me hanging, would not post something like this from my engineering blog. They don't have deeep debugging skills at gusto?

A disappointing ending to the tale. I really want to know how Grammarly desktop works now. It must have interfered with the Chrome process in some bad way.

Thought this was going to be about Gusto "remember this device" which keeps failing. Reported like 2 years ago

Error code 5, on Windows, means that the code tried to dereference a null pointer. I'm guessing that the GIF content is corrupt, containing some 0x00 bytes where they shouldn't be. Then the question becomes, whose responsibility is it to program defensively against things like that? If, as may well be, Chrome is using some third-party library nested several layers deep to render GIFs, then would there be any action that the chrome devs could take, aside from replacing that library with a better one and adding a malformed-GIF test case? (Why don't they already have a malformed-GIF test case...?)

Interesting advertisment, but not really a tech article.

> Unfortunately, with access to neither the Chrome source code

I mean, you basically do! You can just go check out the chromium source.

Twirly prompts have sucked, for weird reasons, since the WWIV BBS days.

I guess some things never change.