macOS 14.4 causes JVM crashes (opens in new tab)

> "This blog doesn’t explain why their JIT thread needs to make illegal memory accesses instead of an explicit check."

Because explicit checks on every memory access (pointer dereference) makes Java significantly slower, even with compiler optimisations to remove redundant checks[1]. Memory protection is a fundamental, very useful, hardware feature and it's perfectly reasonable for user space language runtimes to take advantage of it.

Or, to put it another way, SIGSEGV has been a part of Unix-family OSes for decades. It works perfectly fine on Linux and Windows and there's no reason it shouldn't work on macOS.

[1] (Many years ago I worked on a cross-platform implementation of the Java runtime and wrote much of the threads and signal handling code. We had an option to enable explicit memory checks, which got us up and running faster on new platforms where the SIGSEGV handlers hadn't been written yet. From memory this made everything something like 30-50% slower, so it was definitely worthwhile to implement SIGSEGV handling. In our case SIGSEGV handlers were used both as part of the garbage collector/memory management and to implement Java's NullPointerException)

As Linus famously said: Shut. Up. Don’t break userspace and then blame the user.

https://lkml.org/lkml/2012/12/23/75

So MacOS is trying to be smart, changes their API, and now we're blaming the JVM for doing something we don't understand?

At least they could have provided a path back to the old behavior.

It said it affected back to Java 8, so seems like this design has been there for a while, and since older versions are EOL, any Java level fix would not be patched back.

If something works on OS release version 1 then it should still work on OS release version 2.

Or in apple vernacular, it should just work.

This isn't about files, this is about plain pages of RAM[0]. It is a basic CPU operation to trap on unmapped pages, and OSes rightfully expose this useful feature (in addition to using it themselves), allowing processes to do many things, from lazily-computed memory regions to removing significant amounts of overhead doing a thing the CPU will inevitably do itself anyway.

I believe the "the truncation of memory mapped files" section is for when the Java process memory-maps a file (as Java provides memory-mapping operations in its standard library, and probably also uses them itself), and afterwards some other unrelated process truncates the file, resulting in the OS quietly making (parts of) the mappings inaccessible. Here the process couldn't even check the permissions before reading (never mind how utterly hilariously inefficient that would be, defeating the purpose of memory-mapping) as the mappings could change between the check and subsequent read anyway.

[0]: https://bugs.java.com/bugdatabase/view_bug?bug_id=8327860, "I've managed to narrow this down to this small reproducer:" section

> I also wonder what happens in the (extremely rare) case where the signal the JVM is trapping is a real segfault, and not an operating system signal.

Just an educated guess, but the JVM knows if a thread may expect a segfault at a given point or not. If no thread expects one, then I assume the segfault handler just writes out that a segfault happened with some useful info, and terminates the program. I mean, I’m sure about the effect as I have caused a JVM to segfault a couple of times with native memory, so it handles it as expected.

> Apple has long been telling people (writing JITs) that to write to executable memory, they need the correct entitlements (com.apple.security.cs.allow-jit, allow-unsigned--executable-memory, and or/ .disable-executable-page-protection). I wonder if Oracle has been ignoring them, satisfied with the signal-handler workaround, and Apple finally enforced their policy.

As far as I understand, that’s not the issue, the JIT itself works just fine. The JVM just uses the (quite common) trick that it doesn’t actually bound check everything, but let’s the hardware trigger an interrupt, expecting that to “bubble up” to the program at hand, so it can handle certain cases “for free”. This behavior was changed by apple, which causes issues.

This is honestly a wild and out there claim. The OpenJdk team would never want to see this happen to their user base. They’re some of the most professional programmers I’ve ever seen.

The whole truth is that the Apple kernel team broke user space.

Apple has never been a follower of semantic versioning.

It wasn't in the public beta according to Oracle.

This kind of behaviour is very common from Apple.

I don’t think the article claims that a Java process tries to access some other process’s memory.

In a typical modern operating system, a memory page can be non-writable and non-executable, writable and non-executable, or non-writable and executable, but not simultaneously writable AND executable.

If you generate executable code at runtime, then you need write access to a page to write the executable code into that page. Then you need to tell the operating system to change the page from writable to executable.

If you then try to write to the page, you’ll get a signal (SIGSEGV or SIGBUS, according to the article).

Oracle’s JVM apparently relies on this behavior: a Java process sometimes tries to write to a page (in its own memory space) that is not marked writable. The JVM then catches the SIGSEGV and recovers (perhaps by asking the operating system to change the page back from executable to writable, or by arranging to write to a different page, or to abort the write operation altogether).

It's not. It's trying to access unmapped or protected memory in its own process.

Basically what its used for is to implement an 'if' that's super fast on the most likely path but super slow on the less likely path.

It's not super clear what its being used for (this is often used for the GC but the fact that graal isn't affected means that likely still works). Possibly they are using this to detect attempts to use inline-cache entries that have been deleted.

In a virtual memory operating system, every program has its own address space. Accessing an unmapped address is not the same as trying to access another process's memory.

It's also pretty common to use memory protection to autoextend stacks... Allocate the stack size you need, ask the OS to mark the page(s) after the stack as protected, catch the signal when you hit the protection, allocate some more stack and a new protected page unless the stack is too big. Works for heaps too.

Let the MMU hardware check accesses, so you don't have to check everything in software all the time.

It depends on exactly what is being done.

A fairly common idiom is to use memory protection to provide zero cost access checks, as you can generally catch the signals produced by most memory faults, and then work out where things went wrong and convert the memory access error into a catchable exception, or to lazily construct data structures or code.

So you want the trap, but the trap itself can be handled. It sounds like there’s been a semantic change when the trap occurs for execution of an address or an access to an executable page.

There are also a bunch of poorly documented Mac APIs to inform the memory manager and linker about JIT regions and I wonder if it’s related to those. It really depends on exactly what oracle’s jvm is trying to do, and what the subsequent cause of the fault is.

Certainly it’s a less than optimal failure though :-/

The reasons are literally spelled out in the following paragraphs.

Accessing such areas is sometimes done deliberately since programmers could rely on the OS telling them what just happened using signals instead of nuking the process wholesale. Doing it without signals is usually slow and/or clunky (null-pointer checks, read/write permissions, existence of pages), or straight out impossible.

Accessing other processes' memory is not the concern since virtual memory provides each process the illusion of having the entire address space for itself.

".. is affecting all Java versions from Java 8 to the early access builds of JDK 22. There is no workaround available .."

Do not update until Apple fixes the issue.

I would not update for the time being. If it works it will probably break, if it is broken it may break more.

Btw what sort of problems are you facing? I have had problems with closing figures, but figured it out eventually with a workaround [0].

[0] https://se.mathworks.com/matlabcentral/answers/2027964-matla...

Yes, you mean: https://support.apple.com/en-us/109035

Can you tell from this or any other Oracle bug whether Apple is bending its rules for Java? I can't tell either way.

> it seems highly unlikely that the macos people don't test anything on the jvm during acceptance.

I would be surprised if they do to be honest (Apple doesn't even catch obvious bugs in the new macOS settings panel, which really makes me wonder if there is a software QA process at all). For 3rd party apps they seem to rely on the software vendors to holler if a macOS update breaks their app. That's why the macOS prerelease versions exist. But since the bug wasn't present in the prerelease, affected vendors couldn't catch it. It's still a fuckup in Apple release process of course (which tbh also isn't surprising).

It's not a problem that breaks all JVM based software instantly. So maybe Apple tests but not long enough to trigger this issue.

I really don't know what Apple would be 'warning' against. Don't use Java? There are tens of thousands of business and development tools depending on the JVM. Blocking Java would diminish the value of macOS tremendously and doing so without warning would open Apple up to lots of lawsuits.

Even if this was the right thing, they could / should have changed this behavior in a pre-release because that's exactly the kind of API change in the OS that will catch people off-guard. As another commenter wrote, I'd consider this either a serious flaw in Apples release process or they learned about some very dangerous vulnerability where the old behavior was abused and they decided that they rather annoy all users and vendors of Java software out there than tolerate the vulnerability in MacOS. But in this case I'd surmise that at least now Oracle would have been informed about this.

> I’m assuming it’s not common as you would expect it to have come up during betas and pre -release seeds.

The article specifically says that the issue was not present in early access releases, so it was not possible to discover it before the actual release.

Sarcasm only works when you are actually smart and know what you are talking about.

To finish validating a request and then start executing it creates a race condition. That's why execution always needed to fail in a recoverable way.

It’s not at all like they didn’t have the time or the resources to deal with this.

I think we already saw some of this in particular with the recent bullshit they tried to pull with PWAs in iOS 17.4 that they were hoping to just let things break and were hoping that they could shift the blame and anger towards the EU instead.

This is what I do when my job forced me to use a mac. I think the only thing I installed on the mac outside of it was Firefox.

Worked great for years before I changed jobs that let me bring my own hardware finally.

To be fair, this is the kind of breakage I'd expect from macOS, but never from Windows

Here’s the YouTube link from Mitchell. I was thinking about doing something similar lately too.

https://youtu.be/ubDMLoWz76U?si=ipmho73-r9FzZpBp

Windows, as a kernel itself and by extension as a server, is very resilient stable to a point that there is a Windows NT 4 machine of a certain railroad control system still running continuously for 14 years without any restarts. It even still reboots back without problem in disastrous cases such as power loss due to hurricane or earthquake. Trust me, it is made by Dave Culter, it, just, works.

It is really the client facing side of Windows that really sucks, (warning: explicitly strong language) such as having really shitty software known as Office, like god why Word and not Latex, and why spreadsheet when we have database that we can query efficiently? Or not being able to have multi-user RDP session due to Microsoft having licensing dispute with Citrix about 20-ish years ago (fuck you Citrix, you asshole!). Or why do I have to do a lot of hoops and install a lot of "C++ redistributable" for running some antique software? Or why do I have to jump through a lot of group policy simply to enable WinRM and get remote powershell management?

Either way, I'm typing this on a Windows 11 desktop with WSL2 on. The hybrid experience is incredible, unless you need some performance critical app (WSL2 is in general slower than bare metal Windows and bare metal Linux itself, of course, except in machine learning).

Things like 9P to cross the Window file system access also introduced a lot of pain such as permission control because Windows does not have a POSIX-like permission system, like instead of having a simple 2 bytes that split into 3 octal number (there is a reason it is maxed out at 777), you have an incredibly sophisticated, capability and token-based access control system dated almost 30 years ago that Linux doesn't even have back in the day! But that pile of shit is now full of bugs and exploits such as token/handle duplication. (oh yes I'm talking about black hat territory as I also do some red team CTF regarding these stuff)

When's the last time you had a BSOD on Windows? I honestly can't recall.

Plenty of people develop for java on macs. The issue is that per the article this behavior was not present in the early access macOS builds, which means something changed between beta and release.

> is anyone even using Java on macOS?

IntelliJ IDEA, the product itself, is JVM based.

Minecraft runs on various Javas.

And there's a known issue with an interaction between minecraft, Java, and the video drivers that crashes out and it can be traced back all the way to here: https://github.com/glfw/glfw/issues/1997

It's not fixed.

It's not terminating directly. I've seen a few IDE crashes this week, less than one per day, but since there's no log there's no easy way to determine it's related to a macOS change.

Maybe not a lot of macOS devs use Java, but a lot of Java devs use macOS

It broke since recent MacOS 14.4, even at 14.4 betas it worked.

I'm running RubyMine on 14.3.1 all the time and it's fine. Should I hold off updating to 14.4 until the dust has settled?

For one, it doesn't affect all versions of Java. Java 20 (an LTS release) and 21, for example, don't have this problem.

Sonoma has been out for only one week!

This is not an API. It's the handling of writes to memory the process has protected. In the past this would generate a signal the process could handle and recover from. Now it generates a sigkill which is uncatchable / unrecoverable from.

These behaviours have been historically well documented.