> There has to be a leak of some kind, but I don't know how to diagnose it
Nicholas Nethercote's blog[0] has many entries on memory, memory profiling &al as he's one of the big "memshrink" guys.
about:memory might be a good start, after reading his memshrink entries[1] and maybe contacting him.
Although the very first order of business is to check your extensions, extension leaks is the most common source of trouble (and has been for a long time). Disable everything, see what happens, then re-enable extensions one by one.
[0] https://blog.mozilla.org/nnethercote/
[1] https://blog.mozilla.org/nnethercote/category/memshrink/