There are many optimizations in GCC and LLVM. If you turn them all off you will compile fast and execute slow, and if you turn them all on you will compile slow and execute fast. You can do this on a per function / trace / translation unit / whatever basis.

Production JIT compilers are the same way. For the hottest code paths, all the optimizations get turned on. The coldest ones are interpreted. The first level of jitted compilation has very few optimizations enabled.

The main thing that doesn't cooperate with JIT compilation yet is whole program analysis.

LLVM understands this issue. That is why they have "regular" instruction selection passes and "fast" instruction selection passes. The JIT uses FastISel. But you are definitely right, it is not anywhere near as lightweight compared to, say, Hotspot.

That's all relative, of course, to how long your application is going to run for and how much has to be JIT compiled.

People at Apple working on JavaScriptCore (Safari's JS engine) have recently added a fourth-tier JIT using LLVM, so very much run for only very-very-very-hot code (of course, by waiting for it to become very-very-very-hot, you've already lost a time you could've gained). I can see LLVM making its way into more JITing compilers in such ways: as an ultimate solution where it is clear it's worthwhile spending a lot of time compiling code.

The node.js server case seems worthwhile to bring up here: node.js servers are typically run for days or weeks on end. An extra few milliseconds (or even a second or two) isn't significant if it makes a noticeable difference in performance.

AFAIK, you can manually specify all of the optimizations done by LLVM during JIT compilation. I think it's not hard to disable most of them for fast compilation, but yes, it still may be too heavy for an instant use.