This is why it's imperative that inline caches
learn and adapt to the observed behavior. As long as learning is cheap, identifies profitable cases effectively, and backs off for polymorphic and megamorphic scenarios, it's a win.
VM implementer intuition only goes so far, and as the internet is the greatest fuzzer invented, you're definitely going to encounter programs that break your best laid plans.