There is an extra element here... All of what you said is true, but the actual issue is not with the video decoding, but with the thread that feeds data from the network to the video decoder. That thread has some long running stuff running on it, which causes the task to enquire data to the video decode hardware to be delayed.

It used to work years ago, but as more and more software bloat has been added to run in this thread, the delays have increased to the point of stutteryness.

Some videos are jumpy because they have fewer seconds buffered in the hardware decoder.