It also says ”In the past, synchronizing your video frames with the display’s refresh rate was often a problem, especially if you also had audio. You could only make simple guesses for when to output a frame […] which didn’t take into account possible latency from user interactions, CPU loading, window compositing and so on. The Core Video display link can make intelligent estimates for when a frame needs to be output, based on display type and latencies. […] If for some reason the processing takes longer than expected (that is, the display link’s estimate is off), the video graphics card can still drop frames or otherwise compensate for the timing error as necessary.”.
So, disassembling the code wasn’t strictly necessary.
Combining this with this article, it seems they try to call your program as late as possible, possibly to decrease latency between the time you have influence on what is displayed and the time it is displayed. Could that especially if you also had audio be the reason they do that, as it might make it easier to keep audio in sync with video?
Right. The point of the API is to trigger rendering as late as possible before the next frame swap, not immmediarely after the last frame swap, as the author expected.
To do so, it has to estimate how long your app’s callback will take to render a frame. (E.g. if it expects your callback to render a frame in 10ms it might call it 11ms before the next frame swap.)
It seems the heuristic it uses to estimate the duration of the callback execution is assuming a fairly constant duration. The documentation seemed to suggest that. But as the author has inconsistent rendering duration.
So, e.g., if it estimates 10ms and therefore calls back 11ms before the frame swap but you end up taking 20ms, you’re going to drop a frame. If it then adjusts and calls back 21ms before, but you render in 1ms, you could render two frames in the space of one displayed frame.
Anyway, just thinking through it aloud.
I think the author may not want to use this API for his purpose, though perhaps by adjusting his rendering so that it always takes the “max” rendering time might let its estimates be correct and then things would smooth out.
I'm also not actually working on anything that uses this right now, I was just curious. I'm just warning about the dangers if you have inconsistent rendering time.
I think the bigger concern is latency between HID input and visual feedback, which people are more sensitive to, and which you can’t schedule in advance for.
http://powerpcliberation.blogspot.com/2013/04/os-x-disable-y...
It's not that it calls you as late as possible though, as my experiments show it just calls you at an arbitrary but consistent point in the frame like a timer.
See Metal Triple Buffering (https://developer.apple.com/library/content/documentation/3D...) and MTLCommandBuffer presentDrawable.
For OpenGL there are similar mechanisms, eg. NSOpenGLCPSwapInterval.
IIRC this project Gets to 120 when you drag your mouse https://github.com/jtbandes/metalbrot-playground
And do you have a 120fps+ monitor? If not I'm not sure what you mean by 120fps unless it's broken or not vsynced at all and painting twice per frame. If you have a proper way of waiting on vsync 120fps is just a waste of power with no benefits and you don't want that.
Re: 120fps, I questioned it myself but the difference is perceivable. Idk if something is broken or not I was confused too. Re power consumption, that’s why you enable setneedsdisplay.
What causes that kind of thing? I've noticed stuff like that while playing video in mpv sometimes (despite my Mac being easily fast enough to play the video) and it just baffles me.
http://mirror.informatimago.com/next/developer.apple.com/doc...
IIRC it's somewhat inspired by Android SurfaceFlinger.
One question though. Is this driver-dependent in such a way that it only works on newer hardware? For example on the 10+-year old machine I'm using right now, could I play with this?
I'm just sad about all the people that already read it and won't see the update so will go away with incorrect knowledge.
If I'm not mistaken, then, it would seem you can do the same thing yourself by just blocking on CGLFlushDrawable() and setting up a timer on the refresh rate as soon as it returns. (I guess that's similar to what mstange suggested in the comments.) At least CVDisplayLink tweaks timer parameters in order to get the timer to fire more reliably.
