response time has little to do with latency, it's purely a property of the panel. At 60Hz refresh, an LCD will have a minimum of 16.7ms latency because it buffers an entire frame before displaying. 20ms of latency would mean there is additionally 3.3ms of lag, not that it takes 20ms for each frame (it's already buffering the next frame in those 3.3ms, so the next frame will be displayed 16.7ms later).
Many TVs buffer several frames for video processing in their default mode, which is why they can have 100s of ms of latency when not in game-mode; the response time is identical regardless of which mode it is in though, as the response time is purely a property of the panel.
