Would Numpy be of use? Haven't looked at the source yet, but for a keyboard, 4 ms per matrix would be noticeably laggy, especially if you add debouncing (is that even a thing in keyboards?)
I read it first as 4ms per row, but yeah 4ms per scan wouldn't be too bad. There is of course additional time taken for communication etc, which might take another 4-10ms, which could make it almost a frame behind. Not terrible still.