I just poked around a bit.
So git was first created with u32 time in mind only. However because of the looming year-2038 problem, they are working on expanding that.
Apparently git internals are almost ready to support more interesting timestamps. However, much of the git tooling and UI (like command line parsing and output) refuses to deal with pre-epoch timestamps.
I briefly tried with git 'porcelain' and also via libgit2, but it's all a bit annoying.
In summary, I think you'd need to hack up at least some of git's tooling to make everything work, but it wouldn't be heart surgery, because the internals are already nearly ready for this kind of change.