git annex (or git LFS,
if you buy into github's NIH) is requisite if you want to use git like this, broadly. git will happily store any and all binaries you ask it to, but upon (blind) checkout, it will grab every single revision of said binary, taking up as much however much space that takes.
(partial clones avoid this, but, as git isn't designed for this use case, grabbing all of history happens far too easily.)