The software/firmware/hardware stack is layered. Here's what has worked for me, for several FPGA-based instrumentation applications.
* Integration level: this is typically Yocto or Buildroot. It's architected to want its own repository, and should diverge from "upstream" repositories as little as possible (except for, basically, whatever configuration files tailor the build to your application.)
* Hardware design (PCB tools): Altium, at least, comes with its own Git integration and wants to control the entire repository. I have kept PCB designs separate from other SCM tooling and haven't found it to be too problematic. I think KiCAD is the only tool where I'd even consider combining the schematic/PCB with other stuff, since it uses a text-oriented storage format and doesn't try to micro-manage the SCM flow.
* Firmware (software and RTL) that's closest to the hardware/software design boundary: I really do want these to live in the same repository, so that it's easy to make changes that span the hardware/software divide. I would also argue that corresponding detailed design documentation lives here, too, since it incentivises documents to be maintained like source code, rather than as a grudging afterthought. Only source code is checked in (no build artifacts, no binaries - these would be a constant source of merge conflicts otherwise.)
* User-facing interface code: for me, this is a Python API and access to pre-built firmware releases. These live in a single git repository for user convenience. Firmware images are kept using git-lfs (and are --excluded by default .lfsconfig to keep new clones reasonable.) This has worked very well.