I’ve played with this recently in setting up a shared netboot server with Debian for my pile of ARM SBCs (I really should write up that blog post...). The upstream kernel’s device trees are different from the ones supplied by the Raspberry Pi Foundation, with some slightly different names for devices, and basically no easy way to use overlays. Other than that minor obstacle, it’s perfectly doable to build up your own image with debootstrap and chroot with 100% Debian repositories (or go the easy way and download one of the prebuilt images [0]). You do still need the closed-source blobs (bootcode.bin, start.elf, fixup.dat) for boot though, which are made available in the non-free repo.
The 4’s hardware support is still being merged upstream last I heard though, so the easy way for that is still to use the Foundation’s kernel (which is what Ubuntu does, probably to also make overlays work as expected).
0: https://raspi.debian.net/