> Like using a shell to work on my files
As others have mentioned, Termux is Free and Open-Source. You mentioned trust, but considering Termux is already open source, I'm not sure how the author could gain your trust.
> use git for version control and to sync to other machines
You got me with git, but Syncthing might solve the same problem for #2 - there's a FLOSS client available on F-Droid as well :)
> use vim to edit text
I'd assume Termux has a vim package, meaning you can stay open source for the entire stack. If it doesn't, then my mistake.
> I cannot even easily backup all my data. Like the contacts
Ignoring using a carddav server for to have them backed up all the time, you can easily back them all up in vcard format via Simple Contacts - an open source app availble on F-Droid.
Hell, for the most part you can completely disable whatever you want, root or no. ADB can disable everything you don't want to run, including all the way down to Google Play Services. I can verify that, as I've done it myself on my BlackBerry KeyOne.
It (mostly) can all be done, it just takes a little bit of looking. Like anything else with modern computing, the fun stuff is hidden away, and all we're presented with is the glossiest interface the OEM can shove in front of us.
If there's anything that actually cannot be done via FLOSS software, I'd honestly love to hear it, because I'm drawing a blank at the moment.