Yes you are right.
If my VPS is broken, I don't lose any secrets, and it does not permit any additional access into my LAN or VPN.
For plain HTTP, of course all traffic would be easily intercepted and readable.
For HTTPS, I guess an attacker might compromise the software and IP tables configuration on the VPS and run a MITM attack to decrypt it.
So yes, I am putting a bit of trust on the VPS, for my specific use-case, the most sensitive information they'd be able to access if they went through the trouble of decrypting HTTPS, was getting access to my music-player :)
I am thinking though, that at that point.. well, even if I hosted at home on my own ISP directly, I still need to put that same amount of trust on my ISP, since they could MITM me as well I think.