Gzip or Brotli will help, but the result will still be bigger when base64 is in the middle.

The only problem with using any sort of compression in HTTP is that unless you add extra countermeasures, you open your page up to the BREACH vulnerability:

https://en.wikipedia.org/wiki/BREACH