I have never tried using portage with binary packages, but I am using Paludis with binary packages since a while and I have to say it is very different from using Apt or Pacman.

Package managers which are build for source packages (e.g. ebuilds/exheres) have a much more complicated logic running in the background, come up with much harder to solve problems and are a lot slower (orders of magnitude).

The only aspect where this setup is superior than e.g. arch + pacman is when it comes to using source packages on the same system. So I guess that is nothing you would like to use on a Chromebook anyway. Therefore, using a Debian container is a reasonable choice, in my opinion.