This is correct.
You can’t just say you are invoking the 5th and refuse to testify. If pressed, you actually have to get up there and invoke it — possibly to every question if needed.
The fear of perjury is an interesting twist, though. And the removal of immunity may affect things too.
(Although in your own trial you have the right to not testify, that doesn’t apply here).