You can use SVG sprites to overcome that limitation. My SVG icons are usually only:
<svg><use href="/path/to/sprite.svg#my-icon" /></svg>
Good SVG icons are a single <path> which can be colored in your CSS with
svg { fill: currentColor; }
Then it will take which ever font color it inherits.
Off course I usually wrap this in a component and only write <app-icon icon="my-icon"></app-icon>.