You can set LTO units to 1 which is what I do because it leads to some final binary size reduction -- but it isn't much. :(
I am aware that we have to start being more mindful of our dependencies but in commercial this is a luxury that you often don't have.