Do most people care if someone was a keynote speaker? No(Outside of academia). Is it a big deal if your talk was selected as a speaker or keynote speaker? No. But should you care if your talk is demoted from Keynote to non-keynote without following due process(and possibly because of personal biases)? Yes.
I don't know anything about any of the parties involved, but I feel like declining the talk was the correct thing to do and raising the lack of due process is also the correct thing to do.
