I don't think it has to be. For each user you should already have the hash data computed (using something like imagehash this works out to a hash of 8 bytes per image, though you can obviously tune this up depending on storage/performance requirements).

Each time a comment is posted, you would do a distance measure between the commenter's avatar hash and the channel avatar hash. Mixed in with the network latency and DB I/O operations, I think this additional read/write that only needs to occur when a comment is posted could be done with a pretty minimal additional compute overhead.

That said, if Google doesn't have the compute overhead to do it, I gave an alternative. Simply don't display avatars from commenters other than the channel author.