I could chunk of the brain is visual processing. If people with history of mild infections have brain damage, wouldn't they commonly have visual disturbances? I've not heard any reports of anything of the sort.
Even being vaccinated doesn’t completely protect you from the cognitive effects of Covid. Everyone who’s had it has some level of permanent brain damage.