Cheat software has to do things like nose around in the game's memory space that no legitimate program is going to do. It also contains in its memory space keywords common to cheat programs.
That sort of search isn't really invasive because generally anti-cheat software when it triggers takes an screenshot, and if you're playing the game, there's nothing personal on the screen. Maybe a process is fingerprinted as a known cheat package. Say your kill to death ratio or hit/miss is unusually good. Or you get reported by a higher than normal number of people. Or a human moderator pushes a request, for whatever reason. Or they get a hit on the 'greps' I mentioned, etc.
The goal is to see if you've got a screen overlay that betrays wallhacking (ie "glasswalling") or a cheat system's UI. That's all they care about.