This would generally be the function of a counter-intelligence or internal (criminal) investigation group.
In the US that would be handled by the FBI -- they're legally the only ones that can spy on citizens in in the US. They work hand-in-glove with other agencies, but they own domestic.
A non-US example would be Russia, where the FSB (which Putin used to be the head of) handles internal investigations and counterintelligence like the FBI, while the SVR handles foreign intelligence. Or MI5 vs MI6 in the UK, etc.