I think I know what you're talking about. There are likely files inside the ~/Library/Application Support/ or ~/Library/Caches/ folders for example.
What is the proper, Apple way to make sure these get deleted when we delete apps? Because I fear there is no universal solution here. There are some files that an app creates that some of the time I would probably want to persist uninstalls. But then these files should be in a user home directory, not in application support according to XDG, right? I feel like the OS should detect dragging of an app to the trash can and clean up its app support folders? I don't think it does this today but I think it should.