The point is that it's not friends or hackers trading on information they received from someone privileged but someone totally unrelated using information they gathered legally in public without any expectation of privilege. The cases GP mentioned (hiring private investigators observing public places, evaluating satellite imagery) are legal and used every day.

Additionally, you can even trade on non-public information as long as the insider does not benefit in any way.