Precisely. And much like burglars, people trying to access your WiFi fall broadly into two groups; those who want to access any of the WiFi in the area and those that want access to your specific WiFi. My guess is that in the vast majority of cases it's the former and as long as your 'locks' are better than your neighbours you'll be fine. However if it's the later, then just like burglary you've entered a whole new problem space.