I think it's reasonable to impose restrictions on "smell pollution", as much as it's reasonable to limit neighbors from playing loud music at night.
The good news is that vaporization doesn't smell as much (or linger as long) while being nearly equivalent in effects, and edibles don't smell at all. I think users and non-users should be able to coexist peacefully by making a couple of compromises.