[quote=Halo]
Well, it's a little rude of them not to ask, but they weren't causing any harm, no?[/quote]

The moment you begin to tolerate minor transgressions on your property is the moment major ones begin to occur. 

The next time it happens, make a fuss. If it continues to happen after that, you have a problem.