Provided you do not spend the rest of your time skulking about when you are not drinking alone with your pipe in the nearest tavern, I doubt there would be that much issue.