I think it works. 
There's no real reason to go back, and it highlights Doc's mood and mindset as a player. 
[s]plus if we really want we could just have the characters kill them on the way back, or perhaps have those bandits pose a threat to the town later when the game starts to become more... 'real' [/s]