The night before. 
When they had to carry stone to the gate, under the order of Thorin...?

If you don't remember, then that's okay. 
You can time-skip to the next day, where Bard does turn up.