Rouen was probably already through the door by the time Viola stepped in. I'm sure he will find ways to amuse himself in the feasting chamber. (Observing Sam being one of them) Of course if Viola comes in with a sulky Cleo in tow, that would definitely spark his interest.