I was going to ask something along similar lines.

*Ponders for a moment*

Since Silk/Matthew have no reason to leave the safely of their room Jiro would have no chance of meeting them randomly. Perhaps some sort of commotion in the town that would naturally attract the characters towards it?