That's the way things work. 

You just ignore the characters that have disappeared. Even if they're talking to them