"I am merely worried something has happened. We shouldn't have lost communication so quickly" Rory said