"As long as you're not upset." he said, smiling at her