He didn't pull free from her immediately, he did return it. He just felt like he was taking advantage of his sobstory and didn't want her to embrace him because she felt sorry for him.