This is actually a pretty common concept that's been explored lots, not to detract from your idea. A thing to consider is when you get older, your perception of time changes and goes a lot faster. It's actually a reason why old people drive really slowly; the world seems to be rushing by and it's terrifying. Less grimly, it's like when you're a kid and a month in school seems to take forever, while as an adult you lose track of time and feel like time flies.

Watchmen did a fantastic job with this with Doctor Manhattan. He perceived time so differently from everyone else he actually lost his humanity along the way after the accident turned him into a living god. His relationships were emotionless and fleeting, and he could no longer connect with mortal people. How can you when you're literally living several decades worth of time simultaneously? 

Anyways, point is I think if you were immortal, you'd be so used to seeing life pass by and people come and go, you'd probably see most people in the same light as you would at a subway terminal. Family would be like having a short-lived pet you'd love but be able to move on from pretty easily... if you could ever love someone who was mortal. You'd have seen and experienced so much, trying to relate to someone who has only been mature for what might as well be a blink of the eye would be extremely difficult. It would be like a toddler who is fumbling through life, largely ignorant to the world before they die while you know every answer to every question they'd ever have. There would be no growing together or learning, or even dealing with similar struggles. How could you relate to the pressing need to go to college, or balance bills, or have kids by a certain age? You'd never understand the stresses and worries of a mortal life, especially when you have an eternity to do anything.