Why not make it so that his father seduced your character's mother, but left her alone as soon as she got pregnant.