He'd have to be an idiot to come to Berk and demand his 'daughter' back.
What if he had something over them somehow though? Something dangerous in some way.