Well to be fair genetics doesn't always work like that. Ligers, for example, are larger than either of their parent species. So it's not inconceivable that a human's and a Visha's offspring could be shorter than normal.

And I'm getting a sort of "Indian Dwemer" vibe from the Visha.

EDIT: I've also added some of my history in case anyone's curious.