You could be full siblings or bastards. The only one who wouldn't approve of the High King taking mistresses would be his wife.