A mistress is just the female word for master, and it's come to be a euphemism for when you have...not quite a wife, and often multiple of them.