Your priorities come first.

If a man wasn't willing to put up with your life, he would have been gone a long time ago. I think you made a bad choice honestly.