He wouldn't want to work against her, but would feel compelled to do so, as she was working against his master.  He might defect to Luke, though, as he likes Luke a lot more than Vader or Leia, especially if you can get Padme to join you, though she's not on the ship.