The only time I care about it is when it is impervious to the plot.

Or, well, that's a lie. The second time I care about it in a negative way is if someone creates a character that falls outside of the spectrum of 'male, female, androgynous' simply because they want a special snowflake character, or because they want 'representation'. If it is done to strengthen a character or otherwise make it more interesting it's fine. But I've seen it done in ways where it was basically the same as saying 'my character has heterochromia' and that's it. 

It's like a fashion accessory. It's bothersome. (But that has more to do with bad writing than with gender.)