They think it means that the man in question is less of a man, they think it looks strange, they think men shouldn't do that sort of thing. (No, I'm not singling out women as the bad gals here, it's just that most men who wear panties tend to be straight, and a great proportion of them tend to be married as well.) Oftentimes it seems that some women have a particularly snarky reaction to men wearing panties.
This is a question I ponder when I hear from men who are trapped in relationships with women.
