Women Empowerment Can Shrink Gender Diversity