In 1976, the Democratic nominee for president narrowly carried Mississippi. This man was a peanut farmer with a soft Southern drawl and a modest house in the rolling hills of Georgia. He was a white Mississippian’s idea of a good old boy, and his identity, if you will, brought him to victory in America’s most conservative state.

White Mississippians seldom spring to mind when the topic is “identity politics.” Most often associated with women and minorities, the term tends to surface as a political slur. It’s employed by its critics–and by opponents of broad movements for social justice. But affinity voting is not limited to groups championed by progressive activists. As the cliche goes, “all politics is identity politics.”

As far as I can tell, most of the rage against identity politics flows from resentment towards equality claims by women and ethnic minorities. “Identity politics” as a descriptor has become detached from the dynamics of group self-definition that gave rise to the concept in the 1960s and 1970s. Instead, critics focus on the zeal of left-wing activists, with a heavy overlay of anger at the substance of progressive appeals. Seldom does one see sympathy evinced for the basic objectives–racial and gender equality–that drive this discourse that so exercises cultural conservatives.

Furthermore, few political movements are more identity-driven than right-wing populism. Strikingly, one extremist movement on the European far right called itself an “identitarian” faction. In this country, Donald Trump attracted a solid, coherent bloc of older white men and their wives with a brutish appeal to white-traditionalist identity. His cultural appeals hit these people with thunderbolt force, and they formed an indelible bond with the strongman who promised to restore their identity group to its historic dominance.

For all the hypocrisy and ill-intent on the part of foes of “identity politics,” the cultural left also boasts unfortunate excesses. It’s long been fashionable to heap scorn on people who reside outside the bounds of the progressive coalition, such as when a Hillary Clinton staffer sneered about “the extinction of white men.” Even ugly contempt for privileged people does not compare to the horrors of racism, but it nevertheless reflects a lack of empathy and goodwill, and it fuels conservative resentment. Working-class whites (who enjoy real but limited privileges) are particularly apt to bristle at a discourse that seems to dismiss their suffering. Without succumbing to the canard of “reverse racism,” progressives should reconsider their hotter rhetoric.

But identity politics ultimately says more about the animating passions of the cultural right than it does about the zeal of the cultural left. Almost everyone wants to vote for people who affirm their desire for validation by society; hence, Mississippi voting for Jimmy Carter. In the final analysis, politics corresponds with people’s sense of self, and of group, and of loyalty. When conservatives condemn identity politics, what they really mean to say is: Whose identity counts?