James C. Scott’s Seeing Like a State is a useful lens here. Scott argues that modern states flattened human complexity into legible categories so they could govern: surnames, maps, censuses, standardized occupations. You lose a lot of nuance, but in theory the trade-off is that the state can then build institutions that serve the public.

AI feels like a more extreme version of that flattening, but without the civic purpose that justified it. You end up with legibility without legitimacy. That’s the part I think we don’t have a good framework for yet.