As LLMs become the default interface for search, news, and everyday problem-solving, they may filter and frame political information before citizens ever confront it. Identifying and mitigating partisan “bias”—output with a systematic slant toward a political party, group, or ideology—is therefore a growing concern for researchers, policymakers, and tech companies. Existing methods often treat political slant as an objective property of models, but it may vary depending on the prompt, reader, timing, and context. We develop a new approach that puts users in the role of evaluator, using ecologically valid prompts on 30 political topics and paired comparisons of outputs from 24 LLMs. With 180,126 assessments from 10,007 U.S. respondents, we find that nearly all models are perceived as significantly left-leaning—even by many Democrats—and that one widely used model leans left on 24 of 30 topics. Moreover, we show that when models are prompted to take a neutral stance, they offer more ambivalence, and users perceive the output as more neutral. In turn, Republican users report modestly increased interest in using the models in the future. Because the topics we study tend to focus on value-laden tradeoffs that cannot be resolved with facts, and because we find that members of both parties and independents see evidence of slant across many topics, we do not believe our results reflect a dynamic in which users perceive objective, factual information as having a political slant; nonetheless, we caution that measuring perceptions of political slant is only one among a variety of criteria policymakers and companies may wish to use to evaluate the political content of LLMs. To this end, our framework generalizes across users, topics, and model types, allowing future research to examine many other politically relevant outcomes.
Significance statement: Large language models increasingly act as gatekeepers of political information, yet their ideological leanings remain poorly understood. Most existing audits use automated probes that overlook how real users perceive bias. We develop a scalable, user-centered metric that makes people—not algorithms—the arbiters of partisan slant. Drawing on 180,126 pairwise judgments of LLM responses to thirty political prompts, we find that nearly all leading models are viewed as left-leaning, even by Democratic respondents, and that a simple tweak to system instructions measurably reduces this tilt. The method is readily transferable to any model, topic, or population, giving firms, regulators, and scholars a practical tool for monitoring—and mitigating—ideological distortion in an algorithmically curated information environment.
The full dashboard is available here: https://modelslant.com/