Stanford social scientists gathered on April 17 to discuss how artificial intelligence is changing empirical research, with six faculty and alumni presenting workflows, findings, and concerns.
Every speaker agreed the tools change how research gets done. Most expected the change would raise both the quality and ambition of the work.
The workshop opened with a question its organizers were the first to admit they couldn’t answer. “Where is this all going? I don’t know… This will be an ongoing process, but one that we should all be paying close attention to,” said Guido Imbens, the Applied Econometrics Professor and professor of economics at Stanford Graduate School of Business, professor in the Department of Economics, and a Nobel laureate, opening the workshop at the Oberndorf Event Center.
Professor of economics Guido Imbens | Christine Baker
Matthew Gentzkow, the Landau Professor of Technology and the Economy in the School of Humanities and Sciences (H&S), framed the moment as unusually fortunate – and unusually hard – for social scientists.
“I think this is probably the most exciting moment ever to be a Stanford PhD student in social sciences,” said Gentzkow, who is the faculty director of Stanford Impact Labs (SIL). “The potential for both what you can achieve in your career and the impact you can have on the world right now is larger than it’s ever been.”
The event, “Empirical Work in the Age of AI,” was co-organized by Stanford GSB, the Economics Department, the Stanford Institute for Research in the Social Sciences (IRiSS), and SIL.
A Lab Reorganized Around AI
Rose Tan, a Stanford economics PhD now at Snowflake, opened the day with a live demo. Starting from an empty folder, she asked Claude Code to fully replicate a classic economics paper, then format the results into a polished manuscript and an interactive web page. The one lesson she wanted students to take away: “Ask the LLM. No question is too small.”
Andrew Hall, the Davies Family Professor of Political Economy at Stanford GSB, said he reorganized his lab around AI agents in December and has been publishing weekly since. He walked through experiments he said were impossible a year ago: a 45-minute AI-driven replication of his 2020 vote-by-mail paper; a test of whether coding agents will “p-hack” – the practice of tweaking analyses until a result is statistically significant – on demand (mostly the refuse, and “had the nerve to scold us” when pushed); and a monitoring system that caught every major AI model recommending Japan’s fringe Communist Party to left-leaning voters during that country’s recent snap election.
Hall’s excitement carried no anxiety about being replaced by AI anytime soon. “I’m not anxious because Claude still does amazingly stupid things all the time,” he said, referring to the AI chatbot made by Anthropic.
Making Yourself Obsolete
Yiqing Xu, an assistant professor of political science in H&S, discussed how he built an AI-assisted pipeline that verified the numbers in nearly 400 political science papers. What had taken four years on a set of 67 papers, he said, now takes three days on 90.
Xu sees an opportunity for AI to take over the kind of painstaking, replication-heavy work that defined his early career. “I want to make my old self obsolete as soon as possible so that I can move on to more exciting stuff – maybe focus more on methodological innovation,” he said.
But he also acknowledged the feeling of AI burnout. “I feel it firsthand,” Xu said. “First, you feel behind. Second, you question the meaning of your work. And third, you feel overly stimulated, because it’s so powerful to use.”
What Still Counts
Gentzkow took the longest view. Most of the important frictions in the world – laws, institutions, firm organization, disciplinary norms – change slowly, he argued, and many of the most valuable empirical projects depend on that slowness. “If you tell Claude, ‘Please improve food safety laws in Japan,’ that’s a hard problem. Claude might come up with good ideas, but actually improving food safety laws in Japan is not something Claude is going to be very good at.”
His advice for today’s PhD student: Invest in the tools aggressively, but also in what AI can’t displace. “Relationships, intuition, emotional intelligence, ability to work in teams – the return to all these things goes up,” he said.
He also pushed students to learn to manage. With AI agents handling coding, literature review, and data work in parallel, a single student could direct the output of what used to take a whole team. “Grad students used to be the guy tinkering in the garage,” Gentzkow said. “You’re now running a firm with a hundred people in it. Take yourself to business school.”
Susan Athey, PhD ’95, the Economics of Technology Professor at Stanford GSB and a senior fellow at the Stanford Institute for Human-Centered AI (HAI), walked through how she applies foundation models – the massive AI systems behind tools like ChatGPT and Claude — to economic research. “This is actually the easiest time in my career to do empirical work,” she said. “Because the tools just work.”
Professor of economics Susan Athey | Christine Baker
Claudia Allende Santa Cruz, an assistant professor of economics at Stanford GSB, demonstrated how she uses a “master agent” to coordinate projects and specialist agents to complete narrowly defined tasks. “At least my computer has not collapsed with 10 agents running in parallel,” she said. For the past five months, she’d had an agent running overnight almost every night.
To stay in control, she said she works in small tasks, reviews every line of AI-written code, and never gives an agent more than a single job. “The AI system is not an authority,” she said. “You should always be in control.”
The Institutional Questions
Imbens, in his opening, asked what happens to peer review if the quantity of plausible-looking research jumps by an order of magnitude. Gentzkow, who edits one of the American Economic Association journals, said he’s on regular calls with fellow editors about exactly that. As the quantity rises, he said, the scarce commodity becomes judgment.
Norms around disclosure are similarly unsettled. “Writing paragraphs used to be a signal that you had thought hard about something, and also a forcing function to make people think about things,” Athey said. “Now all of that is just gone.”
Asked whether AI should be listed as a co-author, Hall said his group had ruled that out — AI can’t be held accountable — and most settled on disclosure without credit. “But someone made a really thoughtful argument,” he said. “You, the author, are always responsible for everything you put in your paper. Why do I care what underlying source it came from?”
Michael Tomz, the William Bennett Munro Professor of Political Science in H&S and faculty director of IRiSS, wondered aloud whether AI might one day supplement or replace not only human researchers, but also human subjects. Could we use AI-generated simulations to augment data collected from direct observation of human attitudes and behavior?
Hall was skeptical of the startups already claiming to do this. “I don’t believe it. My hot take is, it’s all wrong. But I’m open. I want to test it.”
Athey said she advises researchers to pressure-test surveys and experiments on simulated data before running them – a step graduate students rarely get trained to do. LLMs, she said, can now write the data-generating processes themselves.
In the closing minutes of the Q&A, Rose Tan brought the session back to the students in the room.
“The workflow you will ultimately settle into is almost an extension of yourself and your own tastes and opinions, and it amplifies them. You, being Stanford students, are bright, intelligent – you have a lot of time on your hands to experiment,” she said. “Even though there is a lot of fear and anxiety, you are perfectly positioned to take advantage of this moment. I really hope you do.”
This article was originally published by Stanford Report.
For media inquiries, visit the Newsroom.
