Quick Study: Could AI Predict Your Next Job?

Note: Transcripts are generated by machine and lightly edited by humans. They may contain errors.

Susan Athey: Throughout my career, I’ve advised corporations, scientists, and government bodies about how to best use data to analyze and predict the economic impacts of their policies and decisions. Now, like so many things, artificial intelligence is revolutionizing the way we do data science. And a key engine of AI called large language models or LLMs has changed the game in terms of our ability to incorporate a wide range of information into statistical models. Instead of crunching numbers, LLMs suck in and spit out text. They’re trained in doing something called next word prediction. So when AI responds to you, it’s really just stringing together a bunch of words that are likely to make sense. It’s gotten really good at making those predictions.

Here at the Golub Capital Social Impact Lab at Stanford GSB, my team and I wondered, could the power of LLMs help us predict more than just words? What about jobs? Could the same architecture used to create breakthroughs like ChatGPT be adapted to predict your future occupation? Social scientists are interested in understanding how people’s careers evolve for a variety of reasons. Predicting job transitions can be helpful to governments who may be planning for disruption if retail or manufacturing declines or if AI replaces jobs. It can also help companies design programs to help workers manage their occupational paths or to assist laid off employees.

To get started, we took a database capturing 40 years of worker surveys about their careers. We translated the survey data into written resumes and then we fed these textual resumes into an open source large language model, essentially customizing it. And we found that this model became astoundingly accurate. It was able to predict a person’s next job better than any other model. We learned that to an LLM, a sequence of jobs is a sequence of words.

And this idea is not just limited to employment. These techniques we use to model worker careers can also be used to analyze data in almost any domain from e-commerce where we might analyze customer journeys, to medicine where a patient experiences a sequence of tests, evaluations, and treatments.

Now, a really important caveat is that LLMs know a lot, but they generally are not very good at the kind of granular predictions we want to make. Here, the goal is accurate, customized predictions about a worker’s next job given their entire work history. Specifically, we use a method known as fine-tuning, which allows us to tailor the model to get really good at a specific task. Because even though big companies spend hundreds of millions of dollars on computing and electricity to produce models that could understand all the language on the internet, they don’t quite get us where we need to be when it comes to specialized topics.

In fine-tuning, you start where big tech left off and have the off-the-shelf AI models also learn about the particular field of expertise you want to use it for. By fine-tuning the large language model, the AI learns not just that engineering manager is likely after being an engineer, but it learns how the probability of becoming an engineering manager changes as the worker gains more experience as an engineer.

It’s really exciting that AI tools designed to analyze language can be easily adapted to solve very different problems across the sciences, including those that people had never thought of as text problems. And this is just the beginning. I have 30 years experience doing this, but now I feel like a toddler in a playground. Some of the most exciting things I’m seeing now are systems that combine different methods, including today’s cutting edge, fine-tuned LLMs, but also rules-based decision-making and traditional machine learning.

As this customized data science becomes more widespread, companies, including small businesses, are going to get more efficient. These methods could assist people developing software and applications, help businesses optimize, and even predict the outcomes of medical treatments. The process of scientific discovery has been upended. Everything I knew about the best ways to do things, I’m questioning again.