The Clean Energy Bottleneck Isn’t Technology. It’s Market Design.

The transition to clean energy is underway, but major structural obstacles are slowing it down. Wind and solar power accounted for a record 17% of electricity generated by U.S. utilities last year. However, fossil fuels still supply the lion’s share, and a huge number of new natural gas plants are in development to power data centers.

“It’s not a matter of technology or cost,” says Ömer Karaduman, an assistant professor of operations, information, and technology at Stanford Graduate School of Business. “The economics generally favor renewables.” The barriers to deployment, he says, stem from issues such as misaligned price incentives, red tape, and mismatched supply with demand.

Over the past few years, Karaduman, a fellow at the Stanford Institute for Economic Policy Research and the Precourt Institute for Energy, has focused on identifying these roadblocks and proposing solutions. The central question in all of this work, Karaduman says, is “How can we redesign electricity markets so that clean technologies are able to scale quickly, while maintaining affordability and reliability?”

The Battery Paradox

One challenge is that solar and wind are intermittent power sources, which means that utilities can rely heavily on them only if they’re paired with grid-scale battery storage — typically sprawling facilities with hundreds of shipping-container-sized units, each packed with thousands of brick-like power cells.

In one paper, Karaduman modeled a wholesale electricity market in which storage operators buy electricity when it’s cheap and abundant, then resell it when demand exceeds supply and prices climb — for instance, buying solar power in the middle of the day and selling the excess at night. In this way, they smooth out supply fluctuations, drawn by the opportunity to arbitrage prices.

Yet Karaduman found that the private return on batteries falls short of the social benefit of emission reductions they enable. “The problem,” he says, “is that storage itself affects prices. By balancing supply and demand, it reduces price volatility — negating the source of profit.” In other words, if grid storage does its job and creates supply-side flexibility, it becomes less attractive financially.

It’s a result that earlier analyses missed because they assumed storage operators cannot set prices. By integrating storage decisions into a more complex dynamic model, Karaduman was able to capture second-order effects that are game-changers.

“In this case, relying on market forces to allocate capital leads to underinvestment,” he says. “So you need to realign incentives.” One idea is to change the business model so that storage operators get paid not per kilowatt-hour delivered but simply for maintaining a certain amount of power on standby. Such guaranteed “capacity payments” would also reduce investment risk.

Breaking Gridlock

An even bigger bottleneck is the engineering review that proposed electrical generation and storage facilities must undergo before construction begins. Here, renewable energy is a victim of its own success. There are currently so many proposals that what used to take a year or two can now take seven or eight, causing many developers to give up.

The review process is rigid and sclerotic, Karaduman says. It used to work because electricity consumption stayed flat for decades, despite economic growth, thanks to gains in energy efficiency. But with the explosion in demand from data centers and consumers opting for electric cars and heat pumps, more generating capacity is needed, and fast.

“The capacity of projects in the interconnection queue is twice the existing capacity in the U.S.,” Karaduman says. “It’s an operations issue, really. We need to increase throughput in the review stage.” That’s the focus of a recent paper, in which he and his Stanford GSB coauthors Yue Hu, an assistant professor of operations, information, and technology, and postdoctoral scholar Lin Zang evaluate ways of speeding up the process.

One such innovation is “cluster studies,” in which projects are evaluated in geographic groups rather than individually. Another approach focuses on screening out proposals that waste reviewers’ time and slow down more feasible projects. For example, requiring large up-front deposits and proof of site acquisition would weed out speculative or underfinanced proposals.

To measure the effect of these reforms, the researchers built a model based on PJM Interconnection, the largest grid operator in the U.S., which shut down its queue for new generation projects in 2022 to tackle its backlog.

Karaduman, Hu, and Zang showed how coordinating cluster studies and financial requirements could help grid operators process projects more efficiently. The measures need to work together: When a project withdraws, others in its cluster may need to be studied again. PJM incorporated these measures when it reopened its queue this year, adopting a “first-ready, first-served” review process. “It’s not a one-size-fits-all solution,” Karaduman notes. “What we provide is a framework that firms can adapt to their own circumstances to design better interconnection queues.”

Automating Demand

While storage allows some flexibility in power supply, it’s not enough to make wind and solar primary sources. “A grid based on renewables needs flexibility on the demand side too,” Karaduman says, “to better align consumer usage with generation patterns.” That’s the focus of a paper he wrote with his Stanford GSB colleagues Stefan Wager, an associate professor of operations, information, and technology, and postdoctoral scholar Mohammad Mehrabi.

Demand shifting has been tried using peak and off-peak pricing, with meager results. “People don’t want to be burdened with yet another complicated optimization problem,” Karaduman says. “I want to plug in my car when I get home and not have to remember to go back out at 10 p.m. to charge it.”

The key, he thinks, is automation. Using a home energy management system, consumers can specify outcomes — such as having the car charged by 8 a.m. or keeping their house cool — and let the system decide how to achieve them. On a hot summer day, for example, it might pre-cool a user’s house in the late afternoon, before they return from work, yet while solar plants are pumping out cheap electrons.

The researchers also developed an algorithm for utilities to determine optimal dynamic prices, based on total demand response. “A system like this allows demand to shift without reducing comfort or needing constant attention from consumers,” Karaduman says.

Transfer of Power

Finally, Karaduman says, innovations at the grid level can spread through industry restructuring. In a study of two decades of mergers and acquisitions involving more than 3,500 fossil-fuel power plants, Karaduman and Mert Demirer of MIT Sloan School of Management found that these ownership changes generally resulted in improvements.

When operational control of power plants changed hands, their efficiency increased by 5%, on average. (When only the ownership of a plant’s parent company changed, there was no improvement.) What’s more, capacity utilization increased, electricity output grew by 7.3%, outages declined by one-third, and carbon emissions per unit of electricity generated declined.

“It seems that acquisitions in this industry facilitate the transfer of organizational know-how,” Karaduman says. “Assets are reallocated to firms with better management.” It’s reasonable to expect the same dynamic to apply as more electricity providers move to clean energy sources.

Other problems are slowing the energy transition, Karaduman notes, such as outdated infrastructure. While fuel-burning power plants are located near cities, the best sites for solar and wind are often in remote deserts and plains, where there are no transmission lines to connect to.

Wind and solar have very low operating costs, but building a reliable electricity system still requires substantial investment. The challenge is designing markets that support that investment and reward flexibility as the generation mix changes. “You no longer have fuel expenses, and that means you can’t rely on pricing through marginal cost, the way markets usually work. At some point, our current pricing system will break down.”

Yet Karaduman is optimistic that these challenges can be solved and the switch to clean energy will accelerate. “It has to, right? Humans are very adaptable when they have no choice,” he says. “I think we’re going to figure this out.”