Bridging the gap: AI's energy demands and the future of power
34:08 Share

Welcome to today's episode of Lexicon. I'm Christopher McFadden, contributing writer for Interesting Engineering. Today we're joined by Benjamin Lee, professor at Penn Engineering and an expert in data center architecture to explore the intersection of AI, energy and sustainability. From natural gas to next-gen data centers, Benjamin shares insights into the challenges of powering AI's growth and how innovation in energy infrastructure could shape the future of technology.

Join us as we dive into the race to balance rapid technology with environmental goals and discover how the US could lead the way in AI and energy efficiency. Before getting into today's episode, here's something to elevate your 2025.

Level up your knowledge with IE Plus. Subscribe today to access exclusive premium articles enriched with expert insights and enjoy members-only technical newsletters designed to keep you ahead in technology and science. Subscribe now. Now let's continue with today's episode. Ben, thanks for joining us. How are you today? Doing great, thanks. How are you? Very well, thank you. Thanks for joining us. For our audience's benefit, can you tell us a little bit about yourself, please?

Right. So I am a professor of electrical and systems engineering and of computer and information science at the University of Pennsylvania. By training, I'm a computer architect, which is to say that I think a lot about how to design

computer hardware, microprocessors, memory systems, and so on. And over the past few years, we've been thinking a lot about deploying all this hardware in the data center context, large, high-performance systems, and how to make those systems more energy efficient and environmentally sustainable. Fantastic. Right, on to our first question then. So,

So how do you see the US balancing the need for rapid data center development with long-term sustainability goals? Right, yes. So I think there is a very real need for rapid data center development. We are seeing unprecedented demand for computation driven, as you probably know, by the demand for generative AI, both on the research side, but also in the deployment and the business side. And

As we know, as we are becoming increasingly aware, data centers require quite a bit of power and energy. And where this power comes from, where this electricity comes from, will affect our ability to pursue long-term sustainability goals. So there is a tension here. In some of our research and some of our own data analysis, we found that data center capacity is growing exponentially.

at 20, 25% per year. That was before generative AI. So those numbers are even bigger today. And at the same time, when we look at renewable energy installations in the United States, those installations are only growing at about 7% per year. Now those numbers might shift a little bit, but that highlights the magnitude of the difference between growth and data center demand and growth and renewable energy supply. So there will be a sustainability impact

Trying to get more renewable energy on the grid is part of the solution, but I think it will be increasingly hard to hit, for example, the net zero targets that many big technology companies are pursuing. Historically, most of these big technology companies pursue net zero by installing more and more renewable energy and then getting credits for those installations. It's hard to keep up. Is there any slack at the moment?

sort of between the demand for energy and the supply is there kind of a how do I say this a grace period before it kind of hits the crunch you know

Well, I think increasingly we are beginning to see that crunch happening now. I think the last two or three years we've seen rapid deployment of data centers and the grids have been willing to hook them up and supply them power. I think increasingly we are seeing regulatory pushback from local communities. So I think people are becoming increasingly aware of the challenges here and there is increasingly friction in that process.

Alright, so if you could put an estimation, like how many, are we talking months or years before it becomes a big problem with the growth rate of the data centers and the actual supply of energy? How long will it be when the power draw or requirements outpaces the energy supply? Is there any way to estimate that?

That's a good question. I'm not quite sure what the timescales are. I do know that it's generally much faster to deploy a data center that draws 100 megawatts than to deploy the corresponding energy and also hook it up. I think the delays are somewhat in the deployment of new energy installations, but also in the permitting and the construction of the transmission lines to connect all the energy to the grid. So

I think the permitting and the regulatory landscape or energy looks very different than that for compute. And that sort of explains the lag. It's interesting. It's kind of an artificial almost restriction. And if it's permitting and stuff, things that could theoretically be removed to speed up the process, right? Am I understanding correctly? Potentially, yes. I mean, I think...

Getting access to right-of-ways so that you can install transmission lines could be expedited, and that could try to go more quickly. I think that there are other things that are slower to figure out. Historically, I think one of the reasons why things have been slow is that when you put a massive load onto a grid, they want to assess risks to grid stability. And that requires a lot of analysis, a lot of analysis.

electrical engineering to figure out the flow of electricity through the grid. And that has historically been a slow process, and that process is

not keeping pace with the area which data centers want to build out. Gotcha. Kind of ironically, AI could be used to help speed up that process, right? Absolutely. Absolutely. I think that'd be a great idea. Yeah. Ironically, it's not the right word, but yeah. Anyway, so what are the benefits of using federal land for data centers compared to private or state-owned land? I don't know.

short their differences there. Are there unique challenges associated with this approach? Right. So I think when we talk about state-owned land or state as in the US government-owned land, I think there is...

There is an interesting opportunity. Certainly, the U.S. government owns quite a bit of land where they do a lot of energy-intensive work and high-performance computing work. The Department of Energy runs large national laboratories with big supercomputers. So it's a matter of opening up some of that capacity, some of those resources to companies.

private compute in some sense potentially to support AI research and development. So I think that's a great idea. I think that could help with the permitting issues that we've just mentioned, the extent to which we can get new energy installations deployed and also hooked up to these big data centers. So I think from a regulatory perspective, it's sort of a bridging measure or a short gap as we figure out how to get more energy connected to the grid in other places.

I think one of the other sides, you mentioned unique challenges associated with relying on federal land. The discussion is focused primarily on the energy side, permitting infrastructure and so on. There is a secondary consideration or maybe another consideration for data centers, which is data flow or data network bandwidth. So the question is, to what extent do these federal lands also have network

network connectivity so they can move this massive amount of data into the data center for compute but also back out again that's why some places like northern virginia is super popular for data centers that's because they have got really great network connectivity really high data rates into and out of data centers in that location so we got to think both about the energy but also the data movement that makes sense yeah i mean i'm not sure about the us but in uk there's um

Lots of areas have kind of protected lands for whatever reason, ecosystems or historical reasons. I guess that's less of a problem in the USA. It's got to be a much bigger land surface. Yeah.

So maybe I'm being a little bit generous or being a little bit optimistic. I'm assuming that they're not going to be using natural or protected lands. But for data center construction, I was envisioning more of the Department of Energy run spaces where they have big natural labs, Department of Defense spaces.

So hopefully that will not be the case where there will be a broader environmental impact. Gotcha. Yeah. So it's already used land, isn't it? For whatever reason. I think we call them gray spaces, I think. Old enough sites and things like that would be ideal candidates, wouldn't they, for federal or state-owned land? Yeah. Fair enough. So you mentioned about renewable energy.

What about the use of, say, natural gas, which is kind of seen as a relatively clean energy source? How does that compare with things like coal and renewables in the context of powering next generation data centers? Right, yeah. So give the gap between the growth in data center capacity and the growth in renewable energy installations, the 20%, 7% number I mentioned earlier.

That's, uh, we need other energy sources and natural gas is a very attractive option, especially in the United States, which is relatively abundant in natural gas.

Certainly, it's cleaner than oil and coal. I think that when you think about natural gas, it's important to think about the methane that is in the natural gas. And when you burn the natural gas for energy, that methane gets translated into carbon dioxide, which contributes to global warming. But

my understanding is that the CO2 emitted by burning natural gas is smaller than the CO2 emitted by burning coal or oil. So there is an advantage with respect to natural gas. The tricky part here is, of course, the methane that isn't burned but ends up being leaked during the extraction or during the transportation process, right? Because methane has a much higher global warming potential than carbon dioxide, right? So I think

There is potential here for natural gas to be a bridging energy source, but we've got to be really careful about the extraction and the transportation and how we actually do a holistic view of the accounting of these numbers. Fair enough. How would nuclear factor in, especially with the rise of small nuclear reactors? Modular reactors. Exactly.

Right. So nuclear is a really interesting option too. I think it's attractive from the perspective of being carbon-free, right? And I think in our own research, we've thought a lot about powering data centers with renewable energy, supporting them with lithium-ion batteries. But with those technologies, that's not enough. It's going to be super expensive to build enough renewable capacity, large enough batteries. So those solutions alone are not enough. So that

has made me a little bit more receptive to these sort of medium or long-term solutions like nuclear.

I've got to say, though, that in the United States, the industry has a fairly poor track record of building nuclear on time and on budget. So we've got to figure out how to do that if that's going to be part of the solution. It's not going to be a near-term fix. That makes sense. That makes sense. So what policy changes could accelerate the deployment of data centers and the necessary energy infrastructure? You kind of touched on this a bit, but...

Right, right. So I think the regulatory aspects are important to consider. I think we mentioned a little bit about the permitting. So I think first, permitting for energy infrastructure will help a lot in terms of connecting any new energy generation, new energy resources to data centers.

Second, we want to think a little bit about regulations about who pays for that energy infrastructure. Right now, there's a lot of concern about local ratepayers being forced to subsidize data center energy. The idea being that if a data center shows up and the utility provider needs to install new transmission lines or other infrastructure, those costs are passed on to regulators.

local consumers who have nothing to do with the data center that was just added to the grid. So I think formalizing or standardizing how those costs are attributed and who pays for what will sort of ease the approval process and sort of reduce the friction between data center operators and the local communities. So I think

On the true side, permitting and energy markets, there needs to be a little bit more sophisticated thinking about how we can make things easier. Okay, that makes sense. But then from the local community's point of view, it could increase local employment, couldn't it? So there's benefits to them directly in that way if they're not using the data center itself. And obviously business is a global benefit. I thought you have business rates in the US. I guess it depends on the states. Other than private individuals,

something some model with the business rate increase or something to cover the cost could potentially help

I don't know. That's right. Yeah, I think thinking a lot about the rates, the tariffs that people are paying for electricity makes a lot of sense. Makes a lot of sense. Okay, great. On the subject of local communities, often they will oppose things like data centers due to a perceived health and safety or financial concerns. What strategies could be employed to address these issues and gain community support?

I think the financial concerns are real for some of the reasons we've just mentioned, right? The notion of new infrastructure being built out and then those costs being passed on to local ratepayers. I think that's a very real concern. We see people in Virginia paying much higher rates for electricity as data centers move into that community.

The health concerns are, that's an interesting one. I don't have enough data to say one way or the other. I know that there's quite a bit of research here. I mean, articles I've read have focused on, for example, local air pollution because data centers require diesel generators to run in case of a power emergency or something.

And certainly those diesel generators can contribute to air and noise pollution. But because they are backups, it's not clear how often they are run, right? Because it should be very rare events that actually trigger the activation of those diesel generators. Some of the literature I've seen suggests that because a company like Google or Microsoft might come in and apply for a permit,

for some level of air pollution, assuming that that diesel generator is going to be running, folks might be thinking that level of air pollution is definitely going into the air. I think it's just talking about

what might go into there and getting a permit for that level of... So getting more detailed data sets on this question will be important. I think there's certainly a possibility of health concerns, but I haven't seen data that definitively points to the nature or the magnitude of it. That's fair enough. And if they are concerned...

Just some more kind of awareness, isn't it really? Educating them exactly what a data center is, what it does, basically. That's right. Yeah.

That's right. Yes. Fair enough. Okay. So with next generation data centers requiring up to one gigawatt of power, how do you envisage system architects innovating to efficiently manage this immense energy demand? Blah, blah, blah. Right, right. Yes. So I mean, as we mentioned at the beginning of this, I'm a computer architect. I think a lot about system design. So there's right up my wheelhouse. And I think

I think there are two things here: the hardware and the software. On the hardware side, there are lots of headwinds that we are encountering that will make it harder and harder for us to get more energy-efficient hardware. Moore's law used to say that transistors would get smaller and also get more energy efficient as they shrank. That's no longer happening to the same degree.

So we can't simply wait for those transistors to get smaller and draw less power. GPU architects like NVIDIA have already done a huge number of optimizations to improve energy efficiency, and there's a little bit more left to be done, but maybe they've already gotten 100x improvement in energy efficiency. There might be 10x remaining based on the various hardware optimizations that they've been thinking about.

So on the hardware side, I think we are certainly working very intently on improving the energy efficiency of the compute. But I think most of the promising solutions are on the software side. So can you compute the same answer for a generative AI model with fewer calculations? And if you think about today's generative AI, it's trained to be massively efficient.

general, right? They can provide a good answer for almost any question you pose to it. And that generality comes at a massive cost. A model is huge, a trillion parameters or more, and it costs a huge amount of energy to train the model, but then also to run the model in response to a query or a prompt.

You can imagine getting a good enough answer or a comparably good answer if you were to specialize the model. And if you knew that the model was targeting, for example, the medical domain or the finance domain or something like that, the model could get smaller. It could be particularly good at answering these types of questions. And it would consume a lot less energy in response to a medical query or a finance query.

So I think specialization of the software could really improve energy efficiency, get you similar experiences with far fewer calculations. And I think that's really the path to go. I think, um,

The last point I'll make here is that why haven't we done this already? Well, I think the difficulty is we don't know what we want to specialize. We haven't found those killer applications, so to speak, the ones that will draw in massive adoption from the broader user base. Once we found that, in a previous generation, that was a search engine, right? Or something like that. And once we knew that the search engine was incredibly important,

a vast community of computer scientists and computer engineers improve the efficiency of it. We're not there yet with generative AI, but once we find that killer application, I think those optimizations will come. Well, with them, like we say, OpenAI, Chet, GBT, I think you can train a version of it for your own purposes. Say you're a business, you can have it trained as a chatbot, basically, for what you do. I don't know if OpenAI are keeping data on that.

But that would be a useful metric, wouldn't it, for this kind of thing, for specializing, for focusing rather, pardon me, on specializing generative AI, wouldn't it? That kind of data, if they have it. Absolutely. I mean, getting a sense of how people are using this massively general model would provide those pathways for specialization and efficiency. So I think it's

The data is there, and I think it's just a matter of seeing what sticks and where the community wants to go, where users want to go. That was fair enough. I think last week I was speaking with Efficient Computer, I think that's what they're called. I don't know if you've heard of them. They're designing, they basically redesigned the chip to be more general than specialized, but it's

It's creating very energy efficient chips. So I'm wondering if that's also a potential, not just the software specialization, but also sort of next gen, I hate this term, but yeah, computer chips, basically. Would also help, wouldn't it? Obviously, data centers, power consumption, whatnot.

Absolutely. So when we think about specializing computer chips, there are really two ways to do it. One is trying to provide more memory or more specialized movement of data from the memory. I think that will certainly allow us to reduce the costs of data movement, which is historically known to be energy intensive.

The other approach, which I think would be great, is if you found particular instructions or particular calculations that happen again and again, can you customize hardware for that sequence of instructions and allow you to do a whole bunch of work with just one invocation of that super complicated sequence?

instruction. That would also be energy efficient because you've only invoked that instruction once, but you're getting a huge amount of work.

So I think both of those strategies are great. And that's what we've seen with GPUs. I think we've seen certainly calculating, performing computation on vectors and matrices are far more efficient than computing on a single value one at a time. So that's what we've seen for GPUs. And there remain opportunities to improve.

to do more of that for chips. Oh, fantastic. Yeah. So kind of crystallizing part of the algorithm really is it's fixing it. It's reused, like you say, you're eased all the time. Yeah. That'd make complete sense. Exactly. Yeah. Exactly. Yeah. Actually. Brilliant. All right. So you mentioned the potential for technology companies to operate energy assets directly. How could this shift impact the broader energy market and traditional utility companies?

Right. So when we think about how data centers consume electricity today, they're essentially retail consumers of electricity. They negotiate with the utility companies, they set up a power purchase agreement, and they pay a particular rate or a particular price for all that electricity. And in return for the paying that rate, they are protected or insulated from infrastructure risks and financial risks on the grid.

So the grid operators might have new energy sources coming online or bidding for supplying energy and pushing energy into the grid. Grid infrastructure, grid operators might have to install utility-scale batteries to make sure that the voltages remain stable throughout the grid. So all of those costs and risks are managed by the operators of the grid, and data centers are shielded from all that complexity and all that risk.

The problem is that that means that data centers are probably paying a little bit more for their electricity, and they're also getting stability. For that stability, they're getting a higher rate. Now, if you look at the other side of the world, data center operators, as they build and operate more of their own energy infrastructure, whether it be batteries or wind farms or solar farms or even nuclear plants,

They have a much larger space of actions and decisions that they can make. Do I, given the energy that I have,

Do I compute with it? Do I charge up my battery? Do I take energy in my battery and sell it back to the grid? Do I sell energy from a nuclear plant and put it back onto the grid? And that larger action space will increase the financial risks and also maybe system stability risks for the data center operator. But it may give it more flexibility or more room to act as an energy trader and optimize this much more

holistic system more aggressively, more efficiently. And once we go into that side of the world, then data centers are essentially wholesale participants in the energy market.

They are just like any other nuclear power plant who might try to sell energy to the grid or manage the risks of voltage emergencies on the grid. So they're going to be taking on much greater risk because they're going to be engaging as an equal participant with many of these other players in the energy market, but they may have more control over their system. That makes sense. I think that's a really interesting scenario to think of that. Yeah. I think I...

maybe a year, a couple of years ago, is it Microsoft put a data center that they model under the sea, I think. Is that right? For the cooling and whatnot. So yeah, thinking of the same terms, you could have tidal energy linked to it. Can you power on it? And they could sell any access back to the grid or tidal ships. They called up their tidal energy ships. Have they got an interesting business model? Yeah. Okay. Fascinating. Yeah.

So yes, you mentioned that easing regulations could help the US maintain a competitive edge in things like AI. So how crucial is this advantage and what are the risks if the US doesn't act quickly? Well, I think sustainability is certainly an important dimension, but at the same time, if you're a computer scientist developing the next generation of AI models, it's

You don't want to be put in a position where you're developing a model and you're wondering whether or not you could have gotten a better model or gotten to the next level of capability if you're just doing another 50 megawatts at the problem. So I think there is a question about to what extent is energy the critical path or the constraint on development?

getting to the next level of models for AI. And by next level, I really mean maybe multimodal models, right? Handling both the language, but also audio, video, et cetera, larger data sets and more sophisticated queries. So I think...

In order to maintain leadership, it's important to provide that energy supply so that we are not constrained for the bleeding edge in terms of next generation models. That being said, I don't think everyone, I don't think every company needs to be

building 100 megawatt data centers to train these next generation models. I think increasingly there will be a question about how many of these companies will need to be trading their own models and doing these very large, heroic 100,000 GPU runs of a next generation model because the costs are prohibitive and there's very little additional benefit once those first few models have been trained and that capability has been demonstrated.

So I think maintaining the U.S. competitive advantage means supporting next-generation models and the infrastructure for it.

I think that the next thing I'll say, or the last thing I'll say here is that the competitive edge in AI also requires finding those really advanced use cases of existing models. There's still a huge amount of uncertainty about what these models are good for and how will they fundamentally change the way we live and work. Chatbots are fun to play with. I don't think they've risen to that level quite yet. So I think

Developing the ecosystem where many different entities, many different companies can play and try different things will be incredibly important. And hopefully they're not going to be constrained

by hardware infrastructure or energy. So during the excitement about generative AI, there was some talk about some startups really having difficulty getting access to some number of GPUs to develop their ideas. I think we want to make sure that there is enough infrastructure that is distributed that many people can try different ideas and explore efficiently. Okay. So you believe then the future is going to be more

what's the word, not niche, kind of smaller scope AIs, new businesses, I mean, coming up rather than, yeah, general, let's just brute force attack AI, rather more specialized niche uses for it basically would be the future ecosystem probably.

Right. And I think it makes sense for everyone involved because the costs are just so prohibitive. If you imagine doing a startup that can compete with an entity like OpenAI backed by Microsoft's infrastructure and you're starting from scratch, that's going to be very difficult. That's a very difficult proposition where you can add value.

But I think if you can find a really interesting use case for OpenAI's models and access those models through some sort of programming interface that Microsoft and OpenAI provide, that is a way that's a way to cost-effective way.

deliver additional value so i think that that would be really exciting to see more interesting so it could be a bit like um the evolution of operations is computer operating systems really you can most basically more or less based on the same thing but specialized in different ways aren't they so you don't have to realize the foundation and yeah okay fair enough um

Right, last question. How does the US's approach to data center development and energy infrastructure compared to other leading countries like China or the EU? And what lessons can be learned, if any? Right, well, I think...

I think for the EU, I think there's been a lot of concern about the regulatory landscape there. I think certainly my sense of where the EU is, is that they're super interested in codifying best practices for energy efficiency. So if you're building a next generation data center, they've got to meet particular guidelines or particular benchmarks for performance and energy efficiency.

For example, power usage effectiveness, right? How much power is the data center using for cooling and how much of it is it using for compute? And you want that to be lower. And so the EU has gone about setting some of these standards. And I think they're sensible. I think they make a lot of sense. And certainly most of the big hyperscaler operators could easily meet those criteria.

I think from the perspective of what's going on in China, I think there's a lot of interest in, of course, building massive data centers. I think the issue there is about energy, but it is certainly also about export controls and sanctions and the flow of hardware. And I think that's probably going to be the dominant factor to figure out whether or not

China will be able to get access to a sufficiently large number of high-performance GPUs and integrate them into a complex system at scale to keep up with the modeling. Okay. I mean, with China, obviously they're opening new power plants all the time. For all their rhetoric on

sustainability they don't really seem to care so Drekna gives them a competitive advantage when it comes to this sort of thing

I think certainly there is a sense that they are willing to use more coal throughout their economy than the United States might. So I think to the extent that that gives them more energy, I think that's sort of offset by the US's advantage in natural gas. I think that there's a massive amount of natural gas in the US.

in the shale reserves in the United States. And it's just a question of how much extraction we want to see. Um, and certainly as we mentioned earlier, natural gas is preferable to coal, uh, and could be a good bridge. Um,

So I think from the energy supply perspective, the United States is pretty well positioned to feed its data centers. Fair enough. Okay, great. That's all my questions, Ben. Is there anything else you'd like to mention that we haven't mentioned or discussed you think is relevant?

No, I think this has been a great conversation. I've been super excited to talk to you about many of these really interesting dimensions to data center computing. And I think it was a broad-ranging discussion on most of our bases. Fantastic. Yes, it's a subject I certainly don't normally think about. It's been quite enlightening, I must admit. Brilliant. In that case, Ben, thank you for your time. That was very, very interesting.

Thanks so much, Chris. I really enjoyed it. Our pleasure. Also, don't forget to subscribe to IE Plus for premium insights and exclusive content.