Rent or own your AI? The 86 billion token question | Dan Waibel
DAN WAIBEL
the rule of thumb I use is about 1,000 tokens for every 750 words of English.
think of it as the meter on the wall that's running, that allows us to understand, the cost
So this whole concept of tok enomics, and that's really how models operate
SAM JARRELL
So those were a couple of clips from last week's episode, our introduction to tokenomics, and we're gonna be continuing with the topic for this week in a part two
MICHAEL BIRD
Yeah, that's right, because as you might imagine, the economics of AI usage is extremely important to businesses, organizations, and even governments at the moment. So how are they responding to this new financial demand in the world of AI
I'm Michael Bird
SAM JARRELL
I’m Sam Jarrell.
And welcome to Technology Now from HPE.
SAM JARRELL
Welcome back to our two-parter all about tokenomics. If you've not heard it yet, we will link to part one in the show notes where we examined what tokens are and the economics of AI usage. But now we wanna look into the real-world realization of how tokenomics actually affects us
MICHAEL BIRD
Yeah, exactly. Now, once you can track how much AI you're actually using, you might start wondering whether it could be a better financial decision to build your own AI infrastructure rather than just renting it from others
SAM JARRELL
For sure. And it's not just going to be businesses and organizations worrying about this. I would expect that from a critical national infrastructure perspective, being able to create tokens in your own country will be vitally important to most governments
MICHAEL BIRD
Controlling your AI strategy, whether as a business or a government, is absolutely imperative at the moment. So here's the big question: What does that actually look like in practice? Well, this week we are returning to our conversation with Dan Waibel, Global Chief Data and AI Officer at HPE, and we are picking back up with Dan's explanation of what an AI factory is in 2026 and why people might consider building one.
DAN WAIBEL
So an AI factory is, your option to make an investment and build infrastructure that will support your AI journey. And it's really a different cost structure because you're making a capital investment upfront versus renting through APIs in the cloud. And we went through and we did a study, where we said, okay, at what point if you're using more than X number of tokens, should you really invest in an AI factory?
And the number we came back to was eighty-six billion tokens. And while that may seem like a high number, it really is not a very high number. for example, at HPE we're using ten times that number of tokens, in a year. So I think that's how you can make that decision or start to think about should I build an on-prem AI factory?
It is an economic decision, but there's other factors as well, Michael, security and privacy. So if you are building an on-prem AI factory, your data, your company's proprietary context can remain within your four walls, right? You do not need to send that context and that data out to the cloud. You can really keep all that private and secure and also close to your factory.
of course, most large organizations are already down this path or have already built an AI factory, but I think it's gonna become much more commonplace in the future
MICHAEL BIRD
I guess potentially there's a conversation about does a country think about token creation like they would think about critical national infrastructure?
if you think about utilities like water, electricity, things like that, if all the AI tokens are all created in one country or a collection of countries, then they could be a commodity that could be restricted or traded. so there is a, there's a benefit in a country investing in their own token creation.
DAN WAIBEL
Absolutely. I think that's really the trend going forward, and I think there's multiple reasons for that. there's the economic side, but to your point, if you think of a country, they want to limit their critical dependencies. And if you're just renting the APIs in the cloud, you have a dependency.
So if you start to build out your national infrastructure and you have this dependency, it creates a lot of risk.
But if you build an AI factory, you have much more control. You have control over the cost, you have control from a security perspective, you have control from a privacy perspective, and then on top of that, the regulations are coming fast and furious, and they are also shifting, and they're regional.
So all these pieces together, I think, are gonna drive this trend of organizations standing up AI factories internally.
MICHAEL BIRD
rfrom a national infrastructure perspective, 'cause there's data and there's so many things else, so many other things involved.
Actually, there is value in having something within your country or within your state or whatever
DAN WAIBEL
the landscape is genuinely messy at this point in time. Europe has its own set of regulations. The US is much looser at the federal level. In some cases, really, there's no rules at all. But then you got state-level data privacy laws within the US that differ from one another. Brazil just released a new set of regulations.
and there are, other locations that have nothing at the moment, but there are regs in development
MICHAEL BIRD
Yeah Fascinating. I mean, It's the speed of change and the fact that governments can't bring regulations in that quickly. But, AI technology is just moving at such a rapid pace. The development of these frontier models is just insane, isn't it?
DAN WAIBEL
it's breakneck speed, in terms of how quickly the whole stack is changing on the software side. but I, will also say there's really a, cost of compliance in this regulated world. The cost of data moving to where it's allowed to be processed, the cost of getting it wrong.
So, data sovereignty in general has become a major factor in where the AI runs, and that's an economic decision as much as a legal one.
the cheapest compliant option is usually to bring the model to the data, not the data to the model
And there's structured data and there's unstructured data, right? So before generative AI, we were very focused on structured data. Think of rows and columns. But now with generative AI, the focus has been on unstructured data. So think documents, images, videos, right? That's what we call unstructured data.
And there's a significant amount of unstructured data within organizations, and oftentimes that data is not well organized. And I think that is really the biggest driver on the data side. It's the unstructured data.
So I think there's multiple things to consider, when you think about AI and you think about, an AI factory, because moving all of that heavy data has a cost,
MICHAEL BIRD
I mean, we're talking like 12 months, maybe even six months, but how have we seen strategies from organizations?
change, and evolve as the economics around token usage has changed?
DAN WAIBEL
I think economics is a major driver here because, first and foremost, we're still working through how do we optimize humans and AI working together. And agents are becoming, I think, much more normalized within organizations. but I, think what we've seen really in the last six months is this explosion of multi-agentic workflows, where we have orchestrators that call other agents, and then we have specialized agents that call other agents, and it creates this kind of network or web of agents, which is great because you can produce high-quality responses and answers.
you can figure out very complex questions you can get answers to.
DAN WAIBEL
And really these multi-agentic workflows, have really driven the cost of AI significantly higher, but they've also driven a much higher level of efficiency and productivity. So it's really this equation, ?
DAN WAIBEL
How much do you wanna invest to drive a higher level of efficiency, to drive more productivity from your team? And then the models continue to improve at the same time. So again, there are some financial decisions to make, but it's also gonna drive how competitive you are in the market. Because if your competitors are moving more quickly, you need to keep up, and there's gonna be a cost to that.
So I think that's what organizations are wrestling with that entire equation.
MICHAEL BIRD
do you think we'll see some sort of like token usage predictor? You know, you put your request into your model And it will say, "Okay, but before I do that, just so you know, this is gonna cost 10 million tokens, which has an approximate value cost at the moment of, I don't know, $55.
do you wish to proceed?" do you think we'll ever get to that stage? Because it feels a bit like, at the moment you're giving users blank checks.
DAN WAIBEL
It's a great point, and I have seen this concept discussed. I think the challenge is how accurate you can actually be, because if you look at the way the model works, as well as the way the kind of the entire agentic orchestrator works, it can be difficult to predict exactly which agent's gonna be called, how long those agents are gonna be run, and how high quality the response is going to be from each agent.
And the more complex tasks may launch hundreds of agents. So I think as we're able to estimate more accurately, I absolutely think there'll be more transparency upfront,
MICHAEL BIRD
'Cause these, these windows that we use to put these requests in, they seem so simple, but,
actually the cost to an organization, it could vary wildly, couldn't it, really?
DAN WAIBEL
It can, yeah. And particularly when you start to think of scaling this, across a number of different processes and having hundreds or thousands of different users leveraging these tools differently. and that's why tokenomics is so important and forecasting token usage is also difficult.
Things are changing so quickly in the space, right? it's difficult to predict what token costs are gonna be six, nine, 12 months from now. But we need to do that, as part of our typical budgeting and financing work. so yeah, it's gonna be critically important to companies and even to, CFOs now going forward.
MICHAEL BIRD
Do you think, AI tokens, they'll ever get, commoditized and traded gr-grain or oil do you think we'll see a stage where organizations will be able to offer out tokens and that's how as organizations you can predict token costs or, maybe you could lock in your token costs for the next 12 months the way, like you can lock in your electricity costs,
DAN WAIBEL
I think we, could eventually get to that stage. but right now I just think everything is changing so quickly on the innovation and the technology front that's what makes it so difficult to engage in a contract like the one you described. Because we really don't know where we're gonna be six or 12 months from now.
MICHAEL BIRD
and if you just think years ahead, eventually we'll have quantum computing, right? that's gonna change the equation completely. how do you think the AI economy is going to change? in the next three to five years,
DAN WAIBEL
from an economy perspective, physical AI, robots and robotics are gonna become much more prevalent as they become more affordable, essentially, and as they scale. So think about, for example, electric cars.
When they were first introduced, they were more expensive and they were not profitable. Now they're much less expensive, and certain companies have figured out how to make them profitable at scale. I think the same kind of pattern, will happen with robotics, And I think it'll be much more prevalent in, my guess is, inside of five years, maybe inside of three years, much of this conversation's gonna be focused around robotics because that's gonna be where the frontier's focused.
MICHAEL BIRD
And presumably that could change the balance of how many input tokens are being used. Because if you're running robotics, one would imagine that is lots and lots and lots of sensors, cameras, things like that,
So that's gonna change the economics of AI tokens again.
DAN WAIBEL
Yes, and I think the expectation is, gonna continue to evolve and will become much more efficient in terms of how we process tokens. And, I think the other piece is being able to run smaller models in a much more efficient way that are localized to, for example, a robot.
That's the space that's evolving, already starting to evolve very quickly, and I expect that to accelerate. And then again, you, you layer in other technological revolutions, that I think are also gonna drive additional change like quantum. so it is difficult to predict, but I think robotics is really the next frontier
MICHAEL BIRD
Yeah, I agree with you. Dan, thank you so much. This has been an absolutely fascinating conversation. really appreciate you coming to, to chat to us, on "Technology Now."
DAN WAIBEL
Also, Michael, well, thank you so much for having me. Enjoyed the conversation, and look forward to talking to you again soon
MICHAEL BIRD
just like the part one we did, that was just chock-full of information. I think the big theme, for that interview was around AI factories.
and we've done stuff on AI factories before, but I've never really thought like, what is an AI factory actually producing? And I suppose the answer here is an AI factory is producing tokens. that is the output of a factory. And, just like factories that are producing physical goods, you think a lot about, where in the world are factories placed?
do you put the factory close to the, where the raw minerals or the raw materials are being produced? Or, do you put it close to a source of electricity? Do you put it close to a, financial center? do you put it into a country where there are favorable regulations?
And I suppose all of that is the same with an AI factory. You would imagine that you'll probably think about a lot of those things as well.
SAM JARRELL
Yeah, it's kind of funny to me. the conversation reminded me a lot of, real estate conversations. do you want your home to be in a good or bad neighborhood? Or, is there an AI equivalent of the moment when renting stops making sense and buying becomes more attractive?
MICHAEL BIRD
Yeah, there, there's an element of that, isn't there? And one of the other things that, we talked about was the sort of regulatory environment for AI, within the EU, there's one set of regulations.
In the US,there's not a lot at federal level.but then at a state level, different states have got different regulations.
I think there are some states that have put some restrictions on how you can talk about things that you create using AI. and because one state has said, "This is what we want you to do," we're just doing it blanket across the whole organization.
So I think many organizations will be trying to figure out how to make sense of this, particularly organizations like HPE, where we have, staff and customers and offices in lots of different countries.
SAM JARRELL
Yeah,you bring up a, a great point. I think that many of the, regulations impacting, you folks across the pond and near Europe and whatnot, because global companies operate globally, many of the, the newer regulations are basically like, if you have people employed at your global company, then these have to, apply, at the company level.
And so there'll be a lot to navigate if more and more and more of these countries come up with their own sort of intertangled web of regulations, and maybe some of those regulations will, conflict with each other.
then those regulations also probably impact, the costs of using AI and, AI factories, and that gets back to the, the question before around is it better to have some of this located, within an organization? Does that even matter if it's located within an organization, if there isn't actual physical infrastructure?
Like what is the line for a lot of these things? And then how will businesses and organizations determine how to be the most, effective, in terms of like pricing out what they choose to do?
MICHAEL BIRD
Yeah. And maybe what we're talking about here is for some organizations, the cheapest option from a compliance perspective will just bring the model to the data, i.e. an AI factory essentially
SAM JARRELL
And on the note of cost,
I hear so many just like this many tokens, but you don't really see the conversion rate, right?
MICHAEL BIRD
Yeah. So, I put this into an AI tool.
just to give you a sense of scale. So for some of the, the big frontier models,
from an input token perspective, you're looking at, say, between $5 and $10 per million tokens, and from an output token perspective, it ranges between $10 to $50.
And then you go all the way to the other end of the scale, so the sort of open source models that we talked about in the previous episode. That's something in the range of, 0.5 of a cent, per million input tokens and about 20 cents per million output tokens.
So the scale is quite vast between those two.
SAM JARRELL
Yes. I think this goes a little bit back to, last week's conversation ever so slightly of maybe some things should still be done in traditional search. And then for other things, uh, we would need to optimize for, okay, it makes sense to run this on AI because the costs here, you hear about it compounding so, so much, but putting it in real dollars and cents, makes me realize, oh gosh, I wonder how much money I've actually spent.
MICHAEL BIRD
Yeah, and I do think if there was a counter on the AI tools that we use, I do wonder if you or I would use those tools in a different way, if you know what the, the potential cost is.
MICHAEL BIRD
Now, Sam, I think I asked you this question, ages ago on the show, but, do you say please and thank you to your voice assistants
SAM JARRELL
Oh, absolutely. Absolutely. Someday those things may have, AGI, and they're gonna know that I was one of the good ones.
MICHAEL BIRD
You are one of the politest people I know. so, it does not surprise me that you say please and thank you. I also do that mainly because I've got kids and, I am currently teaching my children to say please and thank you. I sort of have to practice what I preach.
but given the potential token usage over time by adding unnecessary processing, I, I did actually want to ask Dan, should I stop saying please and thank you to my AI assistants?
DAN WAIBEL
it's really just a fraction of a token,the please and thank you. There's certainly been much written about you shouldn't write please and thank you. And then there's actually some more recent studies that were published that if you treat your model, positively, you will get better responses.
So, I guess in my opinion, saying please and thank you and the cost being really a token or a fraction of a token, I think it's certainly worth the investment
SAM JARRELL
Okay that brings us to the end of Technology Now for this week.
Thank you to our guest, Dan Waibel, again.
And of course, to our listeners.
Thank you so much for joining us.
MICHAEL BIRD
If you’ve enjoyed this episode, please do let us know – rate and review us wherever you listen to episodes and if you want to get in contact with us, send us an email to technology now AT hpe.com
Subject line, sam?
SAM JARRELL
Rent or buy
MICHAEL BIRD
and don’t forget to subscribe so you can listen first every week.
Technology Now is hosted by Sam Jarrell and myself, Michael Bird
This episode was produced by Harry Lampert and Eva Higginbotham with production support from Nik Damarell, Beckie Bird, Alissa Mitry, and Jenessa Ayache. Our theme music was composed by Greg Hooper.
SAM JARRELL
Our social editorial team is Rebecca Wissinger, Judy-Anne Goldman and Jacqueline Green and our social media designers are Alejandra Garcia, and Ambar Maldonado.
MICHAEL BIRD
Technology Now is a Fresh Air Production for Hewlett Packard Enterprise.
(and) we’ll see you next week. Cheers!
SAM JARRELL
Bye y’all