How AI Companies Are Using AI
18:36 Share

Today on the AI Daily Brief, how AI companies are actually using AI. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI.

Hey, hello, friends. Quick announcements. First of all, thank you to today's sponsors, Blitzy, Plum, and Super Intelligent. To get an ad-free version of the show, go to patreon.com slash ai daily brief, where it starts at just $3 a month. And a quick note, due to some last minute travel, I am recording this one a little in advance. That means as often happens with these types of shows, we will be doing no headlines today. So if there's some crazy thing that happened, that's why I'm not talking about it here.

We'll be back tomorrow with a normal episode and then off for July 4th on Friday. And next week should be completely normal once again. Still, I've been really looking forward to talking about this report because whereas most of the reports that we have context to talk about on this show are from an enterprise vantage and enterprise adoption, which of course matches a lot of what you guys are doing. This is a report that's all about how the companies that are building AI are using AI and

And I think that there's a sort of do what I do, not just what I say kind of thing that makes that really valuable. The report comes from Iconic, who is a wealth manager and investment firm. And to cut to the chase, as much as it's distinct here and really useful, a lot of the big themes are very similar. In other words, we are firmly out of the pilot and experimentation stage. We are experimenting with new business models. Spend and budgets are increasing. Talent really matters. Really similar themes that you've heard me talk about in the past, but

but from companies that are frankly best positioned to know.

So what are the types of companies that are included in this? Despite these all being AI builders, there's actually quite a range. On the one end of the spectrum are AI-enabled companies who are just adding AI capabilities to their existing products. So this is something like Atlassian, who has work management software, that's building an AI kind of version of that. That represents about 31% of these 300 survey respondents. Another category of AI-enabled are the companies who are creating a new AI product,

that isn't their core product. So that's like Salesforce, where agent force is obviously a huge priority for them, but it's still not the core Salesforce product. That group represents 37% of the respondents. The last category are the AI native companies who are like 11 labs where their whole focus is their core AI product. And that's 32% of the respondents.

Unsurprisingly, the companies for whom their AI product is their main product are a little bit farther along in the development of their AI product. Whereas 34% of the AI-enabled companies have their AI products still in beta, only 10% of the AI-native companies are stuck in beta. Both categories have around 42% general availability of their AI products. But whereas only 13% of AI-enabled companies are scaling their AI products, a full 47% of the AI-native companies are scaling their AI products.

When it comes to what they're building, it is the big themes, man. It's agentic workflows, vertical AI applications, along with horizontal AI applications, AI platforms and infrastructure, and core AI models. Still 62% of the AI-enabled companies were working on agents in some form or fashion, and 79% of the AI-native companies were as well. One really interesting question, certainly from the standpoint of those of you who are in big companies who are trying to make these types of decisions, was what models they use.

And the clear answer is a number of them.

In fact, the average number of models per respondent was 2.8, meaning that to the extent that your company is laboring over a decision around which model to use, we may be at a stage where trying different models for different purposes may be the play. OpenAI was by far the leader, with Anthropic in second place, and Google and Meta being in pretty close quarters for third place. Mistral, DeepSeq, and Cohere also had some representation as well. Interestingly, when it came to considerations for which model to choose,

accuracy was by far the top consideration. 74% of companies surveyed had accuracy in their top three considerations. On the other end of the spectrum, open source and vendor lock-in were very, very low. Only 9% of companies had open source as a key consideration,

and only 6% had vendor lock-in as a key consideration. Still one difference between 24 and 25 that was really interesting is that whereas in 24, cost was actually the lowest top consideration, this year it was number two with 57% of companies ranking it as a top three consideration. Now, I think that that big jump reflects the fact that we have moved out of the experimentation phase and into the full production phase. In other words, when you are just experimenting and in beta tests,

cost is a lower priority compared to just making the thing work. But when you're actually scaling a product that's going to have tons of usage, boy, does cost make a big difference. On top of that, we also have external factors like lower cost models like DeepSeek coming in and competing, creating the opportunity for cost to be a consideration.

When it came to the challenges they found with different models, I think the list will be pretty familiar to anyone interacting with AI. Hallucinations were at the top of the list, followed by explainability and trust, improving ROI, compute cost. An interesting one, which we'll come back to in a little bit, was a quarter of the respondents had listed finding the right use cases as a top three challenge, which is interesting considering that these are companies who are building AI models and there's still a quarter of them finding the right use cases be a challenge.

What about the big theme, agents? The TLDR is that agents are here, baby. Iconic separated the groups into high growth and all other companies. And among the high growth companies, nearly half of them, 47%, were actively deploying AI agents in production, with another 42% experimenting with AI agents in pilots or internal use cases.

Even among all other companies, the non-high growths of full around a third, 32%, were actively deploying AI agents with another 32% in the pilot or experimentation phase. I thought that that 32% in deployment number was interesting because that's almost exactly the same number that KPMG's quarterly pulse survey found of big enterprises putting AI agents into deployment, which was itself a huge jump from the previous quarter where it was just 11%. Now, how are companies making all these decisions?

The TLDR is that once companies reach a certain scale or size, basically 100 million, they start to more frequently have dedicated AI leadership. And that percentage just goes up the bigger that they get. Across all sorts of different AI-specific roles they are hiring, at the top end are technical roles like AI engineers and data scientists.

But you're also seeing prompt engineers still being hired, AI design specialists being hired, AI product managers are a major category. 46% say that they're not hiring fast enough, and of those, 60% say that hiring is too slow due to a lack of qualified candidates.

Interestingly, the biggest cost center for companies is around talent. Now that does go down over time as products scale, but it still represents a big chunk of the cost of these AI products. For example, among the companies that are actively scaling their AI products, a full 36% of their AI budget is allocated to salaries, hiring, and upskilling. That compares to, for example, 12% for AI model training and 10% for AI model inference.

When it comes to which costs are hardest to control, there are a lot of things that people are considering. Storage costs, training costs, model retraining, inference costs, all have between 40 and 50% of respondents ranking them in the top three most challenging costs to control. But by far the leader in this category is API usage fees, with a full 70% ranking it as a top challenge to control that cost.

Now in terms of how they're trying to control those costs, interestingly, although they didn't rank open source particularly important to them when it came to model consideration, a full 41% said that they're interested in moving to open source models to help control costs.

Now, one really interesting thing about this study that is maybe a little bit different than what we saw in the enterprise sphere is what they're trying to get out of their AI usage. By far, the most tracked ROI category was productivity gains. 75% of organizations are measuring the impact of AI by looking at productivity gains. Another 51% are looking at cost savings. That compares to just 20% who are considering revenue uplift.

Now, this is different than what we saw with the recent KPMG quarterly pulse survey that found 46% of big enterprises equally split between thinking about productivity and efficiency as a goal of AI and revenue growth and new opportunities as a goal of AI. I think that this reflects the fact that many of these companies are just a couple years old.

And whereas those legacy players have much more room for business model disruption and transformation, these companies are just finding their business model for the first time. They're not in the transformation and disruption mode, but what they are is looking to do whatever it is that they're doing more effectively. Budgets for internal AI productivity are set to nearly double in 2025, with

with companies spending on average between 1% and 8% of total revenue on it. And one really important note that I thought was fascinating, and super reflective of what we're seeing in the enterprise space as well, is that the budget for that internal productivity AI is increasingly coming from existing areas like R&D and business units, but not from some innovation budget. Between 24 and 25, the

The percentage coming from innovation budget went from 47% to 23%, reflecting, I think, the move, again, away from the experimentation phase into the full deployment phase. This episode is brought to you by Blitzy. Now, I talk to a lot of technical and business leaders who are eager to implement cutting-edge AI. But instead of building competitive moats, their best engineers are stuck modernizing ancient code bases or updating frameworks just to keep the lights on.

These projects, like migrating Java 17 to Java 21, often mean staffing a team for a year or more. And sure, co-pilots help, but we all know they hit context limits fast, especially on large legacy systems. Blitzy flips the script.

Instead of engineers doing 80% of the work, Blitzy's autonomous platform handles the heavy lifting, processing millions of lines of code and making 80% of the required changes automatically. One major financial firm used Blitzy to modernize a 20 million line Java code base in just three and a half months, cutting 30,000 engineering hours and accelerating their entire roadmap.

Email jack at blitzy.com with modernize in the subject line for prioritized onboarding. Visit blitzy.com today before your competitors do. Today's episode is brought to you by Plum. You put in the hours, testing the prompts, refining JSON, and wrangling nodes on the canvas. Now it's time to get paid for it.

Plum is the only platform designed for technical creators who want to productize their AI workflows. With Plum, you can build, share, and monetize your flows without giving away your prompts or configuration. When you're ready to make improvements, you can push updates to your subscribers with a single click.

Launch your first paid workflow at useplum.com, that's plum with a B, and start scaling your impact. Today's episode is brought to you by Superintelligent, specifically agent readiness audits. Everyone is trying to figure out what agent use cases are going to be most impactful for their business, and the agent readiness audit is the fastest and best way to do that.

We use voice agents to interview your leadership and team and process all of that information to provide an agent readiness score, a set of insights around that score, and a set of highly actionable recommendations on both organizational gaps and high-value agent use cases that you should pursue. Once you've figured out the right use cases, you can use our marketplace to find the right vendors and partners. And what it all adds up to is a faster, better agent strategy.

Check it out at bsuper.ai or email [email protected] to learn more.

To shill aggressively for a minute, this is exactly why we designed the agent readiness audits. No one wants to spend a bunch of time figuring out what exactly to use these tools, which are so clearly so powerful for. They just want to be able to use them to get value. It's why we try to hack down the time to use case discovery in order to have people spend more time just getting that actual value. Now, maybe unsurprisingly, the more companies use AI, the more use cases they find.

For companies that had greater than 50% of their employees actively using AI tools, they had an average of 7.1 different use cases.

The most popular use cases are probably what you'd imagine. Coding assistance at 77%, content generation and writing at 65%, documentation and knowledge retrieval at 57%, product and design at 56%, sales productivity at 45%. One that might be a little bit lower than you think is customer engagement and customer service, which had 42% listed as a top use case. But once again, I think that reflects a slightly different demographic of companies rather than it being not actually that popular.

In other words, these companies are very, very young in their life cycle. They tend to have less sophisticated customer service organizations. And I think that probably explains more why they are deploying that use case less frequently.

And when it comes to which use cases are having the biggest impact in terms of productivity, the vast majority are seeing productivity gains between 15 and 30%. Coding assistance, on the other hand, was by far the top-ranked use case when it came to impact on productivity. In fact, for those high-growth companies, an average of 33% of their total code is currently being written by AI. So more than what we've heard from Google and Microsoft. But across all the other companies, even the ones that aren't high-growth,

27% of their code is being written by AI. This is just so clearly the biggest breakout use case so far, and one that's having a huge impact right now. One area that continues to be an area of experimentation is around the pricing model. 36% of companies are still using primarily a subscription or seat-based model, as opposed to just 19% who are usage-based or 6% who are outcome-based, but hybrid is now the most popular category at 38%.

I expect that we're going to see a lot more experimentation with business model, and we're going to see that usage-based and outcome-based number come up. In fact, 37% of respondents said that they do plan to change their AI pricing model in the next 12 months, thinking about things like consumption and outcome-based pricing, as well as factoring in ROI. Now, one section of this report, which really I can't do justice to as a podcast, but which you should absolutely spend some time with,

is the section titled AI Builder Tech Stack. This is where they look across every different domain, which tools companies are using. So for model training and fine tuning, LLM and AI application development, monitoring and observability, inference optimization, model hosting, model evaluation, data processing, vector databases, synthetic data, DevOps and MLOps, product and design.

If you are trying to build out your company's tech stack, go look at what the builders are actually using. A couple that I want to call out, at least at this stage, the coding assistance is a two-man race between GitHub Copilot, which had nearly three quarters of development teams using it, and Cursor, which is absolutely coming up on their heels with 50% of respondents already using it. We recently saw how Amazon developers are lobbying them to get rid of their internal tool and just use Cursor instead. It seems like that is happening more broadly as well.

One that I wanted to call out, though, because it's something that we've talked about a bunch on this show recently, is around model evaluation. When we were talking about the AI Engineering World's Fair, we talked about how evals are a growing topic of conversation among the builders, but how it's still really lagging behind in a way that seems almost destined to grow in focus in the coming months. That is certainly the case even among the builders.

The key takeaways from the survey around model evaluation were that there was no clear standalone leader. In fact, 20% of respondents didn't know which tool they used for evaluation, and around a quarter admitted to either not knowing or not having a tool in place. That is absolutely going to change and represents a huge opportunity for both companies to build evaluation tools, as well as for companies to get ahead of their competitors by being better at evaluation.

If you're looking to invest in an area to stand out, that is something to consider. Speaking of standing out, the last section of the report is a look at some of the key trends across different internal productivity use cases. And the biggest takeaway that I would say is that the incumbents really do have an advantage.

For example, in sales productivity, one of the key trends was: "Many teams are getting their AI-powered sales features straight out of Salesforce, indicating that an easy path is to lean on your existing CRM's built-in recommendations, forecasting, and opportunity scoring, rather than bolt on a separate service." In marketing automation, they write: "Marketers overwhelmingly turn to canvas-generative features for on-brand visuals and quick content iterations.

making it by far the most common AI touchpoint in the marketing stack. In customer engagement, teams overwhelmingly rely on Zendesk and Salesforce's embedded AI features for customer interactions, signaling that ease of plugging into existing ticketing and CRM workflows still beats adopting a standalone conversational AI platform. Where there is more flexibility and exploration of new solutions comes in areas which, if not being totally new, are so different in the world of AI that it really opens up new opportunities like knowledge retrieval and documentation.

Still, it really does bring up just how challenging it's going to be for all these vertical AI companies who have to compete against these legacy platforms that have such entrenchment even among the startups themselves. There is a ton more in this report.

And I really think that especially if you are in an enterprise who is trying to gut check and understand where you sit and how your adoption is going. In addition to looking at the enterprise focus surveys, go check out this builder's report. In some areas, it will probably scare you with how far ahead they are relative to agent deployments. But in other areas, they're struggling with some of the same things. Helping employees figure out how to actually use these tools, for example.

It's a confirmation of both how fast the industry is moving, but also that we're all in this together. And that even for the companies who are building the technology, much of these transformations are incredibly, incredibly difficult. As I said, I will include the link to this report down in the show notes. Big ups to Iconic for producing this. I think it's super valuable. And thanks, of course, to you guys for hanging out. Till next time. Peace.