Shifting Power Dynamics in AI-Focused Cloud
12:23 Share

Hi, and welcome to Gray Matter, the podcast from Greylock. This is the audio version of an essay written by Greylock partners Jason Risch and Jerry Chen. You can read this essay on our website, greylock.com, which is linked in the show notes. The Big Four Era, Shifting Power Dynamics in the AI-Focused Cloud. Big cloud is finally being disrupted. While the push for AI has made startups' reliance on AWS, Microsoft Azure, and Google Cloud higher than ever, it's also increased the entire ecosystem's reliance on NVIDIA.

Beyond its chokehold on GPUs, which may be temporary, and its numerous investments in partnerships with AI companies of all sizes, NVIDIA has also entered head-to-head competition with the big three with its development of the AI-specific DGX cloud. Collectively, these moves have established NVIDIA as a central power broker in today's AI revolution. This represents the first real shakeup among the cloud providers since they rose to dominance in the cloud platform shift.

Not only has NVIDIA been able to leverage advantages into critical partnerships with two of the three biggest cloud providers, it's also leveled the playing field for small and mid-sized computing companies. Thanks to partnerships with NVIDIA, these companies have been able to claim or reclaim sizable portions of the cloud computing market. As we wrote last year, NVIDIA's partnership with Oracle was instrumental in bringing the legacy company back into the modern age. Databricks and Snowflake,

Already the poster children of success of the cloud transition era, dove headfirst into AI via collaborations with NVIDIA and became powerful players in the AI computing ecosystem. Mosaic has continued to scale after Databricks acquisition, establishing the data platform as a leader in training custom language models for customers and producing state-of-the-art models like DBRX. The Snowflake team now provides a full stack AI platform.

These developments have enabled both Databricks and Snowflake to offer attractive distribution partnerships for AI startups as they leverage AI to continue challenging the big three.

Additionally, NVIDIA's partnerships and sizable investments have enabled a growing class of small, VC-backed, AI-specific cloud companies to thrive. Large rounds of financing and/or GPU allocation from NVIDIA has enabled companies like CoreWeave, Together AI, Crusoe, and Lambda Labs to acquire customers with flexible compute options and availability compared to the often constrained Big 3 clouds.

All of this shows we've entered the "Big Four" era of cloud. The Big Four collectively power the entire AI ecosystem and often lead the largest financing rounds. Each are racing to establish crucial partnerships with startups to build up their AI edge.

The record-breaking financing rounds are heavily tilted towards large language model providers. Microsoft alone put $13 billion into open AI in 2023. AWS recently invested $2.7 billion in Anthropic. And perplexity AI's rapid ascension to higher and higher valuations was achieved in part by backing from NVIDIA. These new power dynamics, funding patterns, incumbent challenger partnerships, and their collective impact on potential exit strategies mean startups must approach competition in the AI era differently.

Today, there are several areas where startups have the advantage, where more opportunity still exists, and areas where we believe incumbents are most likely to win.

Many of our original guidelines for competing with the hyperscale cloud still hold true, such as avoiding direct competition with the incumbents, moving up the stack, establishing deep IP, owning developer community, and so on. We've compiled a list of trends we've observed through our daily work as investors, through our conversations with founders, enterprise executives, and researchers, and through our ongoing data collection analysis for our Castles in the Cloud project, which is Greylock's interactive data project to map funding and opportunities in the cloud ecosystem.

As mentioned, the majority of investments are going into compute and foundation models, which means startups must look to other layers of the stack to build durable value. Here, we'll outline how startups are finding traction with tooling, infrastructure, and various approaches at the application layer, including vertical and horizontal apps, security, code gen, and robotics. We'll start from the bottom and work our way up the stack. Foundation Models: The Intersection of the Cloud and LLMs

While the engine of AI is fueled and powered by the Big Four, it's driven by startup-provided large language models. With massive funding rounds and often exclusive distribution deals, models from OpenAI, Anthropic, and Mistral have become deeply entrenched in the Big Four's AI strategy. Partnerships like Microsoft and OpenAI and AWS and Anthropic highlight the increasing concentration of power between the Big Four and LLMs.

Even Google, the outlier as the lone cloud provider developing its own foundation models in-house, has invested heavily in outside LLMs.

The necessary intersection of the big four in LLMs makes for a crowded field. While we track 10 new challengers in the LLM provider category in our Castles in the Cloud project, we don't expect this field to grow much larger. While we see some potential for domain-specific models to compete, as well as code-specific models that could serve as the basis for AI engineering co-pilots or agents, these are resource-intensive undertakings with significant unknowns still present.

Similarly, compute is another capital-intensive undertaking for startups, and which has already received considerable investment from the big four and venture capital investors. As mentioned earlier, partnerships and investments for NVIDIA have allowed smaller AI-specific clouds to thrive, and there has been an overall boom in funding for the sector. In our annual update to Castles in the Cloud, we tracked nearly $1.4 billion raised by computing startups in 2023. This represents a six-fold increase from the $232 million invested in 2022.

The trend continued well into 2024 with examples like Foundry. The company just came out of stealth with a $350 million valuation to continue developing its public cloud purpose-built for machine learning. For now, we encourage founders to approach opportunities higher up in the stack. Dev tools and infrastructure.

At this point in the AI transition, we see tremendous opportunity for startups building infrastructure. However, startups must focus on navigating the landscape. As the capabilities offered by the foundation models improve, the cloud providers develop additional services, shifting how applications prefer to consume infrastructure.

Therefore, infrastructure must be built to account for this quicksand, but also must be agile and developer-focused to avoid the quicksand themselves. For example, investors and AI builders alike have debated the long-term durability of standalone vector stores. Standout companies like Pinecone and Weeviate raised a combined $150 million last year and have attracted many customers, while other customers have preferred add-ons to existing databases like PG Vector and MongoDB Atlas Vector Database.

As LLM-based applications are poised to become the dominant application paradigm over the next decade, we believe the data component of the stack represents the best opportunity to build significant product depth and IP. Developers need a way to connect to enterprise data and to tailor an open source model to their specific needs, which has given rise to a class of startups like Lomit Index, providers of a data framework to easily build custom RAG applications.

We also see significant demand for tools to make open source models more effective, given the attractive control, performance, and cost open source models offer. Companies like Predibase make it possible for developers to build smaller models with better latency and performance by fine-tuning open source models. The company recently released Loraland, a collection of 25 fine-tuned Mistral 7 models reported to consistently outperform base models.

There are also opportunities for startups to address one of the primary challenges of working with LLM applications. Since they are non-deterministic, enterprises cannot reliably predict the quality of their responses, nor whether a small change in the prompt due to the use of proprietary data could impact the output.

Observability tools have become required products in the age of AGI, as the strength of a startup's IP modes is only as good as their performance. Recently, we invested in Braintrust, which offers developers a toolkit for instrumenting code and running evaluations, enabling teams to assess, log, refine, and enhance their AI-enabled products over time. The emergence of agentic capabilities underlies many of the opportunities at the application layer. Many leading AI application companies are building their core IP at the agentic layer,

orchestrating a number of third-party and fine-tuned models, as well as domain-specific tools to reason out different tasks. We've seen this paradigm begin to play out across several vertical and horizontal domains as well as security, root cause analysis and observability, AI engineers like Devon, and more. This approach has led to some of the fastest growing companies and provides a deeper moat than some of the past wrappers on top of LLMs. While we've seen a number of agent frameworks, most companies we've spoken to end up largely building their own orchestration system.

Adept, which is developing custom AI agents for the enterprise, has taken this tactic. Over time, we believe this approach has become more standardized while also reliably providing industry-specific utility. In the interim, there is opportunity in providing agent-specific tools. The application layer. Vertical and horizontal place.

Continuing last year's trend of increasing vertical specialization, we're seeing more startups carving out space for themselves with ML apps. We added 10 new entrants to the category in our Castles in the Cloud Market Map section, including a mix of vertical and horizontal apps.

As our colleague Christine Kim wrote, vertical-specific AI approaches are showing to be an attractive option for startups and don't appear to be an area of focus for the big three cloud providers. For example, legal-focused startups Harvey and EvenUp launched with funding in 2023, while existing companies in the database such as Adept, Ramp, and Tome also raised new funding.

We expect to see more horizontal apps for functions common to all enterprise organizations, such as sales and recruiting, as companies look to supplement or automate less strategic tasks in those domains. However, it's unclear how much the truly core horizontal categories will be disrupted by AI startups. Unlike the incumbents of past generations, which struggled to adapt to the cloud transition, today's incumbents, so far, seem to be doing a far better job evolving with the latest platform shift.

Security. Security continues to be an all-important issue, and the new risks associated with AI have accelerated the pace of new company creation. In 2023, we added 13 new startups in the Castles in the Cloud database, and the sector racked up nearly $800 million from venture capital investors.

We spoke with dozens of CISOs and data teams leveraging GenAI in conjunction with multiple cloud data platforms. Again and again, they spoke of the need for a new approach to cloud data security to ensure protected information such as regulated data and core IP is not ingested into AI models. We incubated and invested in Bedrock, which has developed a frictionless data security as a service platform tailored to the GenAI era.

Security startups are also using AI agent approaches to revolutionize SOC identity and improve remediation in apps and infrastructure. Kodum, which Greylock invested in last year, leverages deep runtime intelligence to understand true application risk. To provide users with a complete picture of possible vulnerabilities to applications in use, as well as to protect potentially exploitable functions in the future, the company merges its runtime intelligence capabilities with LLM-based techniques.

DAS, which Greylock has been partnered with since the company's seed, provides a platform to minimize and monitor security risks throughout the development pipeline. CodeGen. One of the most exciting opportunities for startups today is in developing AI tools that can understand and speak code just like a human engineer. Our colleague Corinne Riley is heading up our work in this area and is focused on three primary approaches to this goal.

AI co-pilots that enhance existing workflows, AI agents to replace engineers, and code-specific models train on a mix of code and natural language. In 2023, we added the CodeGen category to the Castles in the Cloud database and tracked nine new startups. You can read more about Greylock's thesis on the submergence sector in Corinne's upcoming essay. Robotics.

Foundation models have also provided a boon to robotics fundings in our dataset, as investors recognize the promise of advances in robotics and hardware combined with foundational models. We expect the trend will accelerate further this year. FIGURE recently raised $675 million from OpenAI, NVIDIA, Microsoft, and others, and announced a partnership with OpenAI to develop robotics foundational models that allow them to process and reason about language.

Physical intelligence raised $70 million to bring general purpose AI into the physical world based on the founders' research at Stanford. If you're building at the intersection of AI and robotics, we'd love to chat. The AI revolution has created a new world order.

Startups navigating this increasingly competitive landscape will undoubtedly turn to the big four for resources and partnerships, but can carve out space for themselves at higher layers of the evolving AI stack. From apps to agents, security to code gen, and robotics, we're eager to work with founders who are building where the incumbents prefer to outsource innovation to the challengers.