Who's Building the Agent-First Web: Market Players and Competitive Dynamics Reshaping Internet Infrastructure
- Bradley Slinger
- Sep 21
- 8 min read
Updated: Sep 23
The race to build infrastructure for AI agents has created a new competitive landscape where traditional boundaries between web services, cloud providers, and software companies are dissolving. From well-funded startups building specialized platforms to tech giants embedding agent capabilities into their existing ecosystems, the market is rapidly evolving around a fundamental question: how do you serve customers who are machines, not humans?
In this series:
The SEO Revolution: How AI Agents Are Redefining Search and Digital Marketing
Who's Building the Agent-First Web: Market Players and Competitive Dynamics Reshaping Internet Infrastructure
The Future of Web Economics: New Business Models for the Age of AI Agents
From robots.txt to AI Regulation: How Web Standards and Governance Are Evolving for Machine Consumers
What Happens When Machines Dominate the Web: Future Scenarios and Current Barriers to AI Agent Adoption
The Infrastructure Pioneers
E2B: Cloud Computers for AI Agents
E2B has positioned itself as the "open-source standard for sandboxes for agentic workflows," providing secure, isolated cloud environments where AI agents can execute code, access file systems, and run terminal commands. Founded in 2023, the company has achieved remarkable traction, with 88% of the Fortune 100 signed up on their platform by mid-2025.
Product Differentiation: E2B's approach centers on security, scalability, and low-latency startup times. Their sandbox environments support diverse infrastructure choices—from Kubernetes to Google Cloud, Azure, AWS, Cloudflare, and in-house VPCs. The company's open-source strategy aims to become the de facto standard for agent code execution, similar to how Docker became ubiquitous for containerization.
Market Traction: The company's growth trajectory is impressive: from hundreds of early adopters to hundreds of millions of cloud sandboxes initiated across more than half the Fortune 500. Their funding reflects this momentum, with a $3 million pre-seed, $11.5 million seed round in 2024, and a $21 million Series A in July 2025, bringing total funding to approximately $32 million.
Strategic Vision: E2B is extending beyond basic sandboxes toward a standardized interface allowing AI agents to seamlessly control diverse environments—from Linux interpreters and Windows VMs to Chrome browsers. This vision of universal agent interoperability could make E2B a critical infrastructure layer in the emerging agent economy.
Browserbase: The Eyes and Ears of AI
Browserbase represents a different approach to agent infrastructure, focusing specifically on web browser automation. Founded by Paul Klein IV in early 2024, the company provides cloud-based headless browser infrastructure that enables AI agents to interact with websites like humans—without the human interface.
Rapid Growth: Browserbase's funding trajectory illustrates the market demand for agent-focused web access. Starting with a $6.5 million seed round in June 2024, the company completed a $21 million Series A just nine months later, followed by a $40 million Series B in June 2025, reaching a $300 million valuation. By mid-2025, they had grown to over 1,000 paying customers in just 16 months.
Technical Innovation: The company's flagship project, Stagehand, translates natural language into browser automation code, dramatically simplifying agent development. Released in late 2024, Stagehand represents a key differentiator in making browser automation accessible to AI systems without requiring detailed programming knowledge.
Market Position: Browserbase addresses a critical gap in the current web infrastructure—enabling agents to interact with the existing HTML-first web while the ecosystem transitions to more API-centric architectures. Their success demonstrates the immediate market need for bridging solutions that work with today's web while preparing for tomorrow's agent-first infrastructure.
The Crawler Ecosystem
GPTBot: The Data Acquisition Leader
OpenAI's GPTBot has emerged as one of the most significant forces in the AI crawler landscape. Between May 2024 and May 2025, GPTBot's share of AI crawler traffic surged from 5% to 30%, with raw request volumes increasing by 305%. This dramatic growth reflects OpenAI's aggressive data acquisition strategy for training increasingly sophisticated models.
Market Impact: GPTBot is simultaneously the most frequently blocked bot in robots.txt files and one of the most explicitly allowed, indicating its controversial yet critical role in the ecosystem. Its dominance illustrates how AI companies are prioritizing data collection to maintain competitive advantages in model development.
Technical Approach: Unlike traditional search crawlers, GPTBot focuses primarily on content acquisition for training purposes rather than real-time indexing. This specialized function allows for different optimization strategies and interaction patterns with websites.
Googlebot: The Expanding Giant
Google's Googlebot remains the dominant web crawler, expanding its share from 30% to 50% of all AI and search crawler traffic by May 2025, with request growth of 96%. Google's unique position as both a search engine operator and AI model developer gives Googlebot strategic advantages in data access and utility.
Dual Purpose: Googlebot serves both traditional search indexing and Google's AI initiatives, creating synergies that specialized AI crawlers cannot match. This dual purpose helps justify the massive infrastructure investments required for comprehensive web crawling.
Infrastructure Scale: Google's existing global infrastructure provides significant cost advantages and technical capabilities that newer entrants struggle to match, reinforcing Googlebot's dominant position.
PerplexityBot: The Growth Phenomenon
PerplexityBot demonstrated the most explosive growth among AI crawlers, with a staggering 157,490% increase in requests between May 2024 and May 2025. This growth reflects Perplexity.ai's strategy as an AI-powered conversational search engine that requires real-time web data to provide current, cited answers.
Differentiation Strategy: PerplexityBot's focus on real-time information gathering for conversational search represents a different approach from training-focused crawlers, emphasizing fresh content over comprehensive archives.
Market Validation: The explosive growth suggests strong market demand for AI-native search experiences that combine web data with conversational interfaces, validating Perplexity's strategic positioning.
Big Tech's Agent Strategies
Microsoft: The Ecosystem Integrator
Microsoft has positioned itself at the center of the "age of AI agents" through its 2025 Build conference messaging and strategic product integrations. The company's approach focuses on embedding agent capabilities throughout its existing software ecosystem rather than building standalone agent platforms.
Integration Strategy: Microsoft is integrating agent functionality into Dynamics 365 with Copilot, enabling AI agents to operate within familiar business software environments. This approach leverages Microsoft's existing enterprise relationships and reduces adoption friction.
Platform Vision: Microsoft's vision of an "open agentic web" emphasizes interoperability and standards, potentially creating competitive advantages through ecosystem effects while maintaining openness.
Google: Agent Space and Collaboration
Google's Agent Space platform, launched in April 2025, enables businesses and developers to build AI agents that work together across organizational boundaries. The platform uses an open Agent-to-Agent (A2A) protocol, emphasizing collaboration and interoperability.
Technical Innovation: The A2A protocol represents Google's attempt to standardize agent communication, potentially creating a moat around Google's infrastructure while maintaining the appearance of openness.
Market Strategy: By enabling cross-organizational agent collaboration, Google positions itself as the neutral platform for agent interaction, similar to how it positioned itself as the neutral search engine for human web navigation.
OpenAI: The Autonomous Pioneer
OpenAI's ChatGPT Agent, unveiled in July 2025, represents a significant leap toward autonomous AI systems. Integrating Operator and Deep Research functionalities into a unified system, the ChatGPT Agent can navigate web interfaces, manage calendars, complete forms, and conduct advanced research.
Capability Expansion: The ChatGPT Agent's ability to handle complex, multi-step workflows marks a shift from conversational AI toward truly autonomous operation, setting new benchmarks for agent capabilities.
Competitive Positioning: By advancing agent autonomy, OpenAI maintains pressure on competitors to develop similar capabilities, potentially creating winner-take-all dynamics in the most advanced agent categories.
IBM: Enterprise Agent Networks
IBM's tools and frameworks for building and managing networks of AI agents, announced in May 2025, focus on enterprise applications. Pre-built domain agents for HR, procurement, and sales, integrated with AWS Marketplace technologies, demonstrate IBM's enterprise-focused strategy.
Enterprise Focus: IBM's approach recognizes that enterprise agent adoption requires specialized tools and workflows, creating opportunities for companies that understand enterprise requirements.
Partnership Strategy: Integration with AWS Marketplace illustrates how traditional enterprise vendors are partnering with cloud providers to deliver agent capabilities through existing procurement channels.
The Detection and Security Layer
The Bot Detection Challenge
As AI agent traffic has grown, so has the sophistication of bot detection systems. Companies like DoubleVerify reported that general invalid traffic (GIVT) rose by 86% in the second half of 2024, directly attributing the increase to AI crawlers.
Detection Evolution: Traditional CAPTCHA systems are proving inadequate for distinguishing between legitimate AI agents and malicious bots. New approaches like WebBotAuth use cryptographic signatures for verification, though adoption remains limited.
Market Opportunity: The need for sophisticated bot management creates opportunities for companies that can provide nuanced detection and access control, balancing legitimate AI access with security requirements.
Funding Trends and Market Signals
The venture capital community has shown strong enthusiasm for agent infrastructure companies, with several key trends emerging:
High-Growth Valuations
Companies like Browserbase reaching $300 million valuations within 18 months of founding demonstrate investor confidence in the agent infrastructure market. Similarly, E2B's ability to raise successive funding rounds with Fortune 100 adoption validates the enterprise demand for agent platforms.
Specialized vs. Generalist Approaches
The market is showing room for both specialized solutions (like Browserbase's browser focus) and more general platforms (like E2B's sandbox approach). This suggests the agent infrastructure market is large enough to support multiple successful strategies.
Open Source Strategies
Several leading companies are pursuing open-source strategies to build developer mindshare and create de facto standards. E2B's open-source SDK and Browserbase's Stagehand tool both follow this pattern, suggesting that community building and standardization are critical competitive factors.
Partnership and Integration Patterns
CDN and Infrastructure Partnerships
The partnership between Fastly and TollBit illustrates how traditional web infrastructure providers are adapting to serve AI agent traffic. This collaboration aims to provide better bot management and create monetization opportunities for publishers dealing with AI traffic.
Technical Innovation: These partnerships are developing new technical approaches to handle the scale and patterns of AI traffic, including specialized caching strategies and billing mechanisms.
Market Evolution: Such partnerships suggest the infrastructure layer is becoming more specialized for agent workloads rather than simply scaling existing human-focused infrastructure.
Data Marketplace Integrations
Companies are increasingly integrating with data marketplaces to provide structured access to content. This trend reflects the need for clean, licensed data sources rather than simple web scraping.
Quality Premium: Organizations providing high-quality, structured data are commanding premium prices, creating incentives for better data curation and presentation.
Future Competitive Dynamics
Platform Consolidation vs. Specialization
The market appears to be developing along two tracks: large platforms that integrate agent capabilities into existing ecosystems, and specialized providers that focus on specific agent infrastructure needs. Both approaches show signs of success, suggesting a bifurcated market.
Standards Competition
Control over emerging standards like agent communication protocols, data access formats, and authentication mechanisms will likely determine long-term competitive advantages. Companies like Google and Microsoft are positioning themselves to influence these standards through their platforms.
International Competition
While current market leaders are predominantly US-based, international competitors are emerging, particularly in regions with different regulatory environments or data sovereignty requirements.
The competitive landscape for agent-first web infrastructure is still forming, with significant opportunities for both established tech giants and innovative startups. Success appears to depend on understanding that AI agents have fundamentally different needs from human users, requiring new approaches to infrastructure, security, and business models. Companies that can bridge the gap between today's human-centric web and tomorrow's agent-first infrastructure are positioned to capture disproportionate value in this transformation.
Sources:
Comments