Perplexity of Cloudfare

Most people think the Cloudflare-Perplexity fight is about ethics, but it's actually about something much more interesting:

it is about the future of how value gets captured in AI.

Perplexity, an AI search company worth $18 billion, got caught using stealth crawlers to scrape websites that explicitly blocked them. When Cloudflare called them out, Perplexity fired back, accusing the cybersecurity giant of "fundamental incompetence."

But this isn't a story about web crawling ethics or technical competence. It's about a more important economic question: when machines can learn from human creativity at infinite scale, who gets paid?

Real Battle

This is the collision of two internet economies.

The old internet was built on a simple trade: websites publish content for free, search engines index it respectfully, and everyone benefits when users discover that content. Google became worth $2 trillion largely by being the best middleman in this arrangement.

The new internet that Perplexity represents works differently. Instead of sending users to websites, it reads everything, synthesizes the information, and serves answers directly. Users get what they want faster. Content creators get nothing except server bills.

This isn't necessarily evil, it's probably better for users. But it breaks the economic model that made the web possible.

Why Infrastructure Always Wins

Here's what's interesting about Cloudflare's response: they're not trying to stop AI. They're trying to tax it.

Cloudflare sits between roughly 20% of the internet and everyone trying to access it. Instead of just blocking AI crawlers, they've built a "pay per crawl" system that essentially turns web scraping into a micropayment platform.

This is brilliant positioning. While AI companies burn venture capital building models, and content creators fight over fair compensation, Cloudflare has quietly made itself the toll booth operator for the entire conflict.

They don't need to pick sides. They just need to facilitate the transaction and take a cut.

When new technology disrupts existing business models, the companies that capture the most value aren't usually the ones building the flashy new products. They're the ones building the infrastructure that makes those products possible.

During the California Gold Rush, most miners went broke. The people who got rich sold picks and shovels. During the dot-com boom, most startups failed. The companies that got rich were the ones selling infrastructure, Cisco with networking equipment, Oracle with databases.

AI boom is following the same script. While everyone focuses on which AI model is best, companies like Cloudflare are building the economic infrastructure that all AI companies will eventually depend on.

What This Means for Startups

If you're building anything that depends on web data, pay attention to what's happening here. The rules are changing in real time.

The old rule was: if information is publicly posted, anyone can learn from it. The new rule being negotiated is: if information is publicly posted, the creator gets to control who learns from it and how much they pay.

This shift is going to happen no matter what . When humans read your content and learn from it, they might link to it, share it, or become customers. When AI reads your content and learns from it, you get nothing except the cost of serving that data to machines that will never buy anything.

Scale this across millions of articles and billions of requests, and you can see why the old model is breaking down.

What's really interesting about Perplexity's approach is how it reveals the weakness of traditional competitive defensibility. Content creators thought their moat was exclusive access to information. AI companies are proving that you don't need exclusive access, you just need to be better at synthesis.

Craigslist didn't need journalists to destroy newspaper classified ads. Netflix didn't need movie theaters to disrupt Hollywood. And AI companies don't need to create original content to capture value from information.

They just need to be better at organizing and presenting it.

New Equilibrium

In five years, this dispute will seem quaint. We'll have new norms, new pricing models, and new types of companies we haven't imagined yet.

But the underlying economic forces will remain. Technology keeps making information cheaper to access and process, which benefits users but can devastate creators who depended on information scarcity.

The companies that figure out how to make this transition work, creating sustainable value for both AI developers and content creators, will build the most valuable businesses of the next decade.

Right now, that infrastructure doesn't exist. Someone needs to build the payment rails, the licensing marketplaces, and the technical standards that will make this new economy function efficiently.

That opportunity is probably bigger than most of the AI applications being built today.

Real Lesson

Cloudflare-Perplexity fight isn't really about web crawling or AI ethics. It's about power—who gets to set the rules for how the digital economy works in the age of artificial intelligence. The companies fighting today aren't just building products. They're building the economic infrastructure that will determine how knowledge, creativity, and intelligence get valued for the next twenty years. And like most infrastructure, once it's built, it becomes very hard to change.

Right now, while everyone argues about the ethics of web scraping, the real decisions are being made quietly. Server configurations. Legal terms of service. API pricing models. The boring stuff that nobody pays attention to.

These technical choices will matter more than all the AI breakthroughs combined.

Because in the end, it doesn't matter how smart your AI is if you can't afford to feed it data. It doesn't matter how revolutionary your product is if the infrastructure layer decides you can't access it.

The future isn't being decided in boardrooms or congressional hearings. It's being decided by engineers writing robots.txt files and lawyers drafting API terms.

Which is exactly how the future usually gets decided, quietly, by the people building the roads that everyone else has to drive on.