You can only optimize what you measure

2026-09-11

Before you optimize anything for AI, measure it. You can only optimize what you're measuring, and right now most sites are measuring almost nothing about what AI does on them.

There are two sides to this. The first is the bots and scrapers coming to your website. The second is the people AI sends you afterward. This page is about the bots. The referrals are on the next page.

Bots don't run JavaScript

Most analytics products track people with JavaScript. Bots and scrapers don't run it. So the AI crawlers reading your site are invisible to the tools you'd normally look at, and the picture you get from them is of humans only.

To see the bots you have to count them at the edge, where your pages are served, not in the browser. We're building this now for the sites we host, from the request logs of the servers that hand out the pages. In the alpha I showed at the talk, it's three views:

  1. Which bots, how much, over the last two weeks. OpenAI, Anthropic, Perplexity, Google and the rest, with request counts. This is the baseline you can't get from a browser tag.
  2. Which content the AI crawlers are actually reading. The pages they fetch, ranked.
  3. Which content they are not. This is the one that matters. If an AI never read it, it can't cite it, recommend it, or send you a customer from it. That's even worse than falling to the second page of Google search.

Instead of paying someone to guess, you ask

Once you can see the bots, the question is what to do about it. This is where I think the whole GEO industry gets replaced by a chat window.

We built a chat window into our analytics. It's just a place to ask your data a question in plain language. In the talk I asked it: of my most-engaged content, which pages are AI crawlers ignoring, and what do those pages have in common? It came back with a list and a pattern. In that case: older dates, no clear headings, and no internal links from the pages the bots do visit.

Fix those, and next week you look at the bot numbers again and see whether they came back. That's the loop. Measure, ask, fix, measure again. Nobody needs to sell you a playbook for that.

A few sources I keep returning to are on the reading list.