Agent traffic
See which AI crawlers and assistants visit your website and which pages they retrieve. Learn how to connect your site and read your AI traffic.
Updated
When an AI crawler visits your website, it may retrieve information about your products, services, or articles. Other AI agents fetch pages in response to a specific user's question.
With "Agent traffic", you can see which AI agents visit your website, how often they visit, and which pages they retrieve.
It gives you a different perspective from your AI answer tracking: you're looking at requests made to your website, rather than just what AI says about your business.
Find "Agent traffic"
Go to "Traffic" → "Agent traffic" in the menu.
The page shows data for your currently selected brand.
If you haven't connected a website yet, you'll see the "See which AI agents read your site" card, which explains how to get started through your hosting provider.
Connect your website
Click "Connect a site" to get started.
There are two available connection methods:
| Method | How it works |
|---|---|
| "Vercel" | Sends logs from your Vercel project to Klarvu using a Log Drain. No code changes are required, but you need Vercel Pro or Enterprise. |
| "API / middleware" | Sends AI request information through your website's middleware or directly through an API. Works with Next.js and other platforms, and is available on any hosting plan. |
You'll also see "Cloudflare", "Cloudflare Enterprise", "CloudFront", "Akamai", and "WordPress" marked "Coming soon". These connection methods aren't available yet.
Choose your connection method
Click "Connect a site" and select either "Vercel" or "API / middleware".
Enter your domain
Fill in "Domain" with the website you want to track. The field is prefilled with your brand's website.
You can also use the "Name" field to give the connection a recognizable name, such as "Marketing site".
Then click "Continue".
Klarvu only counts requests to the specified domain and its subdomains.
Complete the setup
Follow the instructions for your selected method.
For "Vercel", you'll configure a Log Drain. For "API / middleware", you'll add the provided code snippet or send requests through the API.
Test your connection
Use the test command displayed by Klarvu to check that requests are reaching the service.
Once the connection is working, you can start seeing recorded AI visits under "Agent traffic".
Only owners and administrators can connect a website.
If you don't have the required permissions, you'll see "Ask an owner or admin to connect a site."
Set up a Vercel connection
If your website runs on Vercel Pro or Enterprise, you can use a Log Drain without changing your website's code.
Klarvu provides a "Drain destination URL" and an "API key header", both with "Copy" buttons. Each site has its own URL and key, so don't reuse them across sites.
You'll need to:
- In Vercel, open your team's "Settings" → "Drains", click "Add Drain", and choose "Custom HTTP endpoint" and "Logs".
- Name the drain, choose "Specific Projects", and pick the project that serves your website.
- Select the sources "Functions", "Edge Functions", "Static Files", and "Rewrites".
- Include at least the "Production" environment and leave "Sampling Rate" at 100%.
- Paste Klarvu's URL under "URL", select "JSON" under "Encoding", turn on "Custom Headers", and paste the provided header. Then click "Create Drain".
Once configured, Vercel sends your logs to Klarvu, which picks out the requests from AI agents.
Set up an API or middleware connection
If you're not using Vercel, you can select "API / middleware".
Klarvu provides a Next.js snippet that goes in proxy.ts (Next.js 16) or middleware.ts (older versions) at the root of your app.
You'll need to:
- Save the provided key as an environment variable named
KLARVU_AGENT_TRAFFIC_KEY. - Add the displayed snippet. If you already have a middleware file, just add the AI check to it.
- Deploy the change to your website.
AI requests are sent in the background, so collection doesn't slow down your website.
If you're not using Next.js, you can send JSON using a POST request to the ingest endpoint shown in Klarvu. Include your key as a Bearer token. You can send up to 500 requests per call.
Check that your connection works
During setup, Klarvu provides a curl command you can use to send a test request with the user agent "Klarvu-Testbot".
Once the test arrives, the status usually turns "Active" within a minute or two.
Test requests appear under "Recent agent requests" labeled "Test", but they're never counted in your stats.
Understand connection statuses
Under "Connected sites", you can check the status of your connected websites. Each row also shows the number of AI visits in the last 24 hours and the time of the last delivery.
| Status | What it means |
|---|---|
| "Pending" | Klarvu hasn't received anything from your website yet. The first data usually arrives within a few minutes of successful setup. |
| "Receiving logs" | Klarvu is receiving logs, but no AI visits have been detected yet. |
| "Active" | Klarvu is recording visits from AI agents. |
If your connection shows "Receiving logs", that doesn't necessarily indicate a problem. It may simply mean no AI agents have visited your website yet.
Owners and administrators can reopen the setup with "Setup" and disconnect a site with "Remove".
Understand your AI traffic
Once Klarvu starts recording AI requests, you'll see an overview of activity on your website.
Choose between "24 hours", "7 days", "30 days", and "90 days". The default is 30 days.
At the top, you'll find four key metrics:
| Metric | What it shows |
|---|---|
| "AI agent visits" | The number of recorded AI agent visits during the selected period, including the change compared with the previous period. |
| "Pages visited" | How many pages AI agents have visited, and how many different agents visited them. |
| "Top agent" | The AI agent responsible for the most recorded visits. |
| "Retrieval visits" | Visits where an AI assistant retrieves content in connection with a specific user's request. |
"Agent visits over time"
This chart shows how AI visits change over time, broken down by the company behind each agent.
It displays the six largest companies, with remaining traffic grouped under "Other".
This helps you notice whether activity is increasing from a particular AI provider.
"Top agents"
Here you can see the ten AI agents that visit your website most, the companies behind them, and when they were last detected.
"Most visited pages"
This section shows which pages AI agents visit most frequently. Use "Filter pages…" to find a specific page.
You'll find:
- "Agent visits": The number of recorded visits to the page.
- "Agents": How many different AI agents visited it.
- "Last visit": When the page was most recently visited.
This can help you understand whether AI agents mainly retrieve your homepage, specific product pages, or articles.
"Recent agent requests"
This table shows your 50 most recent recorded requests.
It includes "Time", "Agent", "Page", and "Status".
If no AI visits have been recorded during the selected period, you'll see "No AI agent visits yet" with guidance to check the connection under "Connected sites".
Why do AI agents visit your website?
Not all AI agents retrieve content for the same reason.
Under "Why agents visit", Klarvu groups traffic into three categories.
| Category | Purpose |
|---|---|
| "Training" | Collecting content that may be used to train AI models. |
| "Indexing" | Collecting content for an AI search product's or AI assistant's index. |
| "Retrieval" | An AI assistant retrieving content in connection with a specific user's question. |
Why is "Retrieval" particularly interesting?
"Retrieval" is especially useful because it shows that an AI assistant fetched content in connection with a user's current request.
Examples of retrieval agents include "ChatGPT-User", "Claude-User", and "Perplexity-User".
It tells you an AI assistant attempted to retrieve a page, but not whether that page was subsequently cited or recommended in the answer.
If you want to investigate the websites AI actually cites, use the Sources page.
Which AI agents does Klarvu track?
Klarvu recognizes the AI crawlers and assistants of all the major AI providers:
| Company | Agents |
|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai |
| Perplexity | PerplexityBot, Perplexity-User |
| GoogleOther, Google-CloudVertexBot, Gemini-Deep-Research | |
| Apple | Applebot |
| Amazon | Amazonbot |
| Meta | Meta-ExternalAgent, Meta-ExternalFetcher |
| ByteDance | Bytespider |
| Common Crawl | CCBot |
| DuckDuckGo | DuckAssistBot |
| Mistral | MistralAI-User |
| Cohere | cohere-ai |
| You.com | YouBot |
| xAI | GrokBot, xAI-Bot |
Frequently asked questions
Why am I not seeing any AI visits?
Start by checking your connection under "Connected sites".
If the status is "Pending", Klarvu hasn't received any data yet. If it's "Receiving logs", data is arriving, but no requests from AI agents have been detected.
Also check that you're looking at the correct reporting period.
Why don't I see visits from a particular AI platform?
It means that platform's agents haven't visited your website during the selected period. Try a longer period, such as "90 days".
Does Klarvu store information about my human visitors?
No. "Agent traffic" only stores requests from AI agents. Requests from human visitors aren't stored.
My website isn't hosted on Vercel. Can I still connect it?
Yes. You can use "API / middleware" even if your hosting provider doesn't have a dedicated integration.
You can send requests through Next.js middleware or make POST requests from another platform.
"Cloudflare", "Cloudflare Enterprise", "CloudFront", "Akamai", and "WordPress" aren't available as standalone connection methods yet.
Does more AI agent traffic mean AI mentions my brand more often?
Not necessarily. "Agent traffic" shows requests made to your website, but doesn't tell you whether the content was subsequently used in an AI answer.
You can investigate page accessibility with "Crawlability Checker", review actual citations under "Sources", and learn about visitors arriving from AI answers in the AI referrals (GA4) guide.