Which AI agents request your content? A guide to reading page-level crawler logs

How to inspect AI agent requests by page and content type

A page-level log audit shows which URLs suspected AI agents requested, what the server returned, and where a closer investigation is useful. Start with server or CDN request records, group them by agent and URL, then compare response patterns with content type and conversion data. Treat a matching User-Agent as a lead, not proof of who made the request.

How to see which pages AI agents request

Export server or CDN logs, filter requests against a maintained list of AI crawler User-Agent strings, and aggregate hits, status codes, response sizes and timestamps by URL. That produces a practical list of pages to investigate while keeping the distinction between an agent *claiming* an identity and an agent whose identity has been verified.

Which records should the audit use?

Use request-level logs from the system that receives the traffic. Depending on the site, that may be a web server such as Nginx, a CDN export, or logs stored for analysis in S3 access logs or BigQuery. A CDN dashboard can help spot a pattern, but request records are needed to inspect the URLs and responses behind it.

For each request, extract the timestamp, client IP, User-Agent, requested URL, response status and response size. Retain the response Content-Type if it is available. Field names and logging formats vary, so establish a consistent set of names before combining exports. Keep the original records: an aggregated dashboard cannot settle a later question about a particular request.

The first filter should be deliberately narrow. Maintain a list of User-Agent strings taken from the relevant operators' current documentation, then search for those strings in the log's User-Agent field. AI traffic can include crawlers fetching content for different purposes and fetches associated with a live user interaction. Do not group those activities into a single measure of demand.

How can a team run a quick page-level check?

The following examples assume a normalized JSON Lines file called `requests.jsonl`. Each line is one request with keys named `timestamp`, `client_ip`, `user_agent`, `url`, `status`, `bytes` and, where recorded, `content_type`. These are example keys for the normalized file, not promised field names in any server or CDN export.

For an ad-hoc inspection, set a User-Agent pattern based on the agents being investigated and print matching request details:

“`sh

UA_PATTERN='replace-with-agent-pattern'

jq -r –arg pattern "$UA_PATTERN" '

select((.user_agent // "" | ascii_downcase) |

contains($pattern | ascii_downcase)) |

[.timestamp, .client_ip, .user_agent, .url, .status, .bytes] |

@tsv

' requests.jsonl

“`

A `grep` check is useful when inspecting a familiar raw-log format, but it searches text rather than a reliably parsed User-Agent field:

“`sh

grep -i — "$UA_PATTERN" access.log

“`

For a quick URL count from the normalized file, `jq` can produce a URL column for `awk` to tally:

“`sh

jq -r –arg pattern "$UA_PATTERN" '

select((.user_agent // "" | ascii_downcase) |

contains($pattern | ascii_downcase)) |

.url

' requests.jsonl |

awk '{hits[$0]++} END {for (url in hits) print hits[url], url}' |

sort -nr

“`

These commands identify candidates. The pattern must be replaced with a User-Agent value relevant to the audit, and raw-log parsing must follow the actual export format. For repeated reporting, use a small `jq` or Python pipeline that groups by both agent and URL rather than relying on a text search.

Which URLs deserve investigation first?

Build a view with one row per claimed agent and URL. Show request count, status-code distribution, typical and unusual response sizes, first and latest request times, and a short time series of hits. This makes a top-requested-page list more useful than a count alone.

Use the following as triage rules, not as evidence that a page influenced an AI answer:

• Investigate frequently requested URLs to understand what an agent could access and whether the returned page contains the intended information.
• Review pages with repeated successful responses alongside weak human conversion signals. Compare traffic and conversion data carefully; a crawler request is not a human visit or a conversion opportunity.
• Inspect unusually small or large responses against the page's normal response size. The request record may point to an error page, redirect, incomplete response or another delivery issue.
• Review recurring error statuses and the timestamps around them. A time-series chart can reveal a burst that a total hit count conceals.

When identity matters, escalate beyond the User-Agent. Compare the client IP with a range published by the claimed operator, and use reverse-DNS as an additional investigative signal where appropriate. An IP or hostname result should be assessed against that operator's current guidance; neither a User-Agent string nor a reverse-DNS result alone proves attribution.

How to analyze AI crawler requests by content type

Classify each candidate request using its URL path, recorded response Content-Type and file extension, then summarize hits, statuses, response sizes and timing by agent and content type. This separates requests for HTML pages from requests for PDFs, API/JSON responses and images without assuming that a particular agent prefers any one format.

How should request records be classified?

Start with the response Content-Type when it is recorded and represents the response actually served. Use the URL path and extension as supporting clues. For example, a path ending in `.pdf` suggests a PDF, while a response labeled `application/pdf` provides stronger evidence about what was returned. A `/api/` path may suggest an endpoint, but its response Content-Type determines whether it returned JSON, HTML or something else.

The script below reads the normalized JSON Lines file used above. It assigns a practical content label and summarizes each User-Agent, content-type and URL combination. The matching rule is intentionally simple: supply a pattern that fits the current investigation, then review the matched records before treating the label as an agent identity.

“`python

import json

import sys

from collections import Counter, defaultdict

from statistics import median

from urllib.parse import urlparse

pattern = sys.argv[1].lower()

groups = defaultdict(lambda: {

"statuses": Counter(), "sizes": [], "times": []

})

def content_kind(record):

content_type = (record.get("content_type") or "").lower()

path = urlparse(record.get("url") or "").path.lower()

if "text/html" in content_type:

return "HTML"

if "application/pdf" in content_type:

return "PDF"

if "application/json" in content_type or "+json" in content_type:

return "API/JSON"

if content_type.startswith("image/"):

return "image"

if path.endswith((".html", ".htm")):

return "HTML (extension)"

if path.endswith(".pdf"):

return "PDF (extension)"

if path.endswith(".json"):

return "API/JSON (extension)"

if path.endswith((".png", ".jpg", ".jpeg", ".webp", ".svg")):

return "image (extension)"

return "unclassified"

with open("requests.jsonl", encoding="utf-8") as source:

for line in source:

record = json.loads(line)

agent = record.get("user_agent") or ""

if pattern not in agent.lower():

continue

key = (agent, content_kind(record), record.get("url") or "")

group = groups[key]

group["statuses"][str(record.get("status"))] += 1

if record.get("bytes") is not None:

group["sizes"].append(int(record["bytes"]))

if record.get("timestamp"):

group["times"].append(record["timestamp"])

for (agent, kind, url), group in sorted(

groups.items(), key=lambda item: -sum(item[1]["statuses"].values())

):

sizes = group["sizes"]

times = sorted(group["times"])

print(json.dumps({

"agent": agent,

"content_type": kind,

"url": url,

"hits": sum(group["statuses"].values()),

"statuses": dict(group["statuses"]),

"median_bytes": median(sizes) if sizes else None,

"first_seen": times[0] if times else None,

"last_seen": times[-1] if times else None

}))

“`

Run it with `python audit.py 'agent-pattern'`. A large export may call for a scheduled pipeline or query tool instead, but the classification logic should remain inspectable. In particular, extension-based labels need review: a URL may have no extension, and an error response may have a different Content-Type from the requested document.

How does content type change the next action?

For HTML pages, inspect the returned status and rendered content before changing copy or page structure. A successful request shows that the server returned a response; it does not show that an AI system used, cited or recommended the page. For PDFs, inspect whether the document and its response are suitable for the intended audience. For API/JSON paths, establish whether the request is expected and authorized before treating its volume as a content opportunity. Images and unclassified responses warrant separate review rather than being folded into a page count.

Travatar is a SaaS platform for AI visibility analytics and web-traffic intelligence. We monitor brand visibility in AI Search, classify bot traffic and help teams assess its impact on conversion. When a manual audit produces more candidates than a team can review, move toward a workflow that connects traffic classification, page-level investigation and conversion analysis. Evaluate any platform against the log sources and reporting detail the team actually needs; do not assume that a dashboard identifies every agent or proves that a fetch caused an AI mention.

The immediate next step is modest: export a request sample, normalize its fields, produce the agent-by-URL list, and inspect the highest-priority records before changing content or crawler access rules.

FAQ

How do I get server or CDN logs if I only use a hosted CMS?

Ask the hosting provider or CDN which request-level exports are available and whether they include URL, timestamp, User-Agent, status, response size and client IP. If request logs are unavailable, use the available traffic dashboard for an initial view, but do not treat an aggregate chart as a page-level audit.

What are the most reliable signals to prove an AI agent fetched a page?

A request record links a claimed User-Agent and client IP to a URL, timestamp and server response. To strengthen attribution, compare the IP with the claimed operator's published guidance and investigate DNS information; a User-Agent string by itself is not proof of identity.

Can I automate content-type mapping for large logs, or is manual parsing required?

A parser can assign initial labels from response Content-Type, URL paths and extensions. Review exceptions and a sample of classified records, especially where Content-Type is absent, the URL has no extension, or the server returned an error.

What quick checks detect User-Agent spoofing?

Compare the request's client IP with ranges published by the claimed operator and investigate its reverse-DNS result against that operator's guidance. Treat a mismatch as a reason to question the attribution, not as a complete classification of the visitor.

When should I move from scripts to a vendor platform for this work?

Consider a platform when recurring log exports, agent classification and comparison with conversion data consume more effort than the team can sustain manually. Define the required data sources, page-level views and validation process before evaluating a vendor.