Search Everywhere: SEO Cheat Sheets: Regex, robots.txt, AI Bot User Agents and Log One-Liners
Copy-ready cheat sheets: regex for Search Console and GA4, robots.txt patterns, AI crawler user-agent tokens and log analysis one-liners.

This post is Appendix B of Search Everywhere: The Practical Guide to SEO, AEO and GEO, the free online edition. Chapter and appendix references link to the matching post.
B1. Regex for Search Console and GA4
Search Console uses RE2 regular expression syntax in its Custom (regex) filter. Matching is case-sensitive unless you add (?i) at the start.
| Purpose | Regex |
|---|---|
| Question queries | (?i)^(who|what|when|where|why|how|which|can|does|do|is|are|should|will)\b |
| Queries containing a question word anywhere | (?i)\b(how|what|why|which)\b |
| Long queries (8+ words) | ^(\S+\s+){7,}\S+$ |
| Short head queries (1–2 words) | ^\S+(\s+\S+)?$ |
| Branded (case study) | (?i)webo\s?creation|rupak|nepali |
| Comparison queries | (?i)\b(vs|versus|or|compare|comparison|alternative|alternatives)\b |
| “Best” and recommendation queries | (?i)\b(best|top|recommended|review|reviews)\b |
| Year-specific queries | \b20(2[4-9]|3[0-9])\b |
| Error and troubleshooting queries | (?i)\b(error|not working|fix|blank|failed|issue|problem|warning)\b |
| OpenCart version queries | (?i)opencart\s?(2|3|4)(\.\d+)* |
| Pages in a folder | ^https://webocreation\.com/(tag|author)/ |
| Exclude parameter URLs (use “Doesn’t match regex”) | \? |
GA4 custom channel group condition for AI assistants (Source, matches regex):
.*(chatgpt|openai|perplexity|gemini\.google|bard\.google|copilot\.microsoft|bing\.com/chat|claude\.ai|anthropic|you\.com|phind|deepseek|meta\.ai|mistral|grok|x\.ai).*
B2. robots.txt patterns
# Block a folder
Disallow: /private/
# Block a single file type
Disallow: /*.pdf$
# Block URLs with any parameter
Disallow: /*?
# Block specific parameters only
Disallow: /*?*sort=
Disallow: /*?*replytocom=
# Allow a file inside a blocked folder
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Several bots sharing one group
User-agent: GPTBot
User-agent: CCBot
Disallow: /
# Declare sitemap
Sitemap: https://example.com/sitemap_index.xml
Reminders: paths are case-sensitive; the longest matching rule wins; a bot obeys only its most specific group; Crawl-delay is ignored by Google.
B3. AI crawler user-agent tokens (late 2026)
| Token | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training |
| OAI-SearchBot | OpenAI | ChatGPT search index |
| ChatGPT-User | OpenAI | User-initiated fetch |
| ClaudeBot | Anthropic | Training |
| Claude-SearchBot | Anthropic | Search index |
| Claude-User | Anthropic | User-initiated fetch |
| PerplexityBot | Perplexity | Search index |
| Perplexity-User | Perplexity | User-initiated fetch |
| Google-Extended | Token: Gemini training and grounding (not Search) | |
| Applebot-Extended | Apple | Token: AI training |
| CCBot | Common Crawl | Open dataset widely used for training |
| Bytespider | ByteDance | AI crawling |
| meta-externalagent | Meta | AI crawling |
| Amazonbot | Amazon | AI and Alexa |
| DuckAssistBot | DuckDuckGo | AI answers |
| MistralAI-User | Mistral | User-initiated fetch |
Always check the vendor’s current documentation; tokens change.
B4. Log analysis one-liners
# Top user agents
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20
# Requests per bot
for b in Googlebot bingbot GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User PerplexityBot CCBot Bytespider; do
printf "%-16s %s\n" $b $(grep -c "$b" access.log); done
# Googlebot status codes
grep Googlebot access.log | awk '{print $9}' | sort | uniq -c | sort -rn
# Googlebot top URLs
grep Googlebot access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30
# 404s served to bots
grep -Ei 'bot|crawl|spider' access.log | awk '$9==404{print $7}' | sort | uniq -c | sort -rn | head
# Googlebot hits per day
grep Googlebot access.log | awk -F'[' '{print substr($2,1,11)}' | uniq -c
# Pages fetched by AI on behalf of users
grep -E 'ChatGPT-User|Claude-User|Perplexity-User' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30
# Verify a Googlebot IP
host 66.249.66.1 && host crawl-66-249-66-1.googlebot.com
# Compressed logs
zgrep Googlebot access.log.*.gz | wc -l
B5. curl checks
# Status and headers only
curl -sI https://example.com/page/
# Follow redirects and show each hop
curl -sIL https://example.com/old-page/ | grep -Ei '^(HTTP|location)'
# Fetch as Googlebot smartphone
curl -s -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/ | head -50
# Fetch as GPTBot and look for text
curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://example.com/faq/ | grep -i "answer"
# Count JSON-LD blocks
curl -s https://example.com/ | grep -c 'application/ld+json'
# Check X-Robots-Tag on a PDF
curl -sI https://example.com/file.pdf | grep -i x-robots-tag
# Time to first byte
curl -o /dev/null -s -w "TTFB: %{time_starttransfer}s\n" https://example.com/
B6. Google search operators
| Operator | Example | Use |
|---|---|---|
site: |
site:webocreation.com opencart 4 |
Pages indexed on a site about a topic |
-site: |
"webocreation" -site:webocreation.com |
Mentions elsewhere |
"..." |
"turnstile.ocmod.zip" |
Exact phrase |
intitle: |
intitle:"opencart 4" turnstile |
Phrase in title |
inurl: |
inurl:resources opencart |
Phrase in URL |
filetype: |
filetype:pdf opencart guide |
File types |
OR |
opencart (captcha OR turnstile) |
Alternatives |
before: / after: |
opencart upgrade after:2025-01-01 |
Date-limited results |
Note: site: counts are rough estimates, not index totals. Use Search Console for real index data.
B7. Core Web Vitals thresholds
| Metric | Good | Needs improvement | Poor |
|---|---|---|---|
| LCP | ≤ 2.5 s | ≤ 4.0 s | > 4.0 s |
| INP | ≤ 200 ms | ≤ 500 ms | > 500 ms |
| CLS | ≤ 0.1 | ≤ 0.25 | > 0.25 |
Measured at the 75th percentile of page loads, mobile and desktop separately.
B8. HTTP status codes for SEO
| Code | Use |
|---|---|
| 200 | Normal page |
| 301 / 308 | Permanent move |
| 302 / 307 | Temporary move |
| 304 | Not modified (conditional request) |
| 404 | Not found |
| 410 | Permanently gone |
| 429 | Rate limited |
| 503 | Temporary maintenance (add Retry-After) |
B9. Meta robots directives
| Directive | Effect |
|---|---|
noindex |
Exclude from results |
nofollow |
Do not follow links on page |
nosnippet |
No text snippet; no use in AI Overviews/AI Mode |
max-snippet:N |
Snippet length limit (-1 = no limit) |
max-image-preview:large |
Allow large image previews |
max-video-preview:-1 |
No limit on video preview |
noimageindex |
Do not index images on page |
unavailable_after: date |
Drop from results after date |
data-nosnippet (attribute) |
Exclude element from snippets |
B10. Answer formats by question type
| Question type | Opening sentence formula | Then |
|---|---|---|
| What is X? | “X is a [category] that [does/has distinguishing feature] to [purpose].” | Detail, example |
| How do I X? | “To X, [summary of the steps in one sentence].” | Numbered steps |
| Is/Does/Can X? | “Yes/No. [Condition or reason].” | Detail, exceptions |
| X vs Y? | “For [situation], X is better because [reason]; choose Y if [condition].” | Comparison table |
| Best X? | “The best X for [situation] are A, B and C.” | List with one line each |
| Why X? | “X happens because [main cause].” | Other causes, fixes |
| How much/long? | “X typically costs/takes [range], depending on [factors].” | Breakdown |
Previous: SEO, AEO and GEO Checklists: Technical, Content, Migration and More
Next: Schema Markup Templates: Copy-Ready JSON-LD Examples
All chapters: Search Everywhere: The Practical Guide to SEO, AEO and GEO






