SEO and Marketing · 5 min read · Updated

Search Everywhere: SEO Cheat Sheets: Regex, robots.txt, AI Bot User Agents and Log One-Liners

Copy-ready cheat sheets: regex for Search Console and GA4, robots.txt patterns, AI crawler user-agent tokens and log analysis one-liners.

SEO Cheat Sheets: Regex, robots.txt and AI Bots: featured image for the webocreation.com guide

This post is Appendix B of Search Everywhere: The Practical Guide to SEO, AEO and GEO, the free online edition. Chapter and appendix references link to the matching post.

B1. Regex for Search Console and GA4

Search Console uses RE2 regular expression syntax in its Custom (regex) filter. Matching is case-sensitive unless you add (?i) at the start.

Purpose Regex
Question queries (?i)^(who|what|when|where|why|how|which|can|does|do|is|are|should|will)\b
Queries containing a question word anywhere (?i)\b(how|what|why|which)\b
Long queries (8+ words) ^(\S+\s+){7,}\S+$
Short head queries (1–2 words) ^\S+(\s+\S+)?$
Branded (case study) (?i)webo\s?creation|rupak|nepali
Comparison queries (?i)\b(vs|versus|or|compare|comparison|alternative|alternatives)\b
“Best” and recommendation queries (?i)\b(best|top|recommended|review|reviews)\b
Year-specific queries \b20(2[4-9]|3[0-9])\b
Error and troubleshooting queries (?i)\b(error|not working|fix|blank|failed|issue|problem|warning)\b
OpenCart version queries (?i)opencart\s?(2|3|4)(\.\d+)*
Pages in a folder ^https://webocreation\.com/(tag|author)/
Exclude parameter URLs (use “Doesn’t match regex”) \?

GA4 custom channel group condition for AI assistants (Source, matches regex):

.*(chatgpt|openai|perplexity|gemini\.google|bard\.google|copilot\.microsoft|bing\.com/chat|claude\.ai|anthropic|you\.com|phind|deepseek|meta\.ai|mistral|grok|x\.ai).*

B2. robots.txt patterns

# Block a folder
Disallow: /private/

# Block a single file type
Disallow: /*.pdf$

# Block URLs with any parameter
Disallow: /*?

# Block specific parameters only
Disallow: /*?*sort=
Disallow: /*?*replytocom=

# Allow a file inside a blocked folder
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Several bots sharing one group
User-agent: GPTBot
User-agent: CCBot
Disallow: /

# Declare sitemap
Sitemap: https://example.com/sitemap_index.xml

Reminders: paths are case-sensitive; the longest matching rule wins; a bot obeys only its most specific group; Crawl-delay is ignored by Google.

B3. AI crawler user-agent tokens (late 2026)

Token Company Purpose
GPTBot OpenAI Training
OAI-SearchBot OpenAI ChatGPT search index
ChatGPT-User OpenAI User-initiated fetch
ClaudeBot Anthropic Training
Claude-SearchBot Anthropic Search index
Claude-User Anthropic User-initiated fetch
PerplexityBot Perplexity Search index
Perplexity-User Perplexity User-initiated fetch
Google-Extended Google Token: Gemini training and grounding (not Search)
Applebot-Extended Apple Token: AI training
CCBot Common Crawl Open dataset widely used for training
Bytespider ByteDance AI crawling
meta-externalagent Meta AI crawling
Amazonbot Amazon AI and Alexa
DuckAssistBot DuckDuckGo AI answers
MistralAI-User Mistral User-initiated fetch

Always check the vendor’s current documentation; tokens change.

B4. Log analysis one-liners

# Top user agents
awk -F'"' '{print $6}' access.log | sort | uniq -c | sort -rn | head -20

# Requests per bot
for b in Googlebot bingbot GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User PerplexityBot CCBot Bytespider; do
  printf "%-16s %s\n" $b $(grep -c "$b" access.log); done

# Googlebot status codes
grep Googlebot access.log | awk '{print $9}' | sort | uniq -c | sort -rn

# Googlebot top URLs
grep Googlebot access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

# 404s served to bots
grep -Ei 'bot|crawl|spider' access.log | awk '$9==404{print $7}' | sort | uniq -c | sort -rn | head

# Googlebot hits per day
grep Googlebot access.log | awk -F'[' '{print substr($2,1,11)}' | uniq -c

# Pages fetched by AI on behalf of users
grep -E 'ChatGPT-User|Claude-User|Perplexity-User' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -30

# Verify a Googlebot IP
host 66.249.66.1 && host crawl-66-249-66-1.googlebot.com

# Compressed logs
zgrep Googlebot access.log.*.gz | wc -l

B5. curl checks

# Status and headers only
curl -sI https://example.com/page/

# Follow redirects and show each hop
curl -sIL https://example.com/old-page/ | grep -Ei '^(HTTP|location)'

# Fetch as Googlebot smartphone
curl -s -A "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/ | head -50

# Fetch as GPTBot and look for text
curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://example.com/faq/ | grep -i "answer"

# Count JSON-LD blocks
curl -s https://example.com/ | grep -c 'application/ld+json'

# Check X-Robots-Tag on a PDF
curl -sI https://example.com/file.pdf | grep -i x-robots-tag

# Time to first byte
curl -o /dev/null -s -w "TTFB: %{time_starttransfer}s\n" https://example.com/

B6. Google search operators

Operator Example Use
site: site:webocreation.com opencart 4 Pages indexed on a site about a topic
-site: "webocreation" -site:webocreation.com Mentions elsewhere
"..." "turnstile.ocmod.zip" Exact phrase
intitle: intitle:"opencart 4" turnstile Phrase in title
inurl: inurl:resources opencart Phrase in URL
filetype: filetype:pdf opencart guide File types
OR opencart (captcha OR turnstile) Alternatives
before: / after: opencart upgrade after:2025-01-01 Date-limited results

Note: site: counts are rough estimates, not index totals. Use Search Console for real index data.

B7. Core Web Vitals thresholds

Metric Good Needs improvement Poor
LCP ≤ 2.5 s ≤ 4.0 s > 4.0 s
INP ≤ 200 ms ≤ 500 ms > 500 ms
CLS ≤ 0.1 ≤ 0.25 > 0.25

Measured at the 75th percentile of page loads, mobile and desktop separately.

B8. HTTP status codes for SEO

Code Use
200 Normal page
301 / 308 Permanent move
302 / 307 Temporary move
304 Not modified (conditional request)
404 Not found
410 Permanently gone
429 Rate limited
503 Temporary maintenance (add Retry-After)

B9. Meta robots directives

Directive Effect
noindex Exclude from results
nofollow Do not follow links on page
nosnippet No text snippet; no use in AI Overviews/AI Mode
max-snippet:N Snippet length limit (-1 = no limit)
max-image-preview:large Allow large image previews
max-video-preview:-1 No limit on video preview
noimageindex Do not index images on page
unavailable_after: date Drop from results after date
data-nosnippet (attribute) Exclude element from snippets

B10. Answer formats by question type

Question type Opening sentence formula Then
What is X? “X is a [category] that [does/has distinguishing feature] to [purpose].” Detail, example
How do I X? “To X, [summary of the steps in one sentence].” Numbered steps
Is/Does/Can X? “Yes/No. [Condition or reason].” Detail, exceptions
X vs Y? “For [situation], X is better because [reason]; choose Y if [condition].” Comparison table
Best X? “The best X for [situation] are A, B and C.” List with one line each
Why X? “X happens because [main cause].” Other causes, fixes
How much/long? “X typically costs/takes [range], depending on [factors].” Breakdown

Previous: SEO, AEO and GEO Checklists: Technical, Content, Migration and More
Next: Schema Markup Templates: Copy-Ready JSON-LD Examples
All chapters: Search Everywhere: The Practical Guide to SEO, AEO and GEO

Search Everywhere (full PDF)The complete guide in one printable PDF.PDF · 2.8 MB
Free download: unlock it in one step

Share this page

or

Mailchimp opens in a new tab to confirm. Unsubscribe any time.

Rupak Nepali
Written byRupak Nepali

Rupak Nepali is a web developer, manage Webocreation website and the author of six books:
Getting Started with OpenCart Module Development (Packt, 2013),
OpenCart Theme and Module Development (Packt, 2015),
OpenCart 4: Dev Guide for Themes & Extensions (2024),
OpenCart 4: User Manual (2025),
Cloudflare for the Enterprise Website (2026) and
The Technical Web Guide for Marketers (2026).
Rupak has built OpenCart stores and extensions since 2010, has published more than 600 tutorials on Webocreation, and shares free OpenCart extensions on GitHub.
Find Rupak on
GitHub | Amazon | LinkedIn | X | YouTube.

Leave a Reply

Your email address will not be published. Required fields are marked *