Search Everywhere: The SEO Tool Stack: Crawlers, AI Trackers and Python Scripts
Screaming Frog, Ahrefs, Bing Webmaster Tools, AI visibility trackers, browser extensions and Python scripts, with a recommended SEO stack for any budget.

This post is Chapter 19 of Search Everywhere: The Practical Guide to SEO, AEO and GEO, the free online edition. Chapter and appendix references link to the matching post.
No single tool does everything. This chapter surveys the other tools that a working SEO, AEO and GEO practitioner uses, what each is best at, and how to build simple tools of your own with Python when nothing off the shelf fits.
Crawlers
Screaming Frog SEO Spider
A desktop crawler (Windows, macOS, Linux), free for up to 500 URLs and paid beyond that. It is the workhorse of technical SEO.
Key capabilities:
- Crawls a site like a search engine, reporting status codes, titles, meta descriptions, headings, canonicals, meta robots, hreflang, structured data, images, links and response times.
- JavaScript rendering using an embedded Chromium, so you can compare raw HTML with rendered output.
- Custom extraction with XPath, CSS selectors or regex, for example to pull author names, publish dates or word counts from every post.
- Custom search to find pages containing (or missing) specific text.
- API integrations with Search Console (clicks, impressions, and URL Inspection index status), GA4, PageSpeed Insights, Ahrefs and others, so crawl data and performance data sit side by side.
- Crawl comparison to see what changed between two crawls.
- Visualisations of site architecture and crawl depth.
- List mode to check a specific list of URLs, such as all URLs from a sitemap or a log file.
Case study recipe: auditing content quality across the whole blog.
- Crawl
https://webocreation.com/with JavaScript rendering off (to see what non-rendering AI crawlers see).- Add custom extractions: author name (CSS selector for the byline), published date, updated date, primary category (from breadcrumbs).
- Connect Search Console (last 12 months clicks and impressions) and the URL Inspection API (index status).
- Export the “Internal HTML” tab.
- In a spreadsheet, filter for pages with zero clicks in 12 months, not indexed, or authored by guest contributors. This becomes the pruning list for Chapter 5’s decision framework.
Sitebulb
A desktop and cloud crawler that emphasises prioritised, explained recommendations (“Hints”) and visual reports. Good for presenting audits to non-technical stakeholders.
Semrush Site Audit, Ahrefs Site Audit, and other cloud crawlers
Cloud crawlers run on a schedule without tying up your computer and alert you to new issues. They are less flexible than Screaming Frog but better for monitoring.
Backlink and competitive tools
Ahrefs
Ahrefs’ main strength is its link index, one of the largest commercial backlink databases. Its Site Explorer, Keywords Explorer, Content Explorer and Rank Tracker overlap with Semrush. Many practitioners use both, or pick one based on preference. Ahrefs also offers Ahrefs Webmaster Tools free for verified sites, which provides a site audit and backlink data for your own properties, and it has added AI visibility tracking (Brand Radar) for monitoring mentions in AI answers.
Moz Pro, Majestic, SE Ranking, Similarweb
- Moz Pro: Domain Authority (a widely quoted metric), keyword research, rank tracking.
- Majestic: link-focused, with Trust Flow and Citation Flow metrics.
- SE Ranking: a lower-cost all-in-one platform with AI visibility tracking.
- Similarweb: traffic and audience intelligence, strongest for larger sites.
Bing Webmaster Tools
Covered in Chapter 15. Free, and important because Bing’s index powers Microsoft Copilot and is used by other AI assistants. Its Site Scan, URL inspection, backlink data, keyword research and IndexNow support are all worth using.
AI visibility trackers
A new category of tools tracks how brands appear in AI answers. They work by running sets of prompts against AI assistants on a schedule and recording mentions, citations, positions and sentiment. Examples in 2026 include Semrush’s and Ahrefs’ AI features, SE Ranking’s AI tracking, and specialist tools such as Profound, Peec AI, Otterly.AI, Scrunch and others. The market is moving quickly; evaluate tools on:
- Which assistants they cover (ChatGPT, Google AI Overviews and AI Mode, Gemini, Perplexity, Copilot, Claude).
- Whether they use real user interfaces or APIs (answers can differ).
- How many prompts you can track and how often.
- Whether they report cited URLs, not just brand mentions.
- Location and language options.
Chapter 24 shows how to build a basic version yourself with a spreadsheet.
Browser extensions
Quick checks without opening a full tool:
- Detailed SEO Extension: titles, descriptions, headings, canonicals, robots, schema and links for the current page.
- SEO META in 1 CLICK: similar overview.
- Web Vitals: live Core Web Vitals.
- Wappalyzer or BuiltWith: detect the technologies a site uses.
- Redirect Path: shows redirect chains and status codes.
- View Rendered Source: compare raw and rendered HTML.
- User-Agent Switcher: browse as Googlebot or an AI crawler.
Structured data and markup helpers
- Schema.org documentation for type and property references.
- Merkle’s Schema Markup Generator (technicalseo.com tools) for quick JSON-LD.
- Rich Results Test and Schema Markup Validator (Chapter 17).
Log analysis tools
Covered in depth in Chapter 20:
- Screaming Frog Log File Analyser (desktop, inexpensive).
- Semrush Log File Analyzer (in Site Audit).
- Botify, Lumar, JetOctopus, Oncrawl for enterprise log analysis.
- Cloudflare analytics, logs and AI Crawl Control for bot visibility at the edge.
- GoAccess, the command line and Python for do-it-yourself analysis.
Content and writing tools
- Semrush SEO Writing Assistant, Surfer, Clearscope, MarketMuse, Frase: content optimisation tools that compare your draft with top-ranking pages. Useful as a coverage check; dangerous if used to stuff terms.
- Hemingway Editor or Grammarly: readability and grammar.
- AlsoAsked and AnswerThePublic: question research.
Performance tools
PageSpeed Insights, Lighthouse, WebPageTest, GTmetrix, DebugBear (continuous monitoring with CrUX history), and the CrUX tools (Chapter 6).
Building your own tools with Python
Sometimes the quickest path to an answer is a short script. Python, with a few libraries, covers most SEO automation needs. The examples below are short, practical and safe to adapt.
Check status codes, titles and canonicals for a URL list
# pip install requests beautifulsoup4
import csv, requests
from bs4 import BeautifulSoup
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; SiteAuditBot/1.0)"}
def check(url):
r = requests.get(url, headers=HEADERS, timeout=20, allow_redirects=False)
row = {"url": url, "status": r.status_code,
"location": r.headers.get("Location", ""),
"x_robots": r.headers.get("X-Robots-Tag", "")}
if r.status_code == 200 and "text/html" in r.headers.get("Content-Type", ""):
s = BeautifulSoup(r.text, "html.parser")
title = s.title.get_text(strip=True) if s.title else ""
canon = s.find("link", rel="canonical")
robots = s.find("meta", attrs={"name": "robots"})
h1 = s.find("h1")
row.update({
"title": title, "title_len": len(title),
"canonical": canon["href"] if canon else "",
"meta_robots": robots["content"] if robots else "",
"h1": h1.get_text(strip=True) if h1 else "",
"jsonld_blocks": len(s.find_all("script", type="application/ld+json")),
})
return row
urls = [line.strip() for line in open("urls.txt") if line.strip()]
rows = [check(u) for u in urls]
with open("audit.csv", "w", newline="") as f:
w = csv.DictWriter(f, fieldnames=sorted({k for r in rows for k in r}))
w.writeheader(); w.writerows(rows)
Note the jsonld_blocks column: this is how you would verify, from the raw HTML, whether a page outputs JSON-LD, which the text-only fetcher used for this book could not.
Extract every URL from a sitemap index
import requests, xml.etree.ElementTree as ET
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
def sitemap_urls(url):
root = ET.fromstring(requests.get(url, timeout=30).content)
if root.tag.endswith("sitemapindex"):
for loc in root.findall("sm:sitemap/sm:loc", NS):
yield from sitemap_urls(loc.text.strip())
else:
for u in root.findall("sm:url", NS):
yield u.find("sm:loc", NS).text.strip(), (u.findtext("sm:lastmod", "", NS))
for loc, lastmod in sitemap_urls("https://webocreation.com/sitemap_index.xml%22:
print(lastmod, loc)
Pull Search Console data through the API
# pip install google-api-python-client google-auth
from google.oauth2 import service_account
from googleapiclient.discovery import build
creds = service_account.Credentials.from_service_account_file(
"service-account.json",
scopes=["https://www.googleapis.com/auth/webmasters.readonly"])
gsc = build("searchconsole", "v1", credentials=creds)
body = {
"startDate": "2026-08-01", "endDate": "2026-10-31",
"dimensions": ["query", "page"],
"dimensionFilterGroups": [{"filters": [{
"dimension": "query", "operator": "includingRegex",
"expression": "^(how|what|why|which|can|does|is)\\b"}]}],
"rowLimit": 25000,
}
resp = gsc.searchanalytics().query(siteUrl="sc-domain:webocreation.com", body=body).execute()
for row in resp.get("rows", [])[:20]:
print(row["keys"], row["clicks"], row["impressions"], round(row["position"], 1))
Add the service account’s email address as a user on the Search Console property before running this.
Batch PageSpeed Insights
import requests, time
KEY = "YOUR_API_KEY"
for url in ["https://webocreation.com/", "https://webocreation.com/blog/",
"https://webocreation.com/opencart-4-cloudflare-turnstile-captcha/%22]:
r = requests.get("https://www.googleapis.com/pagespeedonline/v5/runPagespeed",
params={"url": url, "strategy": "mobile", "key": KEY}, timeout=120).json()
field = r.get("loadingExperience", {}).get("metrics", {})
lab = r["lighthouseResult"]["categories"]["performance"]["score"] * 100
print(url, "lab:", lab,
"LCP p75:", field.get("LARGEST_CONTENTFUL_PAINT_MS", {}).get("percentile"),
"INP p75:", field.get("INTERACTION_TO_NEXT_PAINT", {}).get("percentile"),
"CLS p75:", field.get("CUMULATIVE_LAYOUT_SHIFT_SCORE", {}).get("percentile"))
time.sleep(2)
Find internal link opportunities
Given a crawl export with page URLs and body text, find pages that mention a target phrase but do not link to the target URL:
import pandas as pd
df = pd.read_csv("crawl_export.csv") # columns: url, body_text, outlinks (space-separated)
target_url = "https://webocreation.com/opencart-4-cloudflare-turnstile-captcha/"
phrase = "turnstile"
mask = (df["body_text"].str.contains(phrase, case=False, na=False)
& ~df["outlinks"].fillna("").str.contains(target_url, regex=False)
& (df["url"] != target_url))
print(df.loc[mask, "url"].to_list())
Choosing a stack by budget
| Budget | Stack |
|---|---|
| Free | Search Console, Bing Webmaster Tools, GA4, Looker Studio, PageSpeed Insights, Rich Results Test, Screaming Frog (500 URLs), Ahrefs Webmaster Tools, Google Trends, a spreadsheet for AI prompt tracking |
| Small business | Add Semrush (Pro or Guru) or Ahrefs (Lite or Standard), Screaming Frog licence |
| Agency or growing brand | Add an AI visibility tracker, Sitebulb or a cloud crawler, Screaming Frog Log File Analyser, BigQuery exports |
| Enterprise | Add enterprise log analysis (Botify, Lumar, Oncrawl), Similarweb, dedicated AI visibility platform, data warehouse and BI tooling |
Key takeaways
- Screaming Frog is the essential crawler; connect it to Search Console, the URL Inspection API and PageSpeed Insights for combined analysis.
- Ahrefs and Semrush overlap; pick one as your main competitive tool and use the free webmaster versions of others.
- AI visibility trackers are a fast-moving category; evaluate them on coverage, method, prompt volume and citation detail.
- Short Python scripts fill the gaps: checking raw HTML for JSON-LD, extracting sitemaps, pulling Search Console data, batch PageSpeed tests and internal link discovery.
Previous: A Semrush SEO Analysis Workflow in 11 Steps
Next: Server Log Analysis for Search and AI Crawlers
All chapters: Search Everywhere: The Practical Guide to SEO, AEO and GEO






