SiteAtlas SEO Audit: Broken Links, Hreflang & llms.txt

SiteAtlas crawls a public website and audits on-page SEO: title, meta description, headings, canonical, robots meta, image alt text, hreflang, broken links, duplicate titles and descriptions, and response time. It also checks AI readability once per start domain (robots rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot, plus /llms.txt, /llms-full.txt, and /sitemap.xml). Each page gets a 0–100 score. The run writes a site score and an HTML report and a Markdown report.

It does not run a browser and it does not collect personal data. Fix text is a fixed sentence from the deduction table. The crawler does not call a language model.

The same core/ package runs this portfolio demo and the Apify Actor siteatlas-seo-crawler. The Actor is ready to publish with limited permissions. It is not on the Store yet, and this page does not link to a Store listing.

Store title: SiteAtlas SEO Audit: Broken Links, Hreflang & llms.txt. The Store description is the description field in .actor/actor.json.

Books to Scrape (https://books.toscrape.com/) is a public crawler practice site, not a client.

Problem → Solution → Result

Problem

• A small portfolio of API and operations demos had no example of fetching public pages and turning them into rows.
• SEO checks are easy to describe and tedious to repeat by hand: missing titles, long descriptions, more than one h1, noindex, slow responses, images without alt text, one-way hreflang, and duplicate snippets.

Solution

• An async crawler (httpx, selectolax, no browser) stays on the start hostname, honors robots.txt when asked, and limits each hostname to 5 requests per second.
• Each fetched page becomes one record with a fixed list of issue codes and a page score. A SUMMARY record counts those issues, the average response time, the five slowest pages, the site score, site-level findings, and the AI readiness result.
• The same summary builds REPORT.html (one file, inline CSS, no external resources) and REPORT.md. The demo downloads those two files from the same function.
• The Apify Actor charges page-audited once per stored dataset row, as the row is written, and stops when the run hits the caller's charge limit. When AI readiness is on, it also records ai-readiness-check once per start domain. The platform events apify-actor-start and apify-default-dataset-item stay in place. At listing, price only page-audited. Leave apify-default-dataset-item unpriced so the same page is not charged twice. Leave apify-actor-start at the platform default. Do not put a price for ai-readiness-check in this repository.
• The demo on this host only accepts books.toscrape.com, quotes.toscrape.com, and https://3.1.79.63/portfolio/. It allows at most 50 pages, one audit at a time, and 3 audits per address every 10 minutes.

Result

• The live page runs an audit, shows the score, filters the table by issue code, and downloads CSV, JSON, Markdown, or HTML. Recent audits are kept in SQLite.
• A local Actor run with the prefill input finishes inside a minute and writes at least 20 dataset rows plus SUMMARY, REPORT.html, and REPORT.md.

These are capabilities of this project, not claims about a client engagement.

Features

• Page score from 0 to 100, starting at 100, with one subtraction per page-level issue code. The points are the table below.
• Site score: the average of the page scores, then one subtraction per site-level finding. Copying a finding onto several pages does not subtract it again. Category scores are on-page, links, international, and AI readiness.
• Each issue has a severity (error, warning, or notice) and a fixed English fix.
• Site-level hreflang checks: return link when the target was crawled, x-default, self reference, a simple BCP 47 tag, and a target that is not HTTP 200 or is noindex.
• Duplicate title groups and duplicate meta description groups, for values used by two or more URLs.
• Canonical target not HTTP 200, canonical chain (A to B to C), and several URLs canonicalizing to one noindex page.
• AI readability once per start domain. checkAiReadiness defaults to true.
• HTML and Markdown reports in the key-value store and on the demo.

PageSpeed, Core Web Vitals, and orphan-page detection are not in this version.

Input

{
        "startUrls": ["https://books.toscrape.com/"],
        "maxPages": 20,
        "maxDepth": 2,
        "checkExternalLinks": false,
        "respectRobotsTxt": true,
        "concurrency": 5,
        "checkAiReadiness": true
      }

Output

One dataset item per audited page:

url, finalUrl, statusCode, responseMs, contentType, title, titleLength, metaDescription, metaDescriptionLength, canonical, robotsMeta (noindex / nofollow), h1Count, h1First, wordCount, imagesWithoutAlt, internalLinks, externalLinks, brokenLinks, ogTitle, ogImage, hreflang (language codes), hreflangLinks (lang and href), issues (code strings), issueDetails (code, severity, fix), score, crawledAt.

The key-value record SUMMARY has pages, issueCounts, averageResponseMs, slowestPages (up to five), siteScore, scores (onPage, links, international, aiReadiness), siteIssues, duplicateTitles, duplicateDescriptions, topIssuePages (up to ten), aiReadiness, and charges.

siteScore is the average page score minus each site-level finding once, clamped to 0–100. scores.aiReadiness is null when the check did not run. International starts at 100 when the crawled pages have no international findings. A page with no hreflang is not, by itself, an international finding.

REPORT.html and REPORT.md are built from that summary. The HTML file uses a <style> block and does not load a script, a stylesheet link, or a remote image.

The dataset view Overview shows url, statusCode, title, issues, and responseMs. The Scores view shows url, score, statusCode, issues, and responseMs.

Sample output

Local prefill run on 2026-10-03 against Books to Scrape (maxPages 20, maxDepth 2, checkAiReadiness true). The run stored 20 pages in 5.3 seconds. Every page had MISSING_META_DESC and score 85. SUMMARY reported averageResponseMs 95.8, siteScore 62, and charges of 20 page-audited plus 1 ai-readiness-check. The homepage and /index.html share one title, so the run reported one DUPLICATE_TITLE. robots.txt returned 404, so all five AI crawlers are unspecified and that status is not deducted. /llms.txt, /llms-full.txt, and /sitemap.xml returned 404. responseMs is the HTTP time only.

First dataset row:

{
        "url": "https://books.toscrape.com/",
        "finalUrl": "https://books.toscrape.com/",
        "statusCode": 200,
        "responseMs": 233,
        "contentType": "text/html",
        "title": "All products | Books to Scrape - Sandbox",
        "titleLength": 40,
        "metaDescription": null,
        "metaDescriptionLength": 0,
        "canonical": null,
        "robotsMeta": {"noindex": false, "nofollow": false},
        "h1Count": 1,
        "h1First": "All products",
        "wordCount": 309,
        "imagesWithoutAlt": 0,
        "internalLinks": 73,
        "externalLinks": 0,
        "brokenLinks": [],
        "ogTitle": null,
        "ogImage": null,
        "hreflang": [],
        "hreflangLinks": [],
        "issues": ["MISSING_META_DESC", "DUPLICATE_TITLE", "LLMS_TXT_MISSING", "SITEMAP_UNREACHABLE", "SITEMAP_NOT_DECLARED", "LLMS_FULL_TXT_MISSING"],
        "issueDetails": [
          {"code": "MISSING_META_DESC", "severity": "error", "fix": "Add a meta description that summarizes this page."},
          {"code": "DUPLICATE_TITLE", "severity": "warning", "fix": "Give each URL its own title."},
          {"code": "LLMS_TXT_MISSING", "severity": "warning", "fix": "Publish /llms.txt with an H1 title and a markdown list of links."},
          {"code": "SITEMAP_UNREACHABLE", "severity": "warning", "fix": "Serve /sitemap.xml with HTTP 200."},
          {"code": "SITEMAP_NOT_DECLARED", "severity": "notice", "fix": "Declare the sitemap URL in robots.txt."},
          {"code": "LLMS_FULL_TXT_MISSING", "severity": "notice", "fix": "Publish /llms-full.txt if you want a full-text companion file."}
        ],
        "score": 85,
        "crawledAt": "2026-10-03T07:24:58Z"
      }

SUMMARY from the same run:

{
        "pages": 20,
        "averageResponseMs": 95.8,
        "siteScore": 62,
        "scores": {"onPage": 79, "links": 100, "international": 100, "aiReadiness": 83},
        "issueCounts": {"MISSING_META_DESC": 20},
        "duplicateTitles": [
          {
            "value": "All products | Books to Scrape - Sandbox",
            "urls": ["https://books.toscrape.com/", "https://books.toscrape.com/index.html"]
          }
        ],
        "duplicateDescriptions": [],
        "charges": {"page-audited": 20, "ai-readiness-check": 1},
        "aiReadiness": {
          "checked": true,
          "domains": [
            {
              "domain": "books.toscrape.com",
              "robotsFound": false,
              "robotsStatus": 404,
              "bots": {
                "GPTBot": "unspecified",
                "ClaudeBot": "unspecified",
                "PerplexityBot": "unspecified",
                "Google-Extended": "unspecified",
                "CCBot": "unspecified"
              },
              "llmsTxt": {"exists": false, "statusCode": 404, "contentType": "text/html", "startsWithH1": false, "hasMarkdownLinkList": false, "bytes": 153},
              "llmsFullTxt": {"exists": false, "statusCode": 404},
              "sitemap": {"declaredInRobots": false, "declaredUrls": [], "sitemapXml": {"accessible": false, "statusCode": 404}}
            }
          ]
        }
      }

The issueCounts object in the stored SUMMARY also includes the other page-level codes with a count of 0. The excerpt above shows the non-zero count.

Issue codes

Page-level codes are the only codes in issueCounts. Site-level codes are listed in siteIssues and copied onto the related pages. A page score does not subtract site-level codes.

TITLE_TOO_LONG is not added when the title is missing. The same is true for the meta description and for MISSING_H1 versus MULTIPLE_H1. A length of 60 or 160 is allowed. A response of 2000 ms is allowed.

Deduction table

A page starts at 100. Each page-level code subtracts its own points once. Severity is a label. It does not add a second penalty. Points differ inside one severity because the effect differs: a missing title removes the snippet, and a long title only risks truncation.

The site score is clamp(average(page scores) − site findings). The average is an integer, half up. Each finding subtracts once, even when the code is written onto more than one page. Two different findings that share a code (two one-way hreflang edges, or two disallowed AI crawlers) each subtract once. The result is clamped to 0–100.

Category scores use the same points. On-page and links average the per-page scores in that category, then subtract site findings in that category. International and AI readiness start at 100 and subtract only their findings. The site score is not the average of the four category scores.

Page rules

Site rules

An AI crawler that is not named in robots.txt is unspecified. That status is reported and does not subtract points. User-agent: * is not a named decision for these five crawlers. A missing /llms.txt is LLMS_TXT_MISSING. A file that returns 200 but has no # title or no markdown link list is LLMS_TXT_MALFORMED, not both. /llms-full.txt is only an existence check.

Hreflang findings are raised only on pages that list at least one alternate. The return-link and bad-target checks run only when the target URL was crawled. A canonical chain is three or more URLs (A to B to C), including a cycle. One hop is not a chain. CANONICAL_TO_NOINDEX requires two or more source URLs. Empty titles and empty descriptions are not grouped.

Pricing

Pay per event. The primary event is page-audited at a suggested $1.00 per 1,000 pages ($0.001 per page). At listing, price only page-audited. Keep the event names apify-actor-start and apify-default-dataset-item, but do not set a price on apify-default-dataset-item (the same page would be charged twice). apify-actor-start stays the platform default. ai-readiness-check is recorded once per start domain and has no price in this code. Charge each dataset row as it is written, and stop when eventChargeLimitReached is true. See PRICING.md. The account owner sets the final price after measuring a free-tier run. Publish with limited permissions.

Limits

• No browser and no JavaScript rendering. Content that appears only after a script runs is invisible to this crawler.
• The crawler stays on the start hostname. It does not log in.
• robots.txt is honored when respectRobotsTxt is true. A 404 or an HTML body is treated as "no rules".
• It does not collect email addresses, phone numbers, or personal names. Those strings are not fields in the output.
• Requests to loopback, link-local, and private IP addresses are refused.
• External link checks are off by default and, when on, stop after 20 URLs on a page.
• Each hostname is limited to 5 requests per second. Concurrency does not raise that cap.
• The demo is a smaller door onto the same core: allow-listed hosts, 50 pages, one audit at a time, 3 audits per address every 10 minutes.
• PageSpeed, Core Web Vitals, and orphan pages (a sitemap diff) are left for a later version.

Run the Actor locally

From this directory, with the demo virtualenv (Python 3.9 or newer):

.venv/bin/python -m actor

With no INPUT.json, the run uses the input schema prefill. It writes storage/datasets/default/*.json, storage/key_value_stores/default/SUMMARY.json, REPORT.html, and REPORT.md, then prints SUCCEEDED. On Apify the image is apify/actor-python:3.12 and APIFY_IS_AT_HOME=1 uses the SDK. Do not push this Actor until the account owner says to.

Run the demo

./run.sh

The process listens on 127.0.0.1:18806. Open http://127.0.0.1:18806/. The public URL is https://3.1.79.63/portfolio/siteatlas/. Links are relative, so the same page works at the process root and under that prefix.

curl -fsS http://127.0.0.1:18806/healthz