Data pipelines, Applied AI2025

AI-Powered SEO Audit Engine

Turns a manual SEO QA checklist into a repeatable pipeline that crawls live pages, compares them to the brief with an LLM, and ships actionable fixes.

The starting problem

An agency manually checked whether published pages matched the SEO brief — titles, meta descriptions, H1s, target keywords — across many client sites with wildly different setups. It was slow, inconsistent, and quietly produced wrong results when a spreadsheet column was reordered.

The system in pictures

Screenshots of the system at work, step by step: what your customer sees, and what your team receives.

1. 1.

   The workflow

   

   Sitemap discovery → crawl → LLM comparison against the brief.

2. 2.

   The trigger form

   

   A simple form kicks off the audit: you enter the site and its SEO brief targets, and the engine takes over.

3. 3.

   The overall score

   

   The engine grades each page against the brief and returns a clear compliance score, so you see exactly where the site stands.

How it works

What we built

1. 1

   A crawler discovers sitemaps robustly across heterogeneous setups — pretty permalinks, query-string sitemaps, and www / non-www DNS variants — so both staging and production sites are covered.

2. 2

   Brief data is read by column header, never by position, and fails loudly if a required column is missing — so silent column drift never yields wrong-but-plausible output.

3. 3

   An LLM compares each live page against its brief and returns structured, prioritized recommendations.

4. 4

   Production-grade reliability: retry-with-backoff for rate-limited APIs, an Opus→Sonnet model fallback, and an in-workflow error sub-flow that emails failures instantly.

Highlights

• A manual QA checklist becomes a repeatable, auditable pipeline.
• Robust to messy real-world inputs: staging sites, reordered columns, DNS quirks.
• Resilient to provider outages and per-minute quota limits.

Tools used