Start with a question your crawl can answer
An open-source SEO audit works best as a small investigation: define the pages that matter, crawl them, verify the findings and assign fixes. Installing several tools is the easy part. The useful deliverable is a short list of problems with evidence and a clear way to confirm that each one is fixed.
For a first audit, use a question such as “Can visitors and crawlers reach every public product page after our migration?” That gives you a URL set and a reason to investigate failures. “Find every SEO problem” gives you an unbounded report.
- Choose the scope: production hostname, language, public sections and representative page templates. Keep account areas, checkout actions and unbounded filter combinations outside a routine crawl.
- Write down expectations: which pages should be public, which URLs should redirect and which sections are intentionally excluded.
- Save a baseline: crawl date, tool version, settings, sitemap file and a small list of important URLs. You will need the same scope for the validation run.
This workflow now includes our measured results from a small, intentionally faulty local site. The test covers selected findings, not crawler speed or every feature. Open-source licensing also does not remove hosting, setup or maintenance work.
Choose a crawler and one supporting check
Start with one crawler so you have a consistent URL inventory. Add a specialist tool when a finding needs a different kind of evidence. Running four reports does not make the underlying diagnosis four times stronger.
| Task | Starting point | How to use the output |
|---|---|---|
| Crawl and review SEO issues | SEOnaut | Build a technical issue list, then inspect affected URLs and templates. |
| Inspect a site across technical disciplines | SiteOne Crawler | Use its crawling and analysis reports as starting evidence for SEO, performance and related checks. |
| Run Lighthouse audits across pages | Unlighthouse | Investigate page templates with performance or accessibility findings separately from crawl errors. |
| Compare sitemap snapshots | Sitemap Cohort Auditor | Identify added and removed URLs around a release; verify the affected pages with a crawler. |
These roles follow the projects’ own descriptions: SEOnaut, SiteOne Crawler, Unlighthouse and Sitemap Cohort Auditor. Check installation requirements and current licensing before adopting a project. For a broader selection, see our open-source SEO tools shortlist.
Do not combine different tools’ scores into an “SEO health” average. A Lighthouse score, a sitemap warning and a crawler’s severity label measure different things. Keep the original evidence and explain the consequence in plain language.
Run a focused crawl, then inspect the gaps
- Start conservatively. Crawl a small public section with a low request rate. Check response times and server errors before expanding. Record page limits, exclusions and whether JavaScript rendering is enabled or available.
- Compare discovery paths. Look at URLs found through internal links and URLs declared in the sitemap. A page absent from a crawl may be outside its settings rather than absent from the site.
- Group findings by template. Put product, category and article pages into separate groups. Twenty copies of the same bad canonical often point to one template bug, not twenty independent content tasks.
- Check the actual response. Open a few affected URLs and inspect their status, canonical, robots directives and visible content. If JavaScript supplies key content, compare the initial HTML with the rendered page.
- Use another data source for search outcomes. Search Console can help you investigate indexing and performance. A successful crawl alone cannot tell you whether Google indexed a page or whether it earns clicks.
A useful first export contains URL, final status, redirect destination, canonical, robots directives, title, H1 and links pointing to the page where the tool provides them. If a field is unavailable, record that limitation instead of treating an empty cell as proof of a problem.
Review sitemap discrepancies with the XML sitemap audit checklist. For performance findings, test a few representative pages under comparable conditions and keep that investigation distinct from URL discovery.
Prioritize by consequence, scope and confidence
Sort the queue around the pages you need people to find. A broken primary category link can deserve attention before hundreds of repetitive metadata warnings. Use three questions for every finding:
- Consequence: what can a visitor or crawler no longer do?
- Scope: how many intended public pages or shared templates are affected?
- Confidence: did you reproduce the behavior, or is it still a tool-generated lead?
Then record the owner, proposed change and acceptance check. “Fix canonicals” is too vague. “Product template emits the category URL as canonical; correct the template and verify five products plus the next full crawl” is a reviewable task.
Be especially careful with access and indexing directives. Google needs crawl access to see a page’s noindex instruction; blocking the same URL in robots.txt can prevent that instruction from being read. Check the intended outcome before changing either. Google’s noindex guidance ↗
Our test: turn findings into a fix queue
On 29 September 2026 we ran SiteOne Crawler 2.5.1, lychee 0.24.2 and Sitemap Cohort Auditor 0.2.2 against a controlled local fixture. The defects were intentional; the reports are actual tool output.
| Observed finding | Evidence | Action to validate |
|---|---|---|
| Missing page and image | SiteOne recorded 2 HTTP 404 responses; lychee reported both links. | Restore intended resources or update their source links, then recheck the destinations. |
| Link to a missing fragment | lychee reported /healthy#absent with fragment checking enabled. | Correct the fragment or add the intended target ID. HTTP 200 alone cannot settle this. |
| Missing metadata and noindex | SiteOne reported a missing title, missing description, duplicate titles and one noindex page. | Compare each directive with intent; fix shared templates where appropriate. |
| Duplicate and changed sitemap URLs | Sitemap Cohort Auditor found 1 duplicated URL, 1 invalid date, 3 additions and 1 removal. | Correct duplicate/date output and classify each URL change before release. |
Download the sample, raw results and reproduction commands. The link checker and sitemap auditor supplement the crawl; neither missing metadata in a link report nor a 404 absent from a sitemap-only report is an unexpected failure.
A checklist for the audit handoff
Copy this structure into your issue tracker. Keep one record per underlying problem, with a URL sample and the full affected list attached when available.
Finding:
Expected behavior:
Observed behavior:
Affected URLs / shared template:
Evidence: tool, version, date, settings, response sample
Confidence: reproduced / needs investigation
Proposed change and owner:
Acceptance check:
Release date:
Post-release result:
Search Console follow-up, if relevant:For the overall handoff, confirm that crawl scope and exclusions are documented, significant findings were reproduced, fixes have owners, and the validation run uses comparable settings. Keep unresolved questions visible rather than marking the whole audit “passed.”
Need a different crawler interface or a managed service? Compare the technical SEO audit tools in the directory. If your next question concerns actual impressions and clicks, use the Search Console workflow with authorized data.
Common questions
Can open-source tools replace a paid SEO platform?
They can cover specific jobs such as crawling and sitemap comparison. Decide which data and workflow you need first. Running your own crawler does not provide a commercial keyword database, ongoing hosting or a support team.
Should I fix every warning?
Investigate before changing the site. Some warnings describe intentional behavior, and several may share a single cause. Prioritize reproduced problems on pages that matter to your audience.
How often should I repeat the audit?
Use the release process as a trigger: URL changes, new templates, migrations and access-rule changes deserve focused checks. Choose any recurring crawl frequency around the site’s rate of change and server capacity.