Start with a question your crawl can answer
An open-source SEO audit works best as a small investigation: define the pages that matter, crawl them, verify the findings and assign fixes. Installing several tools is the easy part. The useful deliverable is a short list of problems with evidence and a clear way to confirm that each one is fixed.
For a first audit, use a question such as “Can visitors and crawlers reach every public product page after our migration?” That gives you a URL set and a reason to investigate failures. “Find every SEO problem” gives you an unbounded report.
- Choose the scope: production hostname, language, public sections and representative page templates. Keep account areas, checkout actions and unbounded filter combinations outside a routine crawl.
- Write down expectations: which pages should be public, which URLs should redirect and which sections are intentionally excluded.
- Save a baseline: crawl date, tool version, settings, sitemap file and a small list of important URLs. You will need the same scope for the validation run.
This is an editorial workflow based on project documentation, with a fictional example below. It is not a hands-on benchmark of crawler speed or issue detection. Open-source licensing also does not remove hosting, setup or maintenance work.
Choose a crawler and one supporting check
Start with one crawler so you have a consistent URL inventory. Add a specialist tool when a finding needs a different kind of evidence. Running four reports does not make the underlying diagnosis four times stronger.
| Task | Starting point | How to use the output |
|---|---|---|
| Crawl and review SEO issues | SEOnaut | Build a technical issue list, then inspect affected URLs and templates. |
| Inspect a site across technical disciplines | SiteOne Crawler | Use its crawling and analysis reports as starting evidence for SEO, performance and related checks. |
| Run Lighthouse audits across pages | Unlighthouse | Investigate page templates with performance or accessibility findings separately from crawl errors. |
| Compare sitemap snapshots | Sitemap Cohort Auditor | Identify added and removed URLs around a release; verify the affected pages with a crawler. |
These roles follow the projects’ own descriptions: SEOnaut, SiteOne Crawler, Unlighthouse and Sitemap Cohort Auditor. Check installation requirements and current licensing before adopting a project. For a broader selection, see our open-source SEO tools shortlist.
Do not combine different tools’ scores into an “SEO health” average. A Lighthouse score, a sitemap warning and a crawler’s severity label measure different things. Keep the original evidence and explain the consequence in plain language.
Run a focused crawl, then inspect the gaps
- Start conservatively. Crawl a small public section with a low request rate. Check response times and server errors before expanding. Record page limits, exclusions and whether JavaScript rendering is enabled or available.
- Compare discovery paths. Look at URLs found through internal links and URLs declared in the sitemap. A page absent from a crawl may be outside its settings rather than absent from the site.
- Group findings by template. Put product, category and article pages into separate groups. Twenty copies of the same bad canonical often point to one template bug, not twenty independent content tasks.
- Check the actual response. Open a few affected URLs and inspect their status, canonical, robots directives and visible content. If JavaScript supplies key content, compare the initial HTML with the rendered page.
- Use another data source for search outcomes. Search Console can help you investigate indexing and performance. A successful crawl alone cannot tell you whether Google indexed a page or whether it earns clicks.
A useful first export contains URL, final status, redirect destination, canonical, robots directives, title, H1 and links pointing to the page where the tool provides them. If a field is unavailable, record that limitation instead of treating an empty cell as proof of a problem.
Review sitemap discrepancies with the XML sitemap audit checklist. For performance findings, test a few representative pages under comparable conditions and keep that investigation distinct from URL discovery.
Prioritize by consequence, scope and confidence
Sort the queue around the pages you need people to find. A broken primary category link can deserve attention before hundreds of repetitive metadata warnings. Use three questions for every finding:
- Consequence: what can a visitor or crawler no longer do?
- Scope: how many intended public pages or shared templates are affected?
- Confidence: did you reproduce the behavior, or is it still a tool-generated lead?
Then record the owner, proposed change and acceptance check. “Fix canonicals” is too vague. “Product template emits the category URL as canonical; correct the template and verify five products plus the next full crawl” is a reviewable task.
Be especially careful with access and indexing directives. Google needs crawl access to see a page’s noindex instruction; blocking the same URL in robots.txt can prevent that instruction from being read. Check the intended outcome before changing either. Google’s noindex guidance ↗
Worked example: turn 31 warnings into three decisions
Fictional example. A small directory has 120 intended public URLs. Its audit produces the following groups. These numbers illustrate triage; they are not results from a tested tool or a client website.
| Finding | What the manual check shows | Next decision |
|---|---|---|
| 6 product URLs return 404 | All six products are active; the category template links to an old URL pattern. | Fix the links and handle the old URLs according to the migration plan. Verify the six destinations and the template. |
| 5 category pages have noindex | A staging directive was carried into production; these categories are intended for search. | Remove the accidental directive, inspect the live response and recheck indexing separately. |
| 20 pages share a description | The pages work and serve different products, but the shared summary is unhelpful. | Queue a content-template improvement after the access problems. Do not forecast a CTR increase from the warning alone. |
The raw total is 31 warnings. The fix queue has three owners or work items, with the first two addressing broken access or an unintended indexing instruction. After release, repeat the checks using the original settings. A lower warning count matters only if the intended behavior is now correct.
A checklist for the audit handoff
Copy this structure into your issue tracker. Keep one record per underlying problem, with a URL sample and the full affected list attached when available.
Finding:
Expected behavior:
Observed behavior:
Affected URLs / shared template:
Evidence: tool, version, date, settings, response sample
Confidence: reproduced / needs investigation
Proposed change and owner:
Acceptance check:
Release date:
Post-release result:
Search Console follow-up, if relevant:For the overall handoff, confirm that crawl scope and exclusions are documented, significant findings were reproduced, fixes have owners, and the validation run uses comparable settings. Keep unresolved questions visible rather than marking the whole audit “passed.”
Need a different crawler interface or a managed service? Compare the technical SEO audit tools in the directory. If your next question concerns actual impressions and clicks, use the Search Console workflow with authorized data.
Common questions
Can open-source tools replace a paid SEO platform?
They can cover specific jobs such as crawling and sitemap comparison. Decide which data and workflow you need first. Running your own crawler does not provide a commercial keyword database, ongoing hosting or a support team.
Should I fix every warning?
Investigate before changing the site. Some warnings describe intentional behavior, and several may share a single cause. Prioritize reproduced problems on pages that matter to your audience.
How often should I repeat the audit?
Use the release process as a trigger: URL changes, new templates, migrations and access-rule changes deserve focused checks. Choose any recurring crawl frequency around the site’s rate of change and server capacity.