A website can contain far more pages than visitors see through its main navigation. Learning how to find all pages gives content teams, developers, and site owners a stronger starting point for audits, redesigns, migrations, and ongoing maintenance.
The objective is not to produce the biggest possible spreadsheet. It is to create a dependable inventory of public URLs, understand where each URL came from, confirm what it does, and assign an appropriate next step.
Why URL Audits Matter
A homepage crawl only finds URLs that are reachable through links it can follow. Important pages may be omitted from navigation, hidden in an old folder, available only through a sitemap, or linked from another part of the site. A URL audit combines several views so teams can identify content that needs improvement, redirect planning, internal links, protection, or removal.
This work is especially valuable before a platform change or domain migration. It can also support broken-link reviews, duplicate-content checks, content ownership discussions, and decisions about which pages deserve continued maintenance.
What Counts As A Website Page?
For an audit, a public page is generally a URL that can be requested without private credentials or special permission. That definition should be adjusted to match the project scope. Some audits include only live HTML pages, while others also track PDFs, feeds, images, product files, and downloadable documents.
- Live pages: URLs that return useful content, often with a 200 response.
- Redirecting URLs: Old or alternate addresses that send visitors elsewhere.
- Deleted pages: URLs that return 404 or 410 responses.
- Parameter URLs: Addresses with filters, tracking codes, sort options, or session values.
- Alternate versions: Print pages, paginated listings, language variants, and mobile or legacy paths.
A URL can exist without being indexed by a search engine, and a URL can appear in search data after it has been removed from a CMS. Treat those differences as findings to investigate, not automatic errors.
Where URL Lists Come From
Use more than one source because each source answers a different question. Internal links reveal what a crawler can reach. CMS exports reveal what editors have published. Search and log data can reveal URLs that are no longer well connected but still receive requests.
A sitemap is a declared list of pages and files a site owner considers important. Google explains that sitemaps help search engines discover URLs, but inclusion does not guarantee crawling or indexing. Check sitemap indexes, individual sitemap files, robots.txt references, CMS exports, crawl exports, Search Console data, and server logs where access is available.
A Step-By-Step URL Audit Workflow
1. Set A Clear Scope
Define the domain, subdomains, folders, language versions, and protocols under review. Record the audit date, intended purpose, and whether files such as PDFs belong in the inventory. A clear scope prevents unrelated URLs from obscuring important findings.
2. Collect And Normalize URLs
Combine exports into one working list, while preserving the original URL and its discovery source. Then normalize a separate version by applying consistent rules for HTTPS, trailing slashes, hostname variations, uppercase paths, and removable tracking parameters. Do not overwrite the original address, because it may be needed to diagnose duplicates or redirects.
3. Check Each URL
Capture the status code, final destination, page title, canonical URL, content type, and last-checked date. Record whether the page receives internal links. Pages that matter should be linked clearly within the site, since crawlable internal links help search engines discover pages and help visitors move through related content.
4. Prioritize The Review
Group results by folder, content type, status, and business purpose. Address pages that affect customer journeys, conversion paths, legal information, or high-value content before spending time on low-impact parameter URLs.
How To Compare Sources
Instead of treating one source as the truth, compare where a URL appears and what that difference may mean:
- In the crawl and sitemap: The page is linked and formally listed.
- In the crawl but not the sitemap: The page may be valid, but the sitemap may need review.
- In the sitemap but not the crawl: The page may be orphaned, blocked, or unavailable through ordinary internal navigation.
- In the CMS but not the crawl: The item may be unpublished, poorly linked, or no longer live.
- In search or logs but not the CMS: The URL may be old, migrated, generated, or forgotten.
Common Findings And What They Mean
Orphan Pages And Stale Entries
Orphan pages appear in a sitemap, CMS, search report, or log file but lack internal links. Stale sitemap entries point to redirected, deleted, or outdated URLs. Both findings deserve a decision: add links, update the sitemap, redirect the URL, or remove it from active planning.
Redirect Chains, Errors, And Duplicates
Redirect chains occur when one URL sends visitors through several destinations. Broken URLs return an error response. Duplicate versions can arise from HTTP and HTTPS, www and non-www hosts, trailing slashes, capitalization, and parameters. Review the intended canonical destination before changing or deleting anything.
JavaScript And Restricted Content
Some pages and links appear only after scripts run, so compare raw HTML with rendered pages when auditing a JavaScript-heavy site. Keep login-only dashboards and permission-based content outside a public inventory, and never bypass access controls to expand the audit.
How To Organize Results
A spreadsheet is sufficient for many audits when every row has a clear purpose. Useful fields include the original URL, normalized URL, discovery source, status code, final URL, title, canonical URL, content type, folder, internal-link count, owner, review date, and recommended action.
Use concise action labels such as Keep, Improve, Merge, Redirect, Remove, Investigate, and Protect. Before removing a URL, check whether it has traffic, links, conversions, or a replacement page that should receive a redirect.
Final Checklist
- Define the site area and content types under review.
- Collect URLs from crawls, sitemaps, CMS records, search data, and logs where available.
- Normalize formats while preserving original URLs.
- Check status codes, redirects, canonicals, and internal links.
- Compare sources to identify missing, orphaned, stale, and duplicate URLs.
- Assign a documented action to each important finding.
- Save the audit method and date so future reviews can measure change.
A reliable URL audit creates a practical map of a website rather than a disconnected list of addresses. When teams collect, verify, compare, and organize URLs carefully, they can make more confident decisions about content quality, technical maintenance, internal linking, and future site changes.
