Skip to content

Japon Sakura

Actualité

How to Explore a Website Efficiently Using Its Sitemap?

A sitemap lists a domain’s URLs in a structured file, usually in XML format. Explore a website from…

Femme analysant un plan de site web imprimé sur un bureau en open space, explorant la hiérarchie de navigation d'un site internet

A sitemap lists a domain’s URLs in a structured file, usually in XML format. Exploring a website from this file makes it possible to map its content quickly, but a sitemap does not always reflect every published page.

Google explains that a sitemap helps make URLs known, but does not guarantee that they will be crawled or indexed. Measuring the gap between the URLs declared in the sitemap, those connected by internal links, and those actually indexed by search engines gives a much more reliable picture of a site’s health.

XML sitemap, internal links and Google index: three sources, three views of a site

Each data source covers a different scope. Comparing them reveals blind spots that a sitemap-only crawl would miss.

Source What it lists Main limitation
XML sitemap URLs declared by the site owner May omit existing pages or list outdated URLs
Internal links (crawl) Pages actually linked to from other pages on the site Does not detect orphan URLs (with no inbound links)
Google index (Search Console) URLs crawled and/or indexed by Googlebot A URL in the sitemap may still not be indexed

Cross-referencing these three inventories is the only reliable way to identify orphan pages, ghost URLs and content invisible to search engines. An audit crawler explores the site by following links, while Search Console shows how Google handles each URL.

On the All News sitemap, for example, each section is listed with its URL and last-modified date, making it easy to quickly check the domain’s editorial coverage.

Man standing in front of a screen displaying a structured website sitemap with navigation nodes in a home office

Orphan pages: how the sitemap exposes forgotten URLs

An orphan page is a URL that exists on the server but is not linked to from any other page on the site. It remains accessible via its direct address and may sometimes be listed in the sitemap, but visitors browsing normally will never find it.

Find orphan pages in three steps

  • Extract the complete list of URLs from the XML sitemap (using a parser or a crawling tool such as Screaming Frog, Sitebulb or a basic Python script)
  • Crawl the site starting from the homepage to collect all URLs reached through internal links
  • Compare the two lists: any URL in the sitemap but missing from the crawl may be an orphan, while any URL found in the crawl but missing from the sitemap has been left out

The reverse comparison is just as revealing. A page with plenty of internal links but missing from the sitemap may go unnoticed in an audit limited to the XML file. Both discrepancies point to an architectural problem that hinders crawling by search engines and users alike.

Common case: URL variations

Duplicate pages often appear when the sitemap lists one version of a URL (with or without a trailing slash, HTTP or HTTPS, with or without a parameter). Checking that sitemap URLs match those used in internal links helps identify these variations before they dilute the crawl budget.

Sitemap lastmod tag: useful signal or unnecessary noise

The lastmod tag indicates the date of a page’s last significant modification. Google recommends that it reflect a genuine content change, not an automatic update (cache regeneration or cosmetic template change).

Bing also uses this date as a freshness signal. However, neither search engine takes the changefreq and priority tags into account. Filling in these fields therefore does nothing to improve crawling or SEO.

A sitemap where all lastmod dates are identical (a typical sign of a poorly configured CMS) loses its value as a signal. Search engines can no longer distinguish an article updated yesterday from a page that has not changed in two years. For a content publisher, keeping lastmod dates accurate helps search engines rediscover updated pages faster.

Two colleagues exploring a website sitemap tool together on a laptop in a modern urban café

Crawling a site with an incomplete or missing sitemap

Not every site publishes a sitemap, and those that do do not always list every page. In these cases, crawling relies on other methods.

  • Run a full crawl from the homepage, following all internal links, including those in the footer, menu and pagination
  • Use the site:domain.com query in Google to list pages already indexed, then compare the results with the crawl
  • Check Search Console (the “Pages” report) to identify URLs that have been discovered but not indexed, or indexed but missing from the sitemap
  • Manually check candidate URLs in the address bar (legal notices, terms and conditions, and old landing pages are often disconnected from the site’s navigation)

An audit crawler identifies the pages it reaches, their response codes and their outbound links. Search Console shows how Google actually handles a URL, covering a different angle: a page may be crawled without being indexed, or indexed and then removed.

IndexNow and sitemaps: two complementary mechanisms on Bing

Microsoft clearly distinguishes between a sitemap, which serves as a permanent inventory of URLs, and IndexNow, which sends real-time notifications about additions, changes or deletions. An IndexNow acknowledgment confirms that a notification was received, not that the URL was indexed.

For a news site publishing several articles a day, combining an up-to-date sitemap with IndexNow pings covers both timeframes: the search engine has a complete inventory and receives an instant signal for every new piece of content. On Google, the equivalent approach is to submit the sitemap through Search Console and use the standard ping protocol.

Crawling a site from its sitemap becomes more reliable once you stop treating the sitemap as the only source. Systematically comparing the XML file, internal linking and indexing data turns a simple URL inventory into a genuine visibility audit.

How to Explore a Website Efficiently Using Its Sitemap?