Sitemap and Robuts.txt

As part of this blog, we will try to get overall undertaking around sitemap and rotbots.txt implementation in Edge Delivery Services.

Sitemap

In Edge Delivery Services, Sitemaps are based on indexes. For automatic sitemap generation, please ensure that you have already added the query-index document as explained in the indexing document.

If we already followed steps for setting up an initial query index in the document indexing, the system will generate a sitemap at the location:

https://<branch>–<repo>–<owner>.hlx.page/sitemap.xml

And a sitemap configuration at the following location:

https://<branch>–<repo>–<owner>.hlx.page/helix-sitemap.yaml

Simple Sitemap

The following is a simple helix-sitemap.yaml. It assumes a single index containing all the pages that need to appear in the sitemap.

sitemaps:
   example:
     source: /query-index.json
     destination: /sitemap-en.xml

If you want last modification dates to be included in the URLs of your sitemap, add a lastmod property including a format to your configuration.

sitemaps:
   example:
     source: /query-index.json
     destination: /sitemap-en.xml
     lastmod: YYYY-MM-DD

Multiple Sitemaps

It is common to have sitemaps per section of the sites and/or per country or language. AEM supports sitemaps including the corresponding hreflang references. In the following example we assume that there is a one to one mapping between the indexes and the sitemaps XML files.

sitemaps:
   example:
     languages:
       en:
         source: /en/query-index.json
         destination: /sitemap-en.xml
         hreflang: en
       fr:
         source: /fr/query-index.json
         destination: /sitemap-fr.xml
         hreflang: fr
         alternate: /fr/{path}

Note: HREFLang is supported as part of sitemaps (sitemap.xml) and can be configured in a flexible manner to reference different locales where needed.

Robots.txt

We can add robots.txt as part of our code in root folder as shown below: