How do I add robots.txt to Umbraco?

All of our active development sites have a QA site and sometimes these get tracked by search engines. How do add the robots.txt file to the QA site but a different one for production?

Answer

I researched and found a very good package that we can use and gives us the ability to manage the robots.txt contents through the web.config so these settings can be changed by environment.

Umbraco Package -> Friendly Robots
https://our.umbraco.com/packages/website-utilities/friendly-robots/

On your project install the latest version.
Install-Package Our.Umbraco.FriendlyRobots

In the web.config you will notice 4 new appSettings that have been added.
<add key="Umbraco.Robots.UserAgent" value="" />
<add key="Umbraco.Robots.Allow" value="" />
<add key="Umbraco.Robots.Disallow" value="" />
<add key="Umbraco.Robots.Sitemaps" value="" />

Update Umbraco.Robots.UserAgent to "*" and update Umbraco.Robots.Disallow to "/".
- This will be the default for all environments.

In the QA web.config transform, do the same step as above to replace the value for the 2 fields.

In the prod web.config transform, you need to make these changes as replacements:
1. Umbraco.Robots.Allow to "/"
2. Umbraco.Robots.Dissalow to "/umbraco"
3. set the sitemap if there is a sitemap.

Going to the URL and typing /robots.txt should bring up the page. If it does not that means there the host of the site isn't set up.
1. Go to the admin area and right click the root page in content.
2. Select Culture and Hostnames
3. Ensure the domain that you are viewing the site with is an entry here.

1. Add new Domain
2. Enter the full domain with https and select english.
3. Click Save.

Now the /robots.txt page should display.