What robots.txt does

A file placed in the host's root directory specifies download rules for specific robots. It is not intended to secure data. A blocked address may still appear in results based on links from other sites.

Source: Google: destiny robots.txt ↗

Before you change the rule, write down sample addresses to be accessible and blocked. Test both the catalog and subpages with parameters. Check the correct host - rules for a subdomain are not automatically rules for the entire organization.

When does noindex work?

The noindex directive can be passed in the robots meta tag or the X-Robots-Tag header. For Google to see it, it must be able to download the resource. Blocking the same address in robots.txt may therefore prevent the noindex from being read.

Source: Google: indexing blocking ↗

Example for a public thank you page

If you want your page to be accessible after submitting a form but not appear in Google, you may want to consider noindex while still being downloadable. Do not place customer data on it that is accessible without authentication.

How not to block the entire website

A common organizational pitfall is transferring test environment settings to production. After publishing, check several types of pages, their headers and robots.txt. Don't limit your control to the home page - your product template may have different settings.

Store changes in version history. Include the expected result for each sample address in your order. This allows the intention to be verified even by a person who did not write the rules.

Privacy requires access control

The customer panel, order documents and private files require authentication and correct authorization. Robots.txt is public and does not force any program to follow rules. Noindex also does not provide protection against someone knowing the address. With private data, the task is to secure access, not just to remove the result from the search engine.

FOR USE IN PRACTICE

Your checklist

  • Write down your goal: limit downloads or disable indexing.
  • Don't block reading of a noindex statement by mistake.
  • After publication, check several types of subpages.
  • Secure private resources on the server side.

Sources and news

Documentation reviewed on September 21, 2026. The examples in the article are illustrative - they do not describe audit results or client implementation.

Educational material developed with the help of AI based on the indicated sources. It does not replace an individual assessment of the website by a specialist. Editorial rules.