Identify the robot and its role

OpenAI distinguishes between OAI-SearchBot used for searching, GPTBot related to content that can be used for training, and ChatGPT-User supporting some user-initiated actions. OAI-SearchBot and GPTBot settings are independent; rules for human-initiated downloads may be different.

Source: OpenAI: robot documentation ↗

Don't assume that a similar name means the same feature from another vendor. In the inventory, record the name of the agent, the source of the documentation, the decision made and the date it was reviewed. Work scope and documentation may vary.

Check the declaration for a specific address

Read the current robots.txt and evaluate whether it matches the path of the page you want to share. General Allow does not always describe the effect of a more specific rule. Don't overwrite the entire file with a ready-made example: it could remove intentional limitations on other parts of your site.

An example of a decision, not a ready file to paste

The publisher wants to make guides available in ChatGPT searches and limit downloads via GPTBot. It saves these two preferences separately and then checks them for the guide and the rest of the directories. It does not change the protection of customer accounts.

The rule does not confirm actual access

A firewall, bot protection, or required session may block the download despite no ban in robots.txt. Analyze server responses and logs. The User-Agent inscription itself is not proof of the sender's identity; use vendor verification methods for security exceptions.

Don't disable security for your entire site just to have the test turn green. The scope of the exception should match the actual need and be reviewed by the administrator.

How to read the result of our analysis

The declared access result is an interpretation of a public rule, not a confirmation of a robot visit. File missing, download error and download prohibited are different states. The report does not confirm that the content was used in training or response. Such a claim requires data other than a single page read.

FOR USE IN PRACTICE

Your checklist

  • Separate user search, training and downloads.
  • Record the source and date of the decision for each agent.
  • Test the rules for real paths.
  • Compare declarations with server responses and logs.

Sources and news

Documentation reviewed on September 21, 2026. The examples in the article are illustrative - they do not describe audit results or client implementation.

Educational material developed with the help of AI based on the indicated sources. It does not replace an individual assessment of the website by a specialist. Editorial rules.