Recently, when I tried to have Claude Code read articles from this site to assist with tasks, I encountered a problem where it couldn’t properly retrieve the content.

After investigation, I discovered that the cause was Cloudflare’s automatic robots.txt management feature. This article explains the background and steps I took to resolve this issue through configuration changes.

Problem: AI Bots Couldn’t Read Content

Initially, I hadn’t paid much attention to robots.txt beyond setting it to Allow, and was running with settings close to Cloudflare’s defaults. Specifically, under the “Managed robots.txt” feature, “Content Signals Policy” was selected.

Previous Setting: [Overview] > [Control AI Crawlers] > Managed robots.txt

• Content Signals Policy

This setting blocks crawling from major AI bots (CCBot, GPTBot, anthropic-ai, etc.) at the robots.txt level, with the purpose of preventing site content from being used as AI training data without consent.

As a result, when Claude tried to reference articles on this site, access was denied according to the Content Signals Policy rules.

Solution: Change to Direct AI Bot Traffic Setting

To resolve this problem, Cloudflare provides a more flexible option: the “Direct AI bot traffic via robots.txt” setting.

New Setting: [Overview] > [Control AI Crawlers] > Managed robots.txt

• Direct AI bot traffic via robots.txt

When you change to this setting, Cloudflare adds Disallow rules for the AI bot group (Cloudflare-AI-Bot-Group) to your existing robots.txt file (if one exists). Site operators can use this group to control allow/disallow permissions more granularly, or override with their existing robots.txt.

This time, I decided to change to this setting to promote AI utilization of site information.

Steps to Change Settings

1. Log in to the Cloudflare dashboard.
2. Select the target website.
3. Navigate to [Overview] from the left menu.
4. Confirm that the [Control AI Crawlers] tab is open and find the “Managed robots.txt” section.
5. In the displayed modal window, select “Direct AI bot traffic via robots.txt”.

Result: Claude Code Can Now Read Articles!

After changing the settings and waiting a while, I tried specifying the site’s article URLs in Claude Code again, and this time it successfully read the content, enabling me to delegate tasks in daily product development.

With this change, I hope that the information published on this site will reach and be utilized by more people through generative AI. As AI-powered information gathering becomes commonplace, I was reminded that robots.txt management is once again an important topic for site operators.

I hope this helps others facing similar issues.

That’s all from the Gemba—a report on changing Cloudflare’s robots.txt management from “Content Signals Policy” to “Direct AI bot traffic via robots.txt” to allow AI bot crawling.

References

• Managed robots.txt · Cloudflare Bots docs
• Updated robots.txt Based on Content Signals to Open All Content for AI Training