Run a free Peek AI visibility scan on your own domain before you read further, because the answer to the headline question is usually yes. A January-February 2026 study by Otterly.ai analyzed more than one million AI citations across ChatGPT, Perplexity and Google AI Overviews and found that 73 percent of the sites in the sample had a technical barrier, a robots.txt block, a CDN rule, or JavaScript-only rendering, that kept AI crawlers from ever reading the page, regardless of how good the content was. For a treatment center, that is not a ranking problem. It is an invisibility problem that happens before content quality is even evaluated.
A family in Denver, Nashville, or Scottsdale asking ChatGPT or Perplexity “residential treatment near me that takes my insurance” is not getting an answer synthesized from every program’s website. They are getting an answer built from whatever pages the underlying crawler could actually reach. If GPTBot or PerplexityBot hit a “Disallow: /” rule on your domain, your clinical content, your admissions FAQ, your outcomes page, none of it exists to that model. A competitor with a mediocre page and an open robots.txt shows up. You do not.
The Three Barriers, and How to Check Your Own Site in 10 Minutes
| Barrier | What It Does | How to Check It Yourself |
|---|---|---|
| robots.txt block | Explicitly tells AI crawlers not to read your pages, often added by a security plugin or a CDN default without anyone realizing it applies to AI bots too | Visit yourdomain.com/robots.txt and search for GPTBot, ChatGPT-User, ClaudeBot or PerplexityBot next to a “Disallow: /” line |
| CDN or WAF restriction | Security rules built to stop scrapers and bad bots often block AI crawler user agents as collateral damage | Run curl -A "GPTBot" https://yourdomain.com/ from a terminal and see whether you get the real page or a block page |
| JavaScript-only rendering | If your admissions or program pages only render after JavaScript runs, a crawler that reads raw HTML sees an empty shell | View page source (not the rendered page) and confirm the actual text of your admissions content is present, not just a loading placeholder |
These barriers stack. A site can pass the robots.txt check and still fail on CDN rules or JS rendering, and the Otterly.ai study treated all three as one combined citability score because a crawler only has to fail once to never see the page.
Which Crawlers Actually Matter for a Treatment Center’s Visibility
| Crawler | Platform | What It Does |
|---|---|---|
| GPTBot | OpenAI | Crawls content used to train the underlying model |
| OAI-Searchbot | OpenAI | Fetches pages in response to a live ChatGPT search query |
| ChatGPT-User | OpenAI | Fires when a user action inside ChatGPT or a Custom GPT visits a page directly |
| PerplexityBot / Perplexity-User | Perplexity | Finds and visits pages to answer a user’s question, then links the source in the answer |
| ClaudeBot / Claude-Searchbot / Claude-User | Anthropic | Same split pattern: training crawl, live search fetch, and direct user-triggered visit |
The study’s platform-level breakdown is the part treatment center marketing teams tend to miss. ChatGPT leans on Reddit, Wikipedia and news coverage, and gives a brand a citation without a clickable link 55 percent of the time, meaning ChatGPT will describe your program without ever sending a family to your site. Perplexity leans even harder on community forums (16.9 percent of its citations are Reddit alone) but pairs mentions with real links more consistently, so a Perplexity citation is worth more in referral terms. Google AI Overviews show the strongest brand preference of the three, 59.8 percent of citations go to an actual business site rather than a forum or aggregator, but the AI Overview itself only appears on roughly a third of health queries in the first place. Three different environments, three different jobs, and a single crawler block takes you out of all three at once.
What This Looks Like Inside BSPKN’s Own Work
Internal finding, not a client result: when we baseline-audited BSPKN’s own AI visibility in April 2026 across 30 real prompts a prospective client might ask, we appeared in zero answers. Part of that gap traced back to exactly this problem: a Cloudflare-managed robots.txt rule was silently blocking GPTBot, ClaudeBot and Google-Extended on our own domain before we found and fixed it. If it happened to the agency that watches for this, it is worth checking on a treatment center’s site that has never had a reason to look.
What to Do With What You Find
- If robots.txt disallows any of the crawlers above, add explicit allow rules for GPTBot, ChatGPT-User, PerplexityBot and ClaudeBot.
- If your CDN or security plugin is blocking non-browser agents, allowlist the AI crawler user agents rather than rate-limiting them to zero.
- If your admissions and program pages depend on JavaScript to render the actual text, add a server-rendered or static HTML fallback so the words are in the page source, not just the rendered DOM.
- Once access is confirmed, structure the content itself: direct-answer paragraphs, a real FAQ block, and named locations and program details a model can quote without guessing.
“We found this exact block on our own domain before we ever went looking for it on a client’s,” said Ryan Rivard, BSPKN’s founder. “It is the kind of gap that never shows up until someone runs the check, which is the whole reason a scan matters more than another opinion about your content.” This crawler-access check is the first step in every Propel OS AI visibility engagement, before any content work starts.
None of this requires a redesign. It is usually a robots.txt edit and a CDN allowlist entry, the kind of fix that takes a developer under an hour once someone knows to look for it. The problem is that almost nobody looks, which is exactly what a free Peek AI visibility scan is built to surface: whether your program is reachable, and what a family actually sees when they ask an AI assistant about care in your market.
FAQ
How do I know if AI crawlers can actually reach my treatment center’s website?
Check your robots.txt file for a “Disallow” rule next to GPTBot, ChatGPT-User, PerplexityBot or ClaudeBot, then test with a command like curl -A "GPTBot" https://yourdomain.com/ to see whether the real page loads or a security rule blocks it. A free Peek scan checks this automatically along with your broader AI visibility.
Why would a CDN or security plugin block AI crawlers if I never asked it to?
Most CDN and firewall defaults were written to stop scrapers and bad bots long before AI crawlers existed, so they often catch GPTBot, ClaudeBot and PerplexityBot as collateral damage. Cloudflare’s own “Managed robots.txt” feature has done this by default on sites that never touched the setting.
Does fixing crawler access guarantee my program gets cited by ChatGPT or Google AI Overviews?
No. Fixing access removes the barrier that makes citation impossible; it does not make citation automatic. Once a crawler can read the page, the content still has to answer a real question directly, in a form a model can quote, which is the content side of this same problem.
Does this matter if most of my admissions still come from referrals, not search?
It matters more every quarter, not less, because the families and referral sources doing independent research before ever calling are increasingly starting that research with an AI assistant instead of a search engine. A program invisible to that layer is invisible earlier in the decision than it used to be.
AI visibility work for a treatment center starts with access, not content. GEO strategy built for behavioral health and healthcare marketing programs both start with the same crawler-access check described here, then move to the content and structure that turns access into an actual citation. See more of how we approach this on our success stories page, or read how AI systems decide what to cite once they can actually see your pages in What AI Actually Cites for Treatment Centers in 2026.
Start with the free Peek AI visibility scan to see whether your own site has this problem, then book a 15 minute strategy call to talk through what the scan found.
