AI crawlers decide what gets cited in LLM answers. Here's how to audit your crawl access, fix gaps, and build entity authority before your rivals notice.
Most generative engine optimisation conversations start in the wrong place. Marketers obsess over prompt phrasing, schema markup, and whether their FAQ content sounds conversational enough for AI Overviews — while quietly ignoring the more foundational question: can AI crawlers actually access your site in the first place?
If the answer is no, the rest of the strategy is theatre.
The Common Crawl Problem Nobody Is Talking About
Common Crawl, the nonprofit that maintains one of the largest open web archives on the internet, recently published guidance on what makes a domain visible — or invisible — to AI training pipelines. Search Engine Journal reports that Suganthan Mohanadasan took that guidance and automated it into a free diagnostic tool that checks crawl captures, robots.txt history, block fingerprints, and runs a live CCBot probe against any domain.
Why does this matter for brands in Southeast Asia? Because Common Crawl data feeds directly into the training corpora of GPT-4, Claude, Mistral, and most open-weight LLMs used to power AI search products. A brand that has historically blocked CCBot — often accidentally, through overaggressive crawl-budget rules or legacy robots.txt configurations — may have near-zero presence in the training data underpinning the AI answers their customers are already reading.
The fix isn’t complicated. Audit your robots.txt for CCBot blocks. Check your Cloudflare or WAF bot-management rules. Verify your Crawl-delay settings aren’t timing out AI agents. But you can’t fix what you haven’t diagnosed — and most marketing teams have never looked.
Entity Authority Is the New Domain Authority
Once crawl access is confirmed, the question shifts from can AI find you to does AI trust you. This is where entity authority — the consistency and richness of structured signals about who your brand is, what it does, and where it operates — becomes the quiet differentiator in GEO.
LLMs don’t rank URLs. They surface entities. A brand like Grab isn’t cited in AI answers because it has high PageRank; it’s cited because its entity — name, category, geography, use cases, associations — is densely represented across authoritative sources. Wikipedia entries, Wikidata IDs, Google Knowledge Panel confirmations, structured schema on owned properties, and consistent NAP (name, address, phone) signals across directories all feed that entity graph.
For regional brands targeting multilingual audiences across Thailand, Vietnam, Indonesia, or the Philippines, this creates a concrete to-do list: verify your Google Business Profile in every operating market, implement Organization and LocalBusiness schema with hreflang-consistent translations, and actively pursue mentions in local-language publications that AI crawlers index and weight. A Bahasa Indonesia article on Kompas citing your brand builds entity authority in ways that an English-language press release does not.
Local SEO and AEO Are Converging — Faster Than Expected
Semrush’s updated local keyword research guide frames the challenge clearly: local SEO in 2026 isn’t just about ranking in Google Maps. It’s about appearing in AI-generated answers when someone asks a voice assistant or an LLM chatbot which restaurant, clinic, or contractor to use near them.
The tactical implication is that local keyword research now needs to account for conversational query patterns — not just “digital marketing agency Bangkok” but “who is the best digital marketing agency for e-commerce brands in Bangkok.” These are the phrasings showing up in AI Overviews and ChatGPT responses, and they require content that directly answers the implicit question rather than simply repeating the keyword.
For Southeast Asian brands operating across multiple cities, this means building location-specific content pages that are substantive enough to be cited — not thin doorway pages, but genuinely useful resources that answer the questions a local customer would actually ask. A logistics company operating across Ho Chi Minh City, Jakarta, and Manila needs three distinct content strategies, not one page with city names swapped out. AI systems are good at detecting the difference.
Building an AI Visibility Stack: Where to Start
The practical sequencing matters here. Jumping straight to prompt engineering or content reformatting before fixing crawl access and entity signals is like A/B testing ad creative before confirming your landing page loads on mobile. Start with infrastructure, then build signal richness.
A defensible AI visibility stack for a mid-market Southeast Asian brand looks roughly like this: First, run a CCBot and AI-crawler audit against your robots.txt and WAF configuration — this takes an afternoon and costs nothing. Second, confirm entity completeness across Google Knowledge Graph, Wikidata, and your primary market directories. Third, audit existing content for answer-completeness: does it directly answer the question implied by the target keyword, or does it dance around it? Finally, build a local content layer that treats each operating city as a distinct audience with distinct questions.
None of this is glamorous. But the brands that will dominate AI search results in 2027 are mostly doing unglamorous infrastructure work right now, while their competitors are still debating whether AI search is “real” yet.
Key Takeaways
- Audit your robots.txt and WAF bot rules for CCBot blocks before investing further in GEO content — invisible to the crawler means invisible to the model.
- Entity authority (structured, consistent brand signals across directories, schema, and local-language publications) is what determines AI citation, not backlink profiles.
- Local SEO and AEO have merged: location-specific content must answer conversational queries in full, not just target keyword variants, to appear in AI-generated local answers.
The deeper question worth sitting with: if AI search systems are increasingly trained on historical web data, how do you build visibility in training corpora that haven’t been collected yet — and what signals, planted now in the right places, will still be surfacing your brand in answers two model generations from now?
At grzzly, we spend a disproportionate amount of time in this exact gap — the space between traditional SEO infrastructure and the emerging logic of generative search. We work with growth teams across Southeast Asia to audit AI crawl access, build entity authority strategies for multilingual markets, and design content architectures that earn citations rather than just rankings. If you’re starting to ask whether your brand is actually visible to AI — not just Google — that’s exactly the right question to be asking. Let’s talk
Sources
Written by
Sneaky GrizzlyTracking the quiet revolution inside LLM-powered search — where brand mentions, structured semantics, and entity authority rewrite the rules of discoverability before most marketers notice.