Your client portal is here, with your reports, updates and tasks under one roof. Sign in to Lumen
The Mithril blog
Practical guides for business owners and marketing teams working on search visibility, paid campaigns, websites and lead quality.
People ask aboutJavaScriptrobots.txtllms.txtmeasurement
Built into our thinking
Inside the guides, you’ll find two ways to go deeper: practical diagnostics you can run and common claims held up to the evidence.
Checks, experiments and diagnostic exercises. See what your site actually gives a machine, and what the results mean.
Compare the HTML response, rendered page and interactive content.
Look for the business facts you expect a machine to find.
Record missing information and the dependencies behind it.
Popular claims, checked against primary sources. What holds up, what doesn’t, and what needs context.
Featured myth
“Every website needs llms.txt”
What the evidence says
No universal requirement. A correctly maintained file is not the same as a meaningful benefit. Fix crawlability, conflicting facts and weak content first.
The useful question: which systems would use it, and what would it help them understand?
The AI search guide
18 connected articles. Four chapters. A practical path through how AI search finds, understands and uses your content.
A technical guide to AI crawler access, robots.txt, rendering, bot controls, search crawlers, training bots and agent traffic.
Learn how AI search systems retrieve web sources, use query fan-out and grounding, and why crawlable content still has to earn selection and citation.
Learn how clear entities, consistent facts, visible evidence and structured relationships help search and AI systems understand businesses, people, services and locations.
A practical framework for measuring AI search visibility using Search Console, Bing citations, grounding queries, crawler data, referrals and conversions.
Read this chapterPut it all together
Work through the checks that connect access, content, evidence and measurement.
Start with your question
Start with the decision you need to make. Find the relevant checks and the guides behind them.
Start with the information a crawler actually receives. Compare access rules, the server response and what only appears in a browser.
Separate search discovery, model training and agents acting for a person. Then decide which uses support your business and which controls you need.
Connect visibility with the decisions you need to make. Keep citations, visits, qualified inquiries and reporting limits distinct.
Focus on useful answers and evidence your business can support. Build connected pages around real customer questions.
The reading room
Browse every article, or narrow the list by topic, question or the kind of thinking you’re looking for.
25 articles to explore
Search + AI · 12 min
A technical guide to how websites become visible in AI search, from crawling and retrieval to entities, evidence, citations and measurement.
Search + AI · 11 min
A technical guide to AI crawler access, robots.txt, rendering, bot controls, search crawlers, training bots and agent traffic.
Search + AI · 12 min
How JavaScript rendering, GTM injection, lazy loading, widgets and interaction-dependent content can affect machine access and AI search visibility.
Search + AI · 8 min
Learn how robots.txt applies to AI search crawlers, training bots and agents, including OAI-SearchBot, GPTBot, Google-Extended and Cloudflare controls.
Search + AI · 7 min
A practical framework for deciding whether to allow AI model training, and why search access and training permission should be separate decisions.
Search + AI · 7 min
Learn how to identify AI crawler activity using server logs, Cloudflare, verified bots, user agents and referral data without confusing crawling with traffic.
Search + AI · 7 min
A practical guide to llms.txt, Markdown alternatives, agent discovery, Google Search limitations and when the file is actually worth maintaining.
Search + AI · 9 min
Learn how AI search systems retrieve web sources, use query fan-out and grounding, and why crawlable content still has to earn selection and citation.
Search + AI · 7 min
Learn what query fan-out means in generative search, why one-page-per-prompt strategies fail, and how to build a stronger topic architecture instead.
Search + AI · 6 min
Learn how internal links help discovery, topic relationships, canonical clarity and AI retrieval without turning your site into a pile of isolated blog posts.
Search + AI · 6 min
Learn how duplicate URLs, stale facts, canonical conflicts and weak update signals can make AI retrieval less reliable.
Search + AI · 7 min
Learn how clear entities, consistent facts, visible evidence and structured relationships help search and AI systems understand businesses, people, services and locations.
Search + AI · 5 min
Learn where schema markup helps machine understanding, where AI-search claims go too far, and how to implement structured data without inventing facts.
Search + AI · 6 min
Learn why first-party data, expert observations, examples and original evidence create stronger source material than generic AI-assisted summaries.
Search + AI · 5 min
Learn how locations, business profiles, NAP consistency, people, services and reviews create the entity infrastructure AI systems use for local answers.
Search + AI · 7 min
A practical framework for measuring AI search visibility using Search Console, Bing citations, grounding queries, crawler data, referrals and conversions.
Search + AI · 7 min
A practical technical AI search audit covering crawler access, rendering, robots, canonicals, entities, schema, retrieval readiness, evidence and measurement.
Search + AI · 9 min
A September 2026 guide to Cloudflare crawler controls, search and training preferences, robots.txt, llms.txt and practical policy checks.
Measurement · 2 min
A useful report connects activity with the decisions a business needs to make. Attribution adds context to that picture, but it does not replace lead quality, sales outcomes or the limits of the underlying data.
Planning · 2 min
A recurring plan can bring related marketing work into one engagement. The useful comparison is the scope, responsibilities and decisions it supports, not the number of services in the headline.
Search · 2 min
AI-assisted discovery changes how people encounter information. Keep the search strategy focused on useful answers, accessible pages and evidence, while choosing carefully where AI tools help your own workflow.
Paid & organic · 2 min
SEO and paid campaigns can share useful evidence without having identical jobs or budgets. Start with a common audience, clear conversion definitions and a landing experience that matches the message.
Content planning · 2 min
Choose a publishing pace your team can support with useful answers, reliable sources and proper review. Build the schedule around distinct customer questions and the time each answer needs.
Social media · 2 min
A new platform is worth considering when it fits your audience and the work you can sustain. Make the decision around customer relevance, content requirements and a clear way to evaluate the effort.
Research · 2 min
Use public competitor information to ask better questions about your own offer. Keep observations, interpretations and recommendations separate so the research leads to defensible decisions.
