Website optimization · · 4 min read
llms.txt offers reading guidance
llms.txt is a proposal for giving language models a website overview and useful content links. It is not a universal requirement for search engines or models and cannot guarantee access, training or citation.
Link to real, accessible, accurate public pages. Exclude private workspace URLs, credentials, unauthorized material and unverified endorsements.
Structured data must match visible content
JSON-LD describes entities such as organizations and articles in a machine-readable format. Names, URLs, authors and dates must reflect real page content. Do not invent ratings or prices.
Google's structured data policies require accurate representation of the page. Valid syntax does not guarantee a rich search result.
Crawl rules and sitemaps have different purposes
robots.txt gives access rules to crawlers that follow the protocol. It cannot replace authentication or data protection. sitemap.xml helps discovery; inclusion does not establish indexing.
Exclude login pages, administration and private APIs. Update modification dates when content actually changes.
Review a version, deploy and verify
Check that scanned pages cover the intended site and languages. Review the complete package, its sources and paths. After deployment, check response codes, UTF-8 encoding and public access.
Report deployment, crawler visits, search updates and AI citations separately. Evidence from one stage does not prove the others.