AI agents: Markdown, Content Signals, api-catalog

AI agents (ChatGPT, Claude, Perplexity and others) increasingly visit shops on their own: they look for a product, check the price and stock. The "AI / Агенти" card of the system module "Пошукові роботи" (Crawlers) gives them machine-readable entry points: pages in Markdown, your content-use preferences in robots.txt, the full text of the site and an API catalog.

Available from eCMS9 core 5.0.5-b881 (Crawlers module 1.1.0). Free: the module is part of the core. Everything is on by default after the update.

Where to find it

Admin panel → module "Пошукові роботи" → card "AI / Агенти". The settings are per site. Press Save after a change.

Content Signals in robots.txt

Checkbox "Content Signals у robots.txt" adds one line right after User-agent: *:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes

For each signal choose так (yes), ні (no) or не заявляти (omit the signal):

Signal	Meaning
search	Search results with a link to your site
ai-input	Real-time AI answers (search used to answer a question, RAG)
ai-train	Training or fine-tuning of models

Default: yes for all three. To keep your texts out of model training, set ai-train to ні. A closed site (option "Закрити весь сайт від індексації") keeps a single Disallow: / group without signals.

Robots.txt also gets a comment # llms.txt: https://<site>/llms.txt.

Toggles

Toggle	What it does
Markdown для агентів	The same URL answers Markdown to Accept: text/markdown; also /<url>.md and /index.md for the home page. Browsers and search engines keep getting HTML.
Окремий кеш для Markdown (LiteSpeed)	Writes a small block at the top of the site's .htaccess so the LiteSpeed page cache does not answer agents with cached HTML.
Заголовок Link на головній	Home page Link header (RFC 8288): sitemap, llms.txt, Markdown of the home page, API catalog.
/llms-full.txt	llms.txt plus the full text of content pages (delivery, payment, warranty) and category descriptions.
/.well-known/api-catalog	API catalog per RFC 9727 with the OpenAPI description. Works only when the Headless API module is enabled; otherwise the address answers 404.

Markdown responses

• Product, category and content pages and the home page are available in Markdown. The home page lists catalog sections and a table of the newest products with price and stock at request time.
• Markdown responses carry X-Robots-Tag: noindex and a Link to the HTML page as rel="canonical", so they never appear in search results as duplicates.
• Every response that depends on the Accept header carries Vary: Accept (existing values such as Accept-Encoding are kept).

The .htaccess block

The block sits between # BEGIN eCMS9 agent-ready and # END eCMS9 agent-ready. The rest of the file is never touched. Before the first write the module saves a copy next to the file (.htaccess.bak-<date>). Turning off Markdown для агентів or Окремий кеш для Markdown removes the block and leaves the file byte-for-byte as before. If the file is missing or not writable, the card shows a warning; nothing is created.

The block can also be applied from the console:

php spark cms9:robots-sync

MCP server, WebMCP and OpenAI feed

Available from eCMS9 core 5.0.5-b882 (Crawlers module 1.2.0) and the Headless API 2.15.0 module. Works only when Headless API is installed and enabled.

Toggle	What it does
MCP-сервер	/api/v1/mcp, a Model Context Protocol endpoint. An agent (Claude, ChatGPT, Cursor and others) connects the shop as a tool and looks up products, prices and stock itself.
WebMCP на вітрині	The same tools for an agent running in the shopper's browser (navigator.modelContext). Ordinary browsers load nothing.
Фід товарів для OpenAI	/api/v1/feed/openai.jsonl, the catalog in the OpenAI product feed format for ChatGPT shopping.

MCP tools

Read-only: an agent cannot add to cart, place an order or change anything.

Tool	Returns
search_products	Product search: name, price, stock, link (up to 20 per call)
get_product	Product by id or URL slug: description, variants, images, delivery
product_availability	Current price and stock of the product's variants
list_categories	Catalog sections
store_policies	Delivery, payment, warranty and returns pages
store_contacts	Shop contacts
search_knowledge	Search across the shop information pages (FAQ, delivery, warranty)

The same rules as the rest of Headless API apply: if reading requires a token, MCP requires it too (Authorization: Bearer …), and the rate limit applies. In that case WebMCP turns itself off.

Connecting a client that supports remote MCP servers (Streamable HTTP):

{ "mcpServers": { "shop": { "type": "http", "url": "https://<site>/api/v1/mcp" } } }

OpenAI feed

JSON Lines, one product per line: item_id, title, description, url, brand (falls back to the shop name), seller_name, image_url, availability, price (79.99 UAH), sale_price for discounts. Products without a price, image or URL are left out. ChatGPT checkout (is_eligible_checkout) is off: the shopper goes to your site. Submit the feed URL to the OpenAI merchant program once it is available in your country.

The MCP server and the feed are listed in /.well-known/api-catalog and llms.txt automatically.

Social networks in Site info

Settings → Інформація про сайт → block «Соцмережі»: one field per network (Facebook, Instagram, Telegram, YouTube, TikTok, X, LinkedIn, Pinterest). Links are shared by all languages, themes show them without template edits, and the shop's JSON-LD gets them in sameAs, so search engines and agents connect the site with your social profiles.

Checking

curl -sI -H 'Accept: text/markdown' https://<site>/   # Content-Type: text/markdown
curl -sI https://<site>/                               # Link: ...; Vary: Accept
curl -s https://<site>/robots.txt | grep Content-Signal
curl -sI https://<site>/.well-known/api-catalog        # application/linkset+json

Cloudflare

If the site previously used Cloudflare for this (managed robots.txt with Content Signals, a Worker that converts pages to Markdown, Response Header rules that add Link), turn them off after the update: the site now serves the same itself, and Cloudflare would override the admin settings.

FAQ

Will this change my SEO? No. HTML, titles, structured data and caching for people and search engines are unchanged; Markdown is noindex with a canonical link to HTML.

Does it slow down pages? No. HTML pages are still served from the LiteSpeed cache. Markdown is generated on request and not cached.

Why is api-catalog 404? The Headless API module is not installed or disabled. A catalog pointing to a missing API would be worse than none.