Do agents check headers for llms.txt and API catalogs

Read as Markdown

Do agents closely check headers to check if llms.txt, docs, openapi.yaml, and .well-known/api-catalog exist? Or do they continue to parse text from html tags?