Introduce Sitemap, LLMs.txt, Address LLM observability by ensuring https://rspec.info/robots.txt allows bot crawling#246
Conversation
JonRowe
left a comment
JonRowe
left a comment
There was a problem hiding this comment.
I'm open to adding a sitemap if its auto generated, we use middleman and theres probably a plugin for it. Can be a separate PR (or dropped entirely from this one).
I will think about relaxing therobots.txtgiven its not likely that effective and is certainly not exhaustive enough, however the changes requested to it need to be made.
Lastly I'm also not interested in adding an llms.txt given I don't want to use my limited time to maintain it.
| @@ -0,0 +1,184 @@ | |||
| # RSpec | |||
There was a problem hiding this comment.
I'm not interested in adding this, so please remove.
| @@ -0,0 +1,336 @@ | |||
| <?xml version="1.0" encoding="UTF-8"?> | |||
There was a problem hiding this comment.
I'm happy to add a sitemap, but it needs to be autogenerated.
| User-agent: GPTBot | ||
| Disallow: / | ||
| User-agent: * | ||
| Content-Signal: search=yes, ai-train=yes, ai-input=yes |
There was a problem hiding this comment.
I'm not willing to add this, this is the minimum I will consider:
| Content-Signal: search=yes, ai-train=yes, ai-input=yes | |
| Content-Signal: search=yes, ai-train=no, ai-input=no |
2 participants
Changelog
Fixes #245