# The prompt library, the LLM directory and the paginated blog archives are
# being retired from the index by page-level `noindex, follow` plus removal
# from the sitemaps. They are deliberately NOT disallowed here: a crawler that
# cannot fetch the page cannot see the noindex, so blocking these paths would
# freeze them in the index as blocked results and throw away the link equity
# the `follow` directive is there to pass back to the commercial pages.
# Once Search Console shows them dropped, disallowing them is safe and worth
# doing for crawl budget.
#
# /embed/ stays disallowed: those are iframe targets, they carry
# `noindex, nofollow`, they are already out of every sitemap, and there is no
# indexation to unwind.

User-agent: *
Allow: /
Disallow: /embed/

Sitemap: https://www.appaca.ai/sitemap.xml
Sitemap: https://www.appaca.ai/sitemap-commercial-index.xml
Sitemap: https://www.appaca.ai/sitemap-editorial-index.xml