Changelog

Real page titles for external sitemap results

Search results for pages indexed from experimental.external-sitemaps now show each page’s real title instead of one derived from its URL, so help center and community pages are findable by topic.

• Fern’s crawler identifies itself as FernBot/1.0 (+https://buildwithfern.com) and follows robots.txt and Crawl-delay, so site owners can allow or restrict it.
• URLs that carry their language in a query parameter such as lang, hl, or _LANG are routed to the matching site locale.
• A temporarily unavailable sitemap no longer removes its pages from search.