Skip to content

feat(llm): Serve markdown inline via content negotiation#16023

Merged
dcramer merged 7 commits intomasterfrom
feat/llm-inline-markdown-serving
Jan 16, 2026
Merged

feat(llm): Serve markdown inline via content negotiation#16023
dcramer merged 7 commits intomasterfrom
feat/llm-inline-markdown-serving

Conversation

@dcramer
Copy link
Member

@dcramer dcramer commented Jan 14, 2026

Serve markdown content inline when clients request Accept: text/markdown, instead of redirecting to .md URLs. This enables proper HTTP content negotiation for LLM/AI clients.

Changes

Middleware: Switch from redirect() to rewrite() so markdown is served at the same URL.

Sitemap generation: Generate hierarchical navigation for LLM discovery:

  • Root index.md: Sentry overview + links to top-level sections (platforms, api, product, etc.)
  • Section pages: List direct child pages under "Pages in this section"
  • Platform pages: Additionally list framework guides under "Guides" section
  • Guide pages: List their direct children (no separate Guides section)

Testing

# Request markdown at any URL
curl -H "Accept: text/markdown" https://docs.sentry.io/platforms/python/

# View root sitemap
curl -H "Accept: text/markdown" https://docs.sentry.io/

@vercel
Copy link

vercel bot commented Jan 14, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
develop-docs Ready Ready Preview, Comment Jan 16, 2026 7:27pm
sentry-docs Ready Ready Preview, Comment Jan 16, 2026 7:27pm

Request Review

@dcramer
Copy link
Member Author

dcramer commented Jan 14, 2026

sitemap isnt effective enough - 9000+ links

gonna take a slightly diff approach

@dcramer dcramer force-pushed the feat/llm-inline-markdown-serving branch 2 times, most recently from b22dfed to 347bff8 Compare January 15, 2026 00:59
Change LLM/AI client detection from redirect to rewrite, so markdown
content is served at the same URL when clients send Accept: text/markdown.
Previously clients were redirected to .md URLs.

Generate hierarchical sitemaps for LLM discovery:
- Root index.md includes Sentry overview and links to top-level sections
- Each section index page lists its child pages
- Platform pages list guides separately from other pages
- Guide pages list their sub-pages

The DOCS_ORIGIN is now environment-aware for develop.sentry.dev support.

Co-Authored-By: Claude <noreply@anthropic.com>
@dcramer dcramer force-pushed the feat/llm-inline-markdown-serving branch from 74b099d to 58168ce Compare January 15, 2026 01:01
Replaces the regex-based script tag removal with proper AST-based
removal using unist-util-remove. This fixes CodeQL security alerts for:
- js/incomplete-multi-character-sanitization
- js/bad-tag-filter

The AST-based approach is more robust and handles all edge cases that
the regex could miss.

Co-Authored-By: Claude <noreply@anthropic.com>
Replace existsSync check followed by read/write with try/catch
around readFile. This eliminates the time-of-check to time-of-use
race condition that CodeQL flagged as js/file-system-race.

Co-Authored-By: Claude <noreply@anthropic.com>
1. Fix cache key calculation to exclude scripts - restores original
   caching behavior while keeping AST-based script removal for the
   actual processing.

2. Fix Quick Links for developer docs - conditionally show different
   quick links based on NEXT_PUBLIC_DEVELOPER_DOCS environment variable.

3. Upload modified parent files to R2 - parent index files with
   appended child page listings are now uploaded to R2 CDN.

Co-Authored-By: Claude <noreply@anthropic.com>
Copy link
Contributor

@codyde codyde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks solid!

Add guidesIdx > 0 check to prevent invalid parent path calculation
when guides directory appears at path position 0.

Co-Authored-By: Claude <noreply@anthropic.com>
@dcramer dcramer merged commit 0c99a31 into master Jan 16, 2026
16 checks passed
@dcramer dcramer deleted the feat/llm-inline-markdown-serving branch January 16, 2026 19:28
Comment on lines +84 to +85
# Force markdown via query param
curl "https://docs.sentry.io/platforms/python/?format=md"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also add curl "https://docs.sentry.io/platforms/python.md, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this file dead at this point? im gonna go spend some tim eon this repo to review state of it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think i added the query param ages back, and then setup the automatic switch over to .md. It all still works now, but with the auto-negotiation, I dont think its going to be used much.

I think we use it on the copy page as markdown button though.

File should be updated with the new flows for accessing the MD pages.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file looks like it was an LLM artifact? i didnt look at it, but is this actually supposed to be in the repo? (its a markdown file in a scripts directory is why i ask)

dcramer added a commit that referenced this pull request Jan 20, 2026
)

Moves LLM documentation from `scripts/LLM_MARKDOWN_SERVING.md` to the
contributing guide at `docs/contributing/pages/llm-support.mdx`.

The old file was an implementation-focused artifact from #16023 that
lived in an odd location and was incomplete (missing `.md` extension
access method, per feedback from @BYK and @codyde).

The new page covers our broader approach to designing docs for LLMs:
- Principles (ties to existing minimalist/accuracy approach)
- How we support LLMs (content negotiation, copy button, navigation)
- Writing considerations for contributors

Refs #16023

Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions bot locked and limited conversation to collaborators Feb 5, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants