Proposal: Tool result content types beyond text

[hemanth]

Context

MCP supports multiple content types in tool results. WebMCP should clarify how non-text content is handled.

MCP Content Types

// MCP tool results can include:
{
  content: [
    { type: "text", text: "Search results..." },
    { type: "image", data: "base64...", mimeType: "image/png" },
    { type: "resource", uri: "file:///path/to/doc.pdf" }
  ]
}

Current Gap

The WebMCP proposal doesn't specify:

1. Which content types are supported
2. How images/binary data should be returned
3. How the browser/agent should render different types

Proposal

Define supported content types and their handling:

navigator.modelContext.provideContext({
  tools: [{
    name: "generate_chart",
    description: "Generate a chart from data",
    inputSchema: { ... },
    execute: async (args) => {
      const chartImage = await generateChart(args.data);
      
      return {
        content: [
          { type: "text", text: "Chart generated successfully" },
          { 
            type: "image", 
            data: chartImage.toDataURL(), // or base64
            mimeType: "image/png",
            alt: "Bar chart showing sales data"
          }
        ]
      };
    }
  }]
});

Proposed Content Types

Type	Description	Fields
text	Plain text	text
image	Image data	data (base64/dataURL), mimeType, alt
html	Rich HTML snippet	html (sanitized)
link	Clickable reference	url, title
table	Structured data	headers, rows
error	Error message	message, code

Security Considerations

• HTML content should be sanitized
• Images should have size limits
• External URLs should be validated
• No script execution in content

Benefits

1. Richer tool results for visual applications (design tools, charts, etc.)
2. Alignment with MCP content model
3. Better user experience in human-in-the-loop workflows

This directly supports the "Creative" use case in the proposal where tools like editDesign would benefit from returning visual previews.