ADR: Python context compaction strategy

[eavanvalkenburg]

Summary

Proposes a design for context compaction in the Python Agent Framework, extracted from the open discussion in ADR-0016.

Problem

Long-running agents with many tool calls accumulate unbounded message lists. The current architecture cannot compact messages during the tool loop — stores are read once, and middleware only modifies copies.

Options Proposed

1. Standalone CompactionStrategy object — composed into HistoryProvider and FunctionInvocationConfiguration
2. CompactionStrategy as a mixin for HistoryProvider subclasses
3. Separate CompactionProvider as a ContextProvider subclass
4. Mutable message access in ChatMiddleware — refactor copy semantics

Key Design Points

• Four compaction points: post-load, pre-write, in-run (tool loop), existing storage
• Not applicable to service-managed storage
• Atomic group preservation (tool calls + results)
• Leverages source_id attribution from ADR-0016
• HistoryProvider.save_messages() needs an overwrite mode for storage compaction

References

• ADR-0016: Unifying Context Management with ContextPlugin
• Context Compaction Limitations Analysis

[markwallace-microsoft]

Will our OOB memory providers always read the full chat history by default?

[eavanvalkenburg]

right now, there is no filtering.

[markwallace-microsoft]

We must keep the system message in all cases

[markwallace-microsoft]

Is this the same as #1 when using N tokens

[markwallace-microsoft]

We also want to allow for custom compaction strategies.

[markwallace-microsoft]

In SK we didn't replace older messages instead

• Generate a summary of a range of memories
• Include more recent summary
• Insert the summary into the chat history in the appropriate place
• Return recent messages up to the most recent summary

We don't want to lose any data from chat history

[markwallace-microsoft]

Will this result in data loss?

[moonbox3]

nit: I believe this code has a bug. Instead:

compacted = await config["compaction_strategy"].compact(messages)
messages.clear()
messages.extend(compacted)
# or: messages[:] = compacted

The current code reassigns the local variable but doesn't mutate the list in-place. If the calling code holds a reference to the original list (which it does — prepped_messages in _tools.py), the reassignment has no effect.

[moonbox3]

save_messages is append-only, and _collect_messages only returns new messages from this run. So pre-write compaction only compacts the current turn's messages before appending, it never touches the existing history. Is that the intent?

If you want to compact the full stored history on write, you'd need to read-compact-replace, which
requires the replace_messages extension discussed separately, if I am not mistaken.

Can we clarify whether "pre-write" means:

• Compact only the new messages before appending (current code), or
• Compact the full history (existing + new) and replace

These are different behaviors.

[eavanvalkenburg]

yeah, it is only compacting the response messages, compacting what's already in the store is separate.

[moonbox3]

I feel like this trigger mechanism open question deserves more than a mention here. It feels like one of the most important design decision for in-run compaction. Running compact() after every single tool call feels wasteful, because most runs won't need compaction, and summarization strategies involve an LLM call.

What if we make the trigger part of the strategy interface:

class CompactionStrategy(ABC):
    def should_compact(self, messages: Sequence[Message]) -> bool:
        """Check if compaction is needed. Called after each tool result."""
        return True  # Default: always compact (subclasses override)

    @abstractmethod
    async def compact(self, messages: Sequence[Message]) -> list[Message]:
        ...

In this way, it keeps the trigger logic co-located with the strategy (a token-budget strategy knows its own budget) and avoids the LLM call overhead when compaction isn't needed. Thoughts?

[eavanvalkenburg]

makes sense

[moonbox3]

I get that [assistant with tool_calls] + [tool results] must be treated atomically, but then the doc leaves it to each strategy to implement correctly. Isn't this too error-prone? Every custom strategy would need to re-implement the same grouping logic.

What if we have a utility?

def group_atomic_messages(messages: Sequence[Message]) -> list[list[Message]]:
    """Group messages into atomic units that must be kept or removed together."""
    ...

Or even better, make compact() operate on grouped messages by default, with a lower-level compact_raw() for strategies that need full control. This would prevent the most common class of bugs in custom strategies, I think.

[eavanvalkenburg]

you are right, but that's a implementation detail

[moonbox3]

What happens if compact() raises during the tool loop? I don't see this called out in the doc. For robustness, the loop should probably catch exceptions from compaction and continue with uncompacted messages and log a warning, rather than failing the entire agent.run(). Do you agree? A truncation strategy should never fail, but a summarization strategy that calls an LLM absolutely can.

[eavanvalkenburg]

good one, will add

[moonbox3]

I'd keep CompactionStrategy. I think it's more descriptive as to what is actually happening.

[moonbox3]

I feel like this comes naturally based on Option 1's design. It should be built-in.

class ChainedStrategy(CompactionStrategy):
    def __init__(self, *strategies: CompactionStrategy):
        self.strategies = strategies

    async def compact(self, messages):
        for strategy in self.strategies:
            messages = await strategy.compact(messages)
        return messages

[moonbox3]

A thought in terms of observability: long-running agents with compaction will be hard to debug without visibility into when compaction happened and what was removed. Should we consider whether the strategy should return metadata (like CompactionResult with messages + removed_count + summary) or whether this should be handled via logging/events? Even a simple log line ("Compacted 150 messages to 50") could be helpful.

[eavanvalkenburg]

I need to do some work there, I think it might make sense to actually separate the messages that are sent to the model (with compaction) split from the messages that are returned to the user/session, good logging should also be part indeed