Use more internal iteration#689
Merged
frankmcsherry merged 3 commits intoTimelyDataflow:master-nextfrom Mar 20, 2026
Merged
Conversation
3e7d462 to
d0317c8
Compare
d0317c8 to
e65d33e
Compare
004336e to
fff3af9
Compare
Member
Author
|
I think we'll stop without doing the third step. It also seems worth fixing, but it's a moment where external iteration is acting as a m:n bridge between container types that are fundamentally different (containers of updates, and then columns of batch containers). I can imagine eventually wanting to unify these as chains of the same type, but tbd on just how easy/hard it should be when folks want to shift container types. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A lot of the complexity in DD is "external iteration", where abstractions need to reveal their internal types to support a generic algorithm. This is "anti abstraction" where detail spills outwards, and specific implementations are disempowered. This seems to happen in a few places, that we're going to try to clean up in order:
The first step seems pretty easy, and a reduction in code. It adds a new container trait that consolidates from one instances of the container to another. There is the opportunity to capture a common pattern and provide it, but no longer the requirement to do so. The
TStackchunker is not yet replaced, in that its behavior is more of a vec chunker that is followed by a conversion to a columnation container, and .. there's probably an easier way to express that.The second step is also pretty easy, but a bit more new code is written. Rather than externalize merging with container queues and such, there is a trait with two methods,
which allows an internal implementation of the merge batcher. It misses on some performance nits, like
extractnow extracts from one container into two to completion, rather than popping up for air as either of the destinations "fill". But it's also ~100 lines to put a different merger in place, which seems easier now.