Add discard_spikes to curation, and update to v3
#4287
+210
−65
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Would close #4261
This PR allows you to remove spikes from a unit during curation by specifying the following in your curation json file:
This new feature means that the curation format gets bumped to v3!
You can discard at the same time as merging and splitting.
Tricky bit that took me ages to think through: how to apply it to an analyzer. Decided to discard spikes at the same time as splitting. During the splitting step, we re-wrangle discard spikes into another split unit (call them "discard units") and keep track of the discard unit id. Then remove the full discard units after the splitting. This allows us to use the existing splitting machinery (including splitting extension etc) for discards - nice!
We have to do merges after splitting+discarding. This is because the spike indices change after merging. To avoid wrangling spike indices (gross!) we just do discarding first.
To do: