Fix FootprintSet STDEV detection failure in runSourceDetection#186
Open
Fix FootprintSet STDEV detection failure in runSourceDetection#186
Conversation
Replace detectObjectsInExp with direct detection using Threshold.VALUE and np.std() to avoid STDEVCLIP returning 0 for some guider images. Add starMaskThreshold parameter to exclude bright pixels when computing background standard deviation.
| ) | ||
| absThreshold = threshold * imageStd | ||
| thresh = afwDetect.Threshold(absThreshold, afwDetect.Threshold.VALUE) | ||
| footprints = afwDetect.FootprintSet(exposure.getMaskedImage(), thresh, "DETECTED", 10) |
Contributor
There was a problem hiding this comment.
I don't love hardcoding the min number of pixels to 10, it might be worth making it possible to pass that in.
Member
Author
There was a problem hiding this comment.
Got it. I will make this change.
Member
Author
There was a problem hiding this comment.
Done, now nPixMin is an arg.
Add a uniform [0, 1) random dither per stamp before computing the median coadd. This breaks integer quantization that causes the pixel distribution to be so narrow that STDEVCLIP returns 0 on some guider images. Fix suggested by Robert Lupton. (DM-54263)
Replace the manual sigma68-only background estimation with STDEVCLIP from lsst.afw.math, falling back to sigma68 if STDEVCLIP returns 0. The dithered coadds (previous commit) prevent STDEVCLIP from failing, but the fallback provides a safety net. Also adds a configurable nPixMin parameter for footprint detection. (DM-54263)
estevesjh
force-pushed
the
tickets/DM-54263
branch
2 times, most recently
from
b646895 to
0a211d2
Compare
Address review feedback from Merlin: use explicit assertion instead of mypy ignore comment. (DM-54263)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
quantization, preventing STDEVCLIP from returning 0 (fix suggested by Robert Lupton).
nPixMinparameter for footprint detection (previously hardcoded to 10).Problem
STDEVCLIP(fromlsst.afw.math) returns 0 on some guider coadd images, causingFootprintSetto crash withInvalidParameterError: 'St. dev. must be > 0: 0'.Root cause: the guider coadd is a median stack of ~130 stamps. This averages the noise down
so much that the pixel distribution becomes extremely narrow — many pixels land in the same
quantized bin. When STDEVCLIP iterates, the first-pass sigma is near zero, the clipping
range collapses, and it converges to std=0.
Validation
Full validation on night 2026-02-22: 447 sequences (255 previously failing + 192 passing
control), excluding 24 with DatasetNotFoundError.
STDEVCLIP fix: The dither fix eliminates all STDEVCLIP=0 cases. Before: 437/2040 (21.4%)
guider measurements hit STDEVCLIP=0 on failing images, and 25/1536 (1.6%) on passing
images. After: 0 across all 3576 measurements. The sigma68 fallback was never triggered,
confirming the dither alone is sufficient.
Star recovery: On the 255 previously failing sequences, the new code recovers a median
of 7 stars per sequence (was 0). On the 192 control sequences, star counts are stable: 153
unchanged, 32 improved, 7 lose 1–2 stars due to GalSim
FindAdaptiveMomconvergencefailures (not related to this fix, tracked in DM-54305).