perf: check blob container existence once, then optimistic retry#6912
Open
spboyer wants to merge 2 commits intoAzure:mainfrom
Open
perf: check blob container existence once, then optimistic retry#6912spboyer wants to merge 2 commits intoAzure:mainfrom
spboyer wants to merge 2 commits intoAzure:mainfrom
Conversation
Change ensureContainerExists from being called on every blob operation to a check-once-then-skip pattern with auto-recovery: - First operation: verify container exists (list + create if needed) - Subsequent operations: skip the check entirely - If any operation gets ContainerNotFound: reset flag, re-verify, retry the operation (handles external container deletion) Before: N container listing API calls for N blob operations After: 1 container listing call + 0 for subsequent operations Addresses finding 7 from Azure#6886 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
spboyer
requested review from
JeffreyCA,
hemarina,
rajeshkamal5050,
tg-msft,
vhvb1989 and
weikanglim
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Azure Blob Storage performance by avoiding repeated “list containers” existence checks on every blob operation, while adding an optimistic recovery path that re-verifies the container and retries when the container is deleted externally.
Changes:
- Added per-
blobClientcaching (containerVerified+sync.Mutex) to verify container existence once and skip subsequent checks. - Added
ContainerNotFounddetection and “reset + re-ensure + retry” logic acrossItems,Download,Upload, andDelete. - Refactored blob listing into a
listBlobs()helper to enable retry inItems().
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Upload: only retry with reader if it implements io.Seeker, avoiding silent data corruption from exhausted io.Reader - ensureContainerExists: stop paging once container is found (adds !exists to outer loop condition) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Azure Dev CLI Install InstructionsInstall scriptsMacOS/Linux
bash: pwsh: WindowsPowerShell install MSI install Standalone Binary
MSI
Documentationlearn.microsoft.com documentationtitle: Azure Developer CLI reference
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Check blob container existence only once per session, then skip on subsequent operations. If a
ContainerNotFounderror occurs (e.g., container deleted externally), automatically re-verify and retry.Closes: #6911
Parent: #6886 (Performance Review Audit -- Finding 7)
Changes
pkg/azsdk/storage/storage_blob_client.go:containerVerifiedflag andsync.MutextoblobClientstructensureContainerReady(): checks flag, only callsensureContainerExists()on first useresetAndEnsureContainer(): resets flag and re-verifies (used on container-not-found recovery)isContainerNotFound(): detectsContainerNotFounderror code viaazcore.ResponseErrorItems,Download,Upload,Delete) now use check-once + optimistic retrylistBlobs()helper for retry inItems()Before / After
Design
Testing
go buildclean