optimize count items on get buckets query by SylvainSenechal · Pull Request #369 · scality/s3utils

SylvainSenechal · 2026-01-26T16:50:49Z

bert-e · 2026-01-26T16:50:53Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

SylvainSenechal · 2026-01-26T16:55:53Z

CountItems/CountManager.js

                    method: 'CountManager::_setupQueue',
                    workInQueue: this.q.length(),
-                    bucketInfo,
+                    bucketName: bucketInfo._name,


first little fix : logging bucketInfo would try to log a massive object with quite a lot of useless stuff, and an error "2026-01-20T14:08:08.645643941Z stdout F "[unable to serialize, circular reference is too complex to analyze]"
I think it's reasonable to only log this

we may want to log a few other fields to fully identify the bucket?

Suggested change

bucketName: bucketInfo._name,

bucketInfo: { name: bucketInfo._name, creationDate: bucketInfo._ creationDate, uid: bucketInfo._uid },

@benzekrimaha what do you think?

In case we are not sure which information we want to log, use this method. It will surely provide more information than needed, but it avoids circular references.

I added some more info, and used getters

SylvainSenechal · 2026-01-26T16:57:23Z

utils/S3UtilsMongoClient.js

            });
            const usersBucketCreationDatesArray = await cursorUsersBucketCreationDates.toArray();
-            return usersBucketCreationDatesArray
-                .reduce((map, obj) => ({ ...map, [obj._id]: obj.value.creationDate }), {});


The real problem, found by Maël : It's not even that we are querying all the buckets when we only need one (the perf in our test are still ok when doing this), but it's the reduce with spread operator : making a copy of the map on each iterations. Changing this from reduce to a simple loop reduce script run time from a few minutes to 2 seconds -_-

btw, this function isn't used anymore, because as it turns out later, we don't need to query all buckets info, but only the info of a specific bucket and we have its name, so we can use another function for that.
I still kept this function, it's associated with tests, don't wanna touch too many things, anyways we are supposed to try to replace this script with SUR

utils/S3UtilsMongoClient.js

codecov · 2026-01-26T17:06:40Z

Codecov Report

❌ Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.72%. Comparing base (8ff7ef7) to head (db0e5b3).
⚠️ Report is 8 commits behind head on development/1.17.

Files with missing lines	Patch %	Lines
utils/S3UtilsMongoClient.js	78.57%	3 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                  @@
##           development/1.17     #369      +/-   ##
====================================================
- Coverage             43.76%   43.72%   -0.05%     
====================================================
  Files                    84       84              
  Lines                  5961     5953       -8     
  Branches               1256     1254       -2     
====================================================
- Hits                   2609     2603       -6     
+ Misses                 3306     3304       -2     
  Partials                 46       46

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

francoisferrand · 2026-01-27T11:11:03Z

utils/S3UtilsMongoClient.js

-            }, {});
+            const inflightsMap = {};
+            for (const inflight of inflights) {
+                if (!inflight?._id) {


is this a real case?
I don't think it can happen with mongodb, having an id is mandatory?

right im too defensive here

utils/S3UtilsMongoClient.js

francoisferrand · 2026-01-27T11:37:08Z

utils/S3UtilsMongoClient.js

+                    });
+                });
+            } catch (err) {
+                if (err && err.message === 'Bucket entry not found') {


getUsersBucketCreationDate is used in 2 places only : here and in collectucketMetricsAndUpdate, and it seems to me that in both case we ignore missing bucket info.
Also in that function, we explicitely detect that case and raise an exception.

It thus seems that it would greatly simplify code to directly return "undefined" creation date when the bucket is not found, instead of raising an error; and raise an exception only for an actual error.
This way we don't need extra try/catch and error filtering...

Yes I changed it to return undefined when nothing is found

CountItems/CountManager.js

maeldonn · 2026-01-27T15:44:27Z

CountItems/CountManager.js

                    method: 'CountManager::_setupQueue',
                    workInQueue: this.q.length(),
-                    bucketInfo,
+                    bucketName: bucketInfo._name,


In case we are not sure which information we want to log, use this method. It will surely provide more information than needed, but it avoids circular references.

ISSUE: S3UTILS-219

bert-e · 2026-01-29T09:05:01Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

utils/S3UtilsMongoClient.js

francoisferrand · 2026-02-02T09:56:02Z

utils/S3UtilsMongoClient.js

                    ownerId,
                });
-                return cb(new Error('Bucket entry not found'));
+                return cb(null, undefined);


why are we ignoring error now?

maybe need a comment explaining the behavior of this function on "not found" (in function's doc header?)

yeah I changed it following your recommandations, as it is easier to handle the "no data case" like this

ISSUE: S3UTILS-219

SylvainSenechal · 2026-02-03T09:10:19Z

/create_integration_branches

bert-e · 2026-02-03T09:10:28Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/S3UTILS-219 into
development/1.17
w/1/improvement/S3UTILS-219 will be merged into development/1

The following branches will NOT be impacted:

development/1.13
development/1.14
development/1.15
development/1.16
development/1.4
development/1.7

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2026-02-03T09:10:29Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

SylvainSenechal · 2026-02-03T09:26:58Z

/approve

bert-e · 2026-02-03T09:35:40Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/1.17
✔️ development/1

The following branches will NOT be impacted:

development/1.13
development/1.14
development/1.15
development/1.16
development/1.4
development/1.7

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

SylvainSenechal · 2026-02-03T09:52:15Z

/approve

bert-e · 2026-02-03T09:56:20Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/1.17
✔️ development/1

The following branches have NOT changed:

development/1.13
development/1.14
development/1.15
development/1.16
development/1.4
development/1.7

Please check the status of the associated issue S3UTILS-219.

Goodbye sylvainsenechal.

SylvainSenechal force-pushed the improvement/S3UTILS-219 branch from fc4e5e1 to 1c42fd5 Compare January 26, 2026 16:54

SylvainSenechal commented Jan 26, 2026

View reviewed changes

utils/S3UtilsMongoClient.js Show resolved Hide resolved

SylvainSenechal commented Jan 26, 2026

View reviewed changes

utils/S3UtilsMongoClient.js Show resolved Hide resolved

SylvainSenechal requested review from a team, DarkIsDude, benzekrimaha, francoisferrand and maeldonn and removed request for DarkIsDude January 26, 2026 17:07

francoisferrand reviewed Jan 27, 2026

View reviewed changes

utils/S3UtilsMongoClient.js Outdated Show resolved Hide resolved

francoisferrand reviewed Jan 27, 2026

View reviewed changes

utils/S3UtilsMongoClient.js Outdated Show resolved Hide resolved

francoisferrand reviewed Jan 27, 2026

View reviewed changes

maeldonn reviewed Jan 27, 2026

View reviewed changes

use lighter logs to avoid unable to serialize errors

e013b0d

ISSUE: S3UTILS-219

SylvainSenechal force-pushed the improvement/S3UTILS-219 branch 2 times, most recently from 43e1398 to 9e6723d Compare January 28, 2026 12:04

scality deleted a comment from bert-e Jan 29, 2026

SylvainSenechal requested review from francoisferrand and maeldonn January 30, 2026 11:22

francoisferrand reviewed Feb 2, 2026

View reviewed changes

francoisferrand approved these changes Feb 2, 2026

View reviewed changes

optimize count items on bucket creation date query

41b6cd0

ISSUE: S3UTILS-219

bump version to 1.17.3

db0e5b3

ISSUE: S3UTILS-219

SylvainSenechal force-pushed the improvement/S3UTILS-219 branch from 9e6723d to db0e5b3 Compare February 3, 2026 08:52

maeldonn approved these changes Feb 3, 2026

View reviewed changes

bert-e merged commit 18648ac into development/1.17 Feb 3, 2026
14 of 15 checks passed

bert-e deleted the improvement/S3UTILS-219 branch February 3, 2026 09:56

	bucketName: bucketInfo._name,
	bucketInfo: { name: bucketInfo._name, creationDate: bucketInfo._ creationDate, uid: bucketInfo._uid },

Conversation

SylvainSenechal commented Jan 26, 2026

Uh oh!

bert-e commented Jan 26, 2026

Hello sylvainsenechal,

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bert-e commented Jan 29, 2026

Request integration branches

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SylvainSenechal commented Feb 3, 2026

Uh oh!

bert-e commented Feb 3, 2026

Integration data created

Uh oh!

bert-e commented Feb 3, 2026

Waiting for approval

Uh oh!

SylvainSenechal commented Feb 3, 2026

Uh oh!

bert-e commented Feb 3, 2026

In the queue

Uh oh!

SylvainSenechal commented Feb 3, 2026

Uh oh!

bert-e commented Feb 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

francoisferrand Jan 27, 2026 •

edited

Loading

SylvainSenechal Jan 26, 2026 •

edited

Loading

codecov bot commented Jan 26, 2026 •

edited

Loading