Skip to content

Faster data transfers from MARS.#235

Merged
alxmrs merged 10 commits intomainfrom
dl-faster-copy
Jan 4, 2023
Merged

Faster data transfers from MARS.#235
alxmrs merged 10 commits intomainfrom
dl-faster-copy

Conversation

@alxmrs
Copy link
Contributor

@alxmrs alxmrs commented Sep 19, 2022

I'm taking a leaf from @mahrsee1997's PR #148 so that we can copy data from the MARS server faster (using a larger buffer size).

Thanks for the primary contribution here, Rahul.

@alxmrs alxmrs marked this pull request as ready for review September 19, 2022 23:33
@alxmrs alxmrs marked this pull request as draft September 20, 2022 03:49
@mahrsee1997
Copy link
Collaborator

mahrsee1997 commented Sep 21, 2022

@alxmrs
After thinking a bit on this/discussing with the team -- what if ECMWF server outbound bandwidth is itself capped at some constant rate (too low), in that case this approach would be of no help.

Maybe we can wait until they are back online and do a POC to check their server outbound bandwidth. WDYT ?

@alxmrs
Copy link
Contributor Author

alxmrs commented Sep 21, 2022

That's sounds very reasonable to me. Good insight.

@alxmrs
Copy link
Contributor Author

alxmrs commented Dec 2, 2022

@mahrsee1997 @ksic8 Let's revisit this CL now that ECMWF is back online.

@alxmrs alxmrs marked this pull request as ready for review December 2, 2022 02:26
@alxmrs
Copy link
Contributor Author

alxmrs commented Jan 4, 2023

This LGTM -- I cannot approve this CL since I am also an author.

Copy link
Collaborator

@fredzyda fredzyda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can approve it if Alex can't.

@alxmrs
Copy link
Contributor Author

alxmrs commented Jan 4, 2023

Thanks Fred!

@alxmrs alxmrs merged commit 7cae996 into main Jan 4, 2023
@alxmrs alxmrs deleted the dl-faster-copy branch January 4, 2023 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants