Fix Shelley install failures on bulk VM upgrades

[josharian]

Author: realami
Channel: #feedback
Link: https://discord.com/channels/1405685085923049482/1405686161791516873/1477080213635727371

---

Issue

When upgrading Shelley on multiple VMs in parallel, the first attempt reliably fails on most or all VMs with an internal error. However, retrying 20-30 seconds later succeeds consistently.

Steps to Reproduce

ssh -t exe.dev ls --json|jq -r '.vms[].vm_name'|xargs -P0 -I{} sh -c 'printf "$(ssh exe.dev shelley install {})

"'

Error Output

Installing Shelley on REDACTED...
"shelley install REDACTED": internal error, trace ID: fe0c4c6235339cf5fa44d9336b3d2b21

Trace IDs

Most recent failing batch trace IDs:

• fe0c4c6235339cf5fa44d9336b3d2b21
• 40412bdd6ae4680047eeb730a3cd8d5a
• c5450a030e9643c271a491a8884282fc
• d55c4a0d16ab1434d52a61949b2f9de7
• 1023a765f0dd0934063c2d3496ea058d
• dfdec5d736df31570649f3fe87ba4f23

Expected Behavior

Parallel Shelley installations on multiple VMs should succeed on the first attempt.

Actual Behavior

Parallel installations fail initially but succeed when retried after 20-30 seconds, suggesting a race condition or rate limiting issue.