-
Notifications
You must be signed in to change notification settings - Fork 212
Open
Labels
Description
The dstack server main bottleneck for supporting more active workloads is the implementation of background processing. Background workers hold DB transactions active while processing, which makes DB connections a scarce resource.
The background processing tasks need to be re-implemented so that long running operations are never performed while inside a DB transaction. The main idea is to commit locks to the DB. Here's a detailed PoC for this approach: https://github.com/r4victor/dbqueue_poc
On a related note, background worker execution times should be optimized since ultimately the processing throughput is proportional to the execution time. The longest core task is process_submitted_jobs, so it should be profiled and optimized first.
Reactions are currently unavailable