Skip to content

Transition to efficient background processing #3551

@r4victor

Description

@r4victor

The dstack server main bottleneck for supporting more active workloads is the implementation of background processing. Background workers hold DB transactions active while processing, which makes DB connections a scarce resource.

The background processing tasks need to be re-implemented so that long running operations are never performed while inside a DB transaction. The main idea is to commit locks to the DB. Here's a detailed PoC for this approach: https://github.com/r4victor/dbqueue_poc

On a related note, background worker execution times should be optimized since ultimately the processing throughput is proportional to the execution time. The longest core task is process_submitted_jobs, so it should be profiled and optimized first.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions