Taskmaster

An UNIX-based HTTP server and client job control daemon.

CLI Commands

Commands are sent over Unix Domain Sockets (UDS, .sock files) in the following format:

We use NDJSON (Newline Delimited JSON) to communicate over the socket. Sockets are streams not messages, hence if the server receives two messages to quickly, he could merge them instead of splitting them correctly.

Thus, relying on the \n after the message payload makes the communication more reliable.

Request

• command: enum of the CLI commands
• target: string containing the arg name. If empty, targets all ("all" depends on the context of the command. stop stops all programs, when reload reloads all the config file, server, unix_http_socket and client included).
• name if the target is "program" (using --program), then this parameter lets you enter what program to reload

Target nginx:

{
  "command": "stop",        // The CLI command name
  "target": "program",       // Target a specific program
  "name": "nginx"           // Name of the program
}

Target all programs:

{
  "command": "stop",     // The CLI command name
}

Responses

• status: int, HTTP success / error code
• content: opt, JSON, success data
• error: opt, JSON, error data

{
  "status": 200,
  "content": {}
}

{
  "status": 400,
  "error": {}
}

Config file

Treat your config file as an env file. An example can be found at taskmaster.example.config.toml.

The default places where the config file will be sought after are:

• $CWD/taskmaster.conf.toml
• ../taskmaster.conf.toml

[unix_http_server] Section Values

• file
  • Required: No
  • Default: None
  • A path to a UNIX domain socket on which taskmaster will listen for HTTP requests. taskmasterctl uses HTTP to communicate with taskmasterd over this port. If relative path, it will start where the config file is located.
• chmod
  • Required: No
  • Default: 0o700
  • Change the UNIX permission mode bits of the UNIX domain socket to this value at startup.
• chown
  • Required: No
  • Default: Use the username and group of the user who strats supervisord
  • Change the user and group of the socket file to this value. May be a UNIX username (e.g. chrism) or a UNIX username and group separated by a colon (e.g. chrism:wheel). You can only have one group at a time.

[unix_http_server]
file = "/tmp/taskmaster.sock"
chmod = 0o700
chown = "tomgernez:everyone"

[taskmasterd] Section Values

• logfile
  • Required: No
  • Default: $CWD/taskmaster.log
  • Path to the activity log of the taskmasterd process. Relative path assume start at at where the taskmaster.conf.toml was found.
• pidfile
  • Required: No
  • Default: $CWD/taskmasterd.pid
  • Path to the pid file of the taskmasterd process. Relative path assume start at where the taskmaster.conf.toml was found.
• umask
  • Required: No
  • Default: 0o22
  • Applies to log files created by the server and used as a default for files creates by the programs
• nodaemon
  • Required: No
  • Default: false
  • false daemonizes the taskmasterd
• minfds
  • Required: No
  • Default: 1024
  • Minimum number of available file descriptors that must be available before taskmasterd starts successfully.
• minprocs:
  • Required: No
  • Default: 200
  • Minimum number of available process descriptors that must be available before taskmasterd starts sucessfully.
• user
  • Required: No
  • Default: do not switch user
  • Requires taskmasterd to be launched as root user. If taskmasterd can't switch to the specific user, the program will not be started. The user will be changed using setuid only. This does not start a login shell and does not change env variables like USER or HOME.
• directory
  • Required: No
  • Default: do not cd
  • Only available if daemonized. taskmasterd cd's in that directory before starting.
• environment: Written in [taskmasterd.env], as KEY=VALUE. See below:

[taskmasterd]
logfile = "/tmp/taskmaster.log"
pidfile = "/tmp/taskmaster.pid"
umask = 0o22
nodaemon = true
minfds = 512
minprocs = 100
user = "user"
directory = "/tmp"

[taskmasterctl] Section Values

• serverurl
  • Required: No
  • Default: unix:///tmp/taskmaster.sock
  • For UNIX domain sockets: unix:///absolute/path/to/file.sock
• prompt
  • Required: No
  • Default: taskmaster
• history
  • Required: No
  • Default: /tmp/taskmaster_history.txt
  • Path to the history file

[program:x] Section Values

• command
  • Required: Yes
  • Default: No default
  • Path to the commands to be executed. No arguments (see below). If relative path, $PATH will be searched for the executable.
• args:
  • Required: No
  • Default: None
  • Array of strings. Example: ["-la", "."]
• numprocs
  • Required: No
  • Default: 1
  • Taskmaster will start as many instances of this program as named ny numprocs.
• numprocs_start:
  • Required: No
  • Default: 0
  • An integer offset that is used to compute the number at which process_num starts.
• autostart:
  • Required: No
  • Default: true
  • If true, this program will start automatically
• startsecs
  • Required: No
  • Default: 1
  • The total number of seconds which the program needs to stay running after a startup to consider the start successful (moving the process from the STARTING state to the RUNNING state). Set to 0 to indicate that the program needn't stay running for any particular amount of time. Even if a process exits with an "expected" exit code (see exitcodes), the strat will still be considered a failure if the process exits quicker than startsecs
• startretries
  • Required: No
  • Default: 3
  • Number of tries before the program goes into FATAL state. After each failed start, process will be put in BACKOFF. Each retry will take increasingly more time.
• autorestart
  • Required: No
  • Default: unexpected
  • Can be false, true, unexpected. Unexpected makes it restart if the exitcode of the program isn't in exitcodes. It is a STRING, not a boolean.
• exitcodes
  • Required: No
  • Default: 0
  • An array of "expected" exit codes for this program used with autorestart
• stopsignal
  • Required: No
  • Default: TERM
  • Signal used to kill the program when a stop is requested. Can be specified with the signal's name, shortname or number. Ex: "SIGINT", "INT", "2" (2 as a STRING)
• user
  • Required: No
  • Default: do not switch user
  • Requires taskmasterd to be launched as root user. If taskmasterd can't switch to the specific user, the program will not be started. The user will be changed using setuid only. This does not start a login shell and does not change env variables like USER or HOME.
• stdout
  • Required: No
  • Default: /tmp/<program>.stdout
  • The path to the file where the stdout will be printed.
• stderr
  • Required: No
  • Default: /tmp/<program>.stderr
  • The path to the file where the stderr will be printed.
• directory:
  • Required: No
  • Default: No chdir (inherit taskmaster's)
  • A file path representing a directory to which taskmasterd should temporarily chdir before starting the child
• umask
  • Required: No
  • Default: No special mask (inherit taskmaster's)
  • An octal number representing the umask of the process.
• serverurl:
  • Required: No
  • Default: AUTO
  • URL passed in the env of the subprocess process as SUPERVISOR_SERVER_URL to allow the subprocess to easily communicate with the internal HTTP server. If provided, it should have the same syntax and structure as the [taskmaskterctl] section option of the same name. If this is set to AUTO, or is unset, taskmaster will automatically construct a server URL.
• environment: Written in [programs.program.env], as KEY=VALUE. See below.

[programs.ls]
cmd = "/bin/ls"
args = ["-ls", "."]
numprocs = 1
numprocs_start = 1000
autostart = true
startsecs = 5
startretries = 1
autorestart = "unexpected"
exitcodes = [0, 2]
stopsignal = "TERM"
user = "user"
stdout = "/tmp/ls.stdout"
stderr = "/tmp/ls.stderr"
umask = 0o022
serverurl = "unix:///tmp/taskmaster-example-ls.sock"

[programs.ls.env]
TEST = true
ANSWER = 42

Subprocesses

Subprocesses will inherit the env of the shell used to start the supervisord program. Several env vars will be set by taskmasterd itself in the child's env also, including TASKMASTER_ENABLED (a flag indicating the process is under taskmaster control), TASKMASTER_PROCESS_NAME (the config-file-specified process name for the process). These vars may be overriden in the [taskmasterd] section of the config file, or per program [programs.x.env].

See supervisord subprocess-environment for reference.

Process States

See supervisord process-states for reference.

Logs

For supervisord: Logging: YYYY-MM-DD HH:MM:SS,SSS TYPE LOG

Types: DEBG, INFO, WARN, ERRO

Lib Choices

Parsing with serde, toml

serde and toml is the best combinaison to parse TOML in Rust. Using the Deserialize macro (also configure it in your Cargo.toml), serde is able to automatically turn a TOML string into a Rust struct.

Jobs

• Job number n may be reffered to as %n.
• %%, %+, % (alone) refer to the shell's notion of the current job. It's either:
  • The last job that ended in the foreground
  • The last job that started in the background
• %- refers to the previous job (see jobs command, it uses + and -)
• A job can be reffered to using a prefix of the name used to start it, or using a substring that appears in its command line. For example, %ce refers to a stopped ce job. %?ce refers to any job containing the string ce in its command line. If it matches more than 1 job, Bash reports an error
• Simply naming a job bring it to the foreground: %1 is synonym for fg %1, and %1 & is synonym for bg %1
• The shell learns immediately whenever a job changes state. Normally, Bash waits until it is aout to print a prompt before reporting changes in a job's status, so as to not interrupt any other output. If -b is enabled, Bash reports immediately. Any trap on SIGCHLD is exectued for each child process that exits.
• Jobs are grouped by ID. All jobs started from a given shell will have the same group ID, the one of the shell, unless put in the background. In that case, they will have their own.

Job Control Builtins

• jobs: list all active jobs. With the -x option, allows to interfere with the Job group ID.
• kill: send signal to jobs or processes. Use kill -l to list all available signals
• disown: remove jobs from the actiev jobs table
• suspend: suspend the execution of this shell until it receives a SIGCONT signal.

Job Control Variables

• auto_resume: controls how the shell interacts with the user and job control (it is not a Bash native feature. zsh, csh, tcsh feature it however). It is used to resume a suspended (CTRL + Z, not background) job using its name, like fg that job.

Architecture

Supervisor: A Process Control System

A client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems. Similar programs: launchd, daemontools, runit.

• PID are unreliable, file polling are unreliable, whereas the main process + subprocesses structured allows for great supervision of the processes
• Allows for normal users to control jobs that would require root privileges
• Subprocesses don't daemonize

Supervisor Components

See supervisor components. Below is a short description, but the documentation give more details.

supervisord

Server piece of supervisor. Interacts with the config file. Treat the config file as a .env file: it must be secured as it can contain some secret data. On launch, it daemonizes itself.

See supervisord command-line options for all options.

supervisord may be sent signals, which causes it to perform certain actions. See signals for the list of signals.

supervisorctl

Provides a shell-like interface to the features provided by supervisord. CLI client talks to the server across a UNIX domain socket or an internet (TCP) socket. The server may ask the client for auth credentials before allowing him to perform commands. The client typically uses the same config file as the server, but any configuration file with a [supervisorctl] section in it will work.

• If invoked with arguments, it usually doesn't start the CLI but executes as a command. Example: supervisorctl stop all, and returns the exit code (0 for success, non-zero for an error).
• If invoked without arguments supervisorctl , it starts the CLI with the given parameters. See supervisorctl command-line options for all CLI options.

In CLI mode, supervisorctl can perform multiple actions, see supervisorctl actions

Web Server

If supervisord is started against an internet socket, provide a supervisorctl-like interface but on a web interface. Settings are configured under the [inet_http_server] section of the configuration file.

Supervisor Config File

See file format

• Contains default search paths
• Allow environment variables expansion

[unix_http_server] Section Settings

Configuration parameters for an HTTP server that listens on a UNIX domain socket. If the config file has no [unix_http_server] section, no UNIX domain socket HTTP server will be started.

Configurable values include:

• file: path to a UNIX domain socket, on which supervisor will listen for HTTP/XML-RPC requests. %(here)s can be used, which expands in the directory containing the supervisord config file.
• Other values: chmod, chown, username, password

See values unix_http_server section settings

Example:

[unix_http_server]
file = /tmp/supervisor.sock
chmod = 0777
chown= nobody:nogroup
username = user
password = 123

[inet_http_server] Section Settings

Configuration parameters for an HTTP server that listens on a TCP socket. If the config gile has no [inet_http_server] section, no inet HTTP server will be strated.

Configurable values include:

• port: a TCP host:port value on which supervisor will listen for HTTP/XML-RPC requests.
• Other values: username, password

See values inet_http_server section settings

Example:

[inet_http_server]
port = 127.0.0.1:9001
username = user
password = 123

[supervisord] Section Settings

Global setttings related to the supervisord.

Configurable values include:

• logfile, logfile_maxbytes, logfile_backups, loglevel, pidfile, umask, nodaemon, silent, minfds, minprocs, nocleanup, childlogdir, user, directory, strip_ansi, environment, identifier

See values supervisord section settings

Example:

[supervisord]
logfile = /tmp/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /tmp/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
user = chrism
identifier = supervisor
directory = /tmp
nocleanup = true
childlogdir = /tmp
strip_ansi = false
environment = KEY1="value1",KEY2="value2"

See values supervisord section settings

[supervisorctl] Section Settings

Configuration for the supervisorctl interactive shell program.

• serverurl: URL that should be used to access the supervisord server (e.g. http://localhost:9001, or UNIX domain sockets: unix:///absolute/path/to/file.sock)
• username, password, prompt, history_file see supervisorctl section settings

Example:

[supervisorctl]
serverurl = unix:///tmp/supervisor.sock
username = chris
password = 123
prompt = mysupervisor

[program:x] Section Settings

Settings for the programmes started and controlled by supervisord. The header is a composite, where program tells supervisord that it's a program, and :x tells what's the program to start (here, it's x). It's an error if not present.

See the list of configurable variables program section settings.

• command: path (absolute or relative, if relative then search $PATH). Programs can accept arguments: program foo bar.
• process_name: used to compose the supervisor process name.
• numprocs: the number of instances from that program to start.
• numprocs_start: integer offset used to compute the number at which process_num starts.
• priority: relative priority of the program in the start and shutdown ordering. Defaults to 999. The higher the prio, the higher it strats first and ends last.
• autostart: if true, this program autostart when supervisord is started. Default: true
• startsecs: Total number of seconds which the program needs to stay running after a startup to consider the start successful (mosing the process from the STARTING state to the RUNNING state). 0 for instant.
• startretries: number of serial failure attempts that supervisord will allow when attempting to start the program before giving up the process ito a FATAL state. After each retry, the process will be in BACKOFF state. Each retry attempt takes increasingly more time.
• autorestart: supervisord should restart the process if it exits when in the RUNNING state. Can be:
  • false: never restarted
  • unexpected: restarted if the exit code isn't in exitcodes
  • true: always restarted
• exitcodes: list of "expected" exit codes. Default: 0
• stopsignal: Signal used to kill the program when a stop is requested. Can be the signal's name or number: TERM, HUP, INT, QUIT, KILL, USR1 or USR2. Default: TERM
• stopwaitsecs: number of seconds to wait for the OS to return a SIGCHLD to supervisord after the program has been sent a stopsignal. If this number of seconds elapses before supervisord receives a SIGCHLD from the process, supervisord will attempt to kill it with a final SIGKILL. Deault: 10
• stopasgroup: If true, flag causes supervisor to send the stop signal to the whole process group and implies killasgroup is true. This is useful for programs, such Flask in debug mode, that do not propagate stop signals to their children, leaving them orphaned. Default: false
• killasgroup: If true, when resorting to send SIGKILL to the program to terminate it send it to its whole process group instead, taking care of its children as well, useful e.g with Python programs using multiprocessing. Default: false
• user: Instruct supervisord to use this UNIX user account as the account which runs the program. The user can only be switched if supervisord is run as the root user. If supervisord can't switch to the specified user, the program will NOT be started (user is changed using setuid, so env vars like USER and HOME stay the same).
• redirect_stderr: if true, causes the process' stderr output to be sent back to the supervisord on its stdout file descriptor (in UNIX shell terms, this is the equivalent of executing /the/program 2>&1). Default: false
• stdout_logfile, stdout_logfile_maxbytes, stdout_logfile_backups, stdout_capture_maxbytes, stdout_events_enabled, stdout_syslog, stderr_logfile, stderr_logfile_maxbytes, stderr_logfile_backups, stderr_capture_maxbytes, stderr_events_enabled, stderr_syslog
• environment: list of key / value pairs in the form KEY="val",KEY2="val2" that will be placed in te child process' env. Besides exceptions, the subprocess will inherit the env vars of the shell used to start supervisord.
• directory: a file path representing a directory to which supervisord should temporarily chdir before exec'ing the child.
• umask: Octal number, representing the umask fo the process. Default: inherit supervisor's
• serverurl: the URL passed in the env to the subprocess process as SUPERVISOR_SERVER_URL, to allow the subprocesss to easily communicate with the internal HTTP server. If provide, it should have the same syntax and structure as the [supervisorctl] section option of the same name. If AUTO (or unset), supervisor will automatically construct a serverl URL, giving prefernce to a server that listens on UNIX domain sockets over one that listens on an internet socket.

Example:

[program:cat]
command=/bin/cat
process_name=%(program_name)s
numprocs=1
directory=/tmp
umask=022
priority=999
autostart=true
autorestart=unexpected
startsecs=10
startretries=3
exitcodes=0
stopsignal=TERM
stopwaitsecs=10
stopasgroup=false
killasgroup=false
user=chrism
redirect_stderr=false
stdout_logfile=/a/path
stdout_logfile_maxbytes=1MB
stdout_logfile_backups=10
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
stderr_logfile=/a/path
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false
environment=A="1",B="2"
serverurl=AUTO

[include] Section Settings

If the config file contains an [include] section, it must contain a single key named files. The values in this key specify other configuration files to be included within the configuration.

This section is only processed only by supervisord, ignored by supervisorctl.

See details on values include section settings.

• files: A space-separated sequence of file globs. Each file glob may be absolute or relative. If relative, it's relative to the location of the config file which includes it. If [include] section is used, it is required.

Example:

[include]
files = /an/absolute/filename.conf /an/absolute/*.conf foo.conf config??.conf

[group:x] Section Settings

See group section settings

[fcgi-program:x] Section Settings

See fcgi-program section settings

[eventlistener:x] Section Settings

See eventlistener section settings

[rpcinterface:x] Section Settings

See rpcinterface section settings

Resources

• Bash Documentation (see Chap 7 for Job Control)
• SUpervisor: A Process Control System
• daemontools
• launchd
• runit