[BUG] SSH connections to MikroTik routers drop after exactly 120 seconds (Keepalive timeout) #509
Description
Title
SSH connections to MikroTik routers drop after exactly 120 seconds (Keepalive timeout)
Platform
Website - Firefox
Server Installation Method
Docker
Version
1.11.1
Troubleshooting
- I have examined logs and tried to find the issue
- I have reviewed opened and closed issues
- I have tried restarting the application
- I have read Known Issues (Known Issues (Read Before Posting) #296)
The Problem
When connecting to a MikroTik router (RouterOS) via Termix, the session reliably disconnects exactly 2 minutes after establishing the connection, throwing an Error: Keepalive timeout in the backend logs.
This happens because Termix hardcodes keepaliveInterval: 30000 and keepaliveCountMax: 3 in its SSH connection configurations (found in files like src/backend/database/routes/credentials.ts and src/backend/ssh/server-stats.ts).
Because MikroTik's lightweight SSH server ignores SSH-level keepalive pings (keepalive@openssh.com requests), it never sends a response. The underlying Node.js ssh2 client sends a ping every 30 seconds, gets no response 3 times in a row, and proactively terminates the connection itself at exactly the 120-second mark.
Standard SSH clients (like OpenSSH or PuTTY) do not suffer from this issue because their equivalent setting (ServerAliveInterval) is disabled (0) by default. Termix should ideally allow keepaliveInterval to be configurable per-host, or default to 0 and rely purely on the existing tcpKeepAlive network-level pings to maintain connections to non-standard SSH servers.
How to Reproduce
- Set up and host the Termix server.
- Create an SSH connection to a MikroTik router (or any other device with an SSH server that ignores SSH-level keepalives).
- Connect to the device via the Termix UI.
- Leave the terminal open and wait exactly 120 seconds (2 minutes).
- The terminal will disconnect and attempt to reconnect.
- Check the Termix backend logs, which will show:
Additional Context
Logs:
[2:34:34 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:34:34 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }
termix | [2:34:34 PM] [INFO] [🖥️] SSH connection closed [op:terminal_ssh_disconnected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:34:35 PM] [INFO] [🖥️] Terminal WebSocket disconnected [op:terminal_ws_disconnect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:34:35 PM] [INFO] [🖥️] Terminal WebSocket connection established [op:terminal_ws_connect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:34:35 PM] [INFO] [🖥️] Resolving SSH host configuration [op:terminal_host_resolve,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:34:35 PM] [INFO] [🖥️] Initiating SSH connection [op:terminal_ssh_connect_attempt,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:34:36 PM] [INFO] [🖥️] Host key verified successfully [op:host_key_verified,user:fSo_LAEYhE6QE9D7O-WGD,host:7]
termix | [2:34:36 PM] [SUCCESS] [🖥️] SSH connection established [op:terminal_ssh_connected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:34:36 PM] [INFO] [🖥️] Creating shell [op:ssh_shell_start,host:7]
termix | [2:34:36 PM] [SUCCESS] [🖥️] Terminal shell channel opened [op:terminal_shell_opened,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:36:36 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:36:36 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }
termix | [2:36:36 PM] [INFO] [🖥️] SSH connection closed [op:terminal_ssh_disconnected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:36:36 PM] [INFO] [🖥️] Terminal WebSocket disconnected [op:terminal_ws_disconnect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:36:36 PM] [INFO] [🖥️] Terminal WebSocket connection established [op:terminal_ws_connect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:36:36 PM] [INFO] [🖥️] Resolving SSH host configuration [op:terminal_host_resolve,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:36:36 PM] [INFO] [🖥️] Initiating SSH connection [op:terminal_ssh_connect_attempt,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:36:37 PM] [INFO] [🖥️] Host key verified successfully [op:host_key_verified,user:fSo_LAEYhE6QE9D7O-WGD,host:7]
termix | [2:36:37 PM] [SUCCESS] [🖥️] SSH connection established [op:terminal_ssh_connected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:36:37 PM] [INFO] [🖥️] Creating shell [op:ssh_shell_start,host:7]
termix | [2:36:37 PM] [SUCCESS] [🖥️] Terminal shell channel opened [op:terminal_shell_opened,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:37:05 PM] [WARN] [📊] Removing unhealthy connection during cleanup [op:cleanup_unhealthy]
termix | [2:38:37 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:38:37 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }
termix | [2:38:37 PM] [INFO] [🖥️] SSH connection closed [op:terminal_ssh_disconnected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:38:37 PM] [INFO] [🖥️] Terminal WebSocket disconnected [op:terminal_ws_disconnect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:38:37 PM] [INFO] [🖥️] Terminal WebSocket connection established [op:terminal_ws_connect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:38:37 PM] [INFO] [🖥️] Resolving SSH host configuration [op:terminal_host_resolve,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:38:37 PM] [INFO] [🖥️] Initiating SSH connection [op:terminal_ssh_connect_attempt,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:38:38 PM] [INFO] [🖥️] Host key verified successfully [op:host_key_verified,user:fSo_LAEYhE6QE9D7O-WGD,host:7]
termix | [2:38:38 PM] [SUCCESS] [🖥️] SSH connection established [op:terminal_ssh_connected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:38:38 PM] [INFO] [🖥️] Creating shell [op:ssh_shell_start,host:7]
termix | [2:38:38 PM] [SUCCESS] [🖥️] Terminal shell channel opened [op:terminal_shell_opened,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:40:39 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:40:39 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }
termix | [2:40:39 PM] [INFO] [🖥️] SSH connection closed [op:terminal_ssh_disconnected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:40:39 PM] [INFO] [🖥️] Terminal WebSocket disconnected [op:terminal_ws_disconnect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:40:39 PM] [INFO] [🖥️] Terminal WebSocket connection established [op:terminal_ws_connect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:40:39 PM] [INFO] [🖥️] Resolving SSH host configuration [op:terminal_host_resolve,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:40:39 PM] [INFO] [🖥️] Initiating SSH connection [op:terminal_ssh_connect_attempt,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:40:40 PM] [INFO] [🖥️] Host key verified successfully [op:host_key_verified,user:fSo_LAEYhE6QE9D7O-WGD,host:7]
termix | [2:40:40 PM] [SUCCESS] [🖥️] SSH connection established [op:terminal_ssh_connected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:40:40 PM] [INFO] [🖥️] Creating shell [op:ssh_shell_start,host:7]
termix | [2:40:40 PM] [SUCCESS] [🖥️] Terminal shell channel opened [op:terminal_shell_opened,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:42:40 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:42:40 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }
termix | [2:42:40 PM] [INFO] [🖥️] SSH connection closed [op:terminal_ssh_disconnected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:43:01 PM] [INFO] [🖥️] Terminal WebSocket disconnected [op:terminal_ws_disconnect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:43:01 PM] [INFO] [🖥️] Terminal WebSocket connection established [op:terminal_ws_connect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:43:01 PM] [INFO] [🖥️] Resolving SSH host configuration [op:terminal_host_resolve,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:43:01 PM] [INFO] [🖥️] Initiating SSH connection [op:terminal_ssh_connect_attempt,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:43:02 PM] [INFO] [🖥️] Host key verified successfully [op:host_key_verified,user:fSo_LAEYhE6QE9D7O-WGD,host:7]
termix | [2:43:02 PM] [SUCCESS] [🖥️] SSH connection established [op:terminal_ssh_connected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:43:02 PM] [INFO] [🖥️] Creating shell [op:ssh_shell_start,host:7]
termix | [2:43:02 PM] [SUCCESS] [🖥️] Terminal shell channel opened [op:terminal_shell_opened,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:45:02 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:45:02 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }
termix | [2:45:02 PM] [INFO] [🖥️] SSH connection closed [op:terminal_ssh_disconnected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:45:02 PM] [INFO] [🖥️] Terminal WebSocket disconnected [op:terminal_ws_disconnect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:45:02 PM] [INFO] [🖥️] Terminal WebSocket connection established [op:terminal_ws_connect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:45:02 PM] [INFO] [🖥️] Resolving SSH host configuration [op:terminal_host_resolve,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:45:02 PM] [INFO] [🖥️] Initiating SSH connection [op:terminal_ssh_connect_attempt,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:45:03 PM] [INFO] [🖥️] Host key verified successfully [op:host_key_verified,user:fSo_LAEYhE6QE9D7O-WGD,host:7]
termix | [2:45:03 PM] [SUCCESS] [🖥️] SSH connection established [op:terminal_ssh_connected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:45:03 PM] [INFO] [🖥️] Creating shell [op:ssh_shell_start,host:7]
termix | [2:45:03 PM] [SUCCESS] [🖥️] Terminal shell channel opened [op:terminal_shell_opened,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:47:03 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:47:03 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }
termix | [2:47:03 PM] [INFO] [🖥️] SSH connection closed [op:terminal_ssh_disconnected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:47:03 PM] [INFO] [🖥️] Terminal WebSocket disconnected [op:terminal_ws_disconnect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:47:03 PM] [INFO] [🖥️] Terminal WebSocket connection established [op:terminal_ws_connect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:47:03 PM] [INFO] [🖥️] Resolving SSH host configuration [op:terminal_host_resolve,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:47:03 PM] [INFO] [🖥️] Initiating SSH connection [op:terminal_ssh_connect_attempt,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:47:04 PM] [INFO] [🖥️] Host key verified successfully [op:host_key_verified,user:fSo_LAEYhE6QE9D7O-WGD,host:7]
termix | [2:47:04 PM] [SUCCESS] [🖥️] SSH connection established [op:terminal_ssh_connected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:47:04 PM] [INFO] [🖥️] Creating shell [op:ssh_shell_start,host:7]
termix | [2:47:04 PM] [SUCCESS] [🖥️] Terminal shell channel opened [op:terminal_shell_opened,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:49:04 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:49:04 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }
termix | [2:49:04 PM] [INFO] [🖥️] SSH connection closed [op:terminal_ssh_disconnected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:49:04 PM] [INFO] [🖥️] Terminal WebSocket disconnected [op:terminal_ws_disconnect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:49:04 PM] [INFO] [🖥️] Terminal WebSocket connection established [op:terminal_ws_connect,user:fSo_LAEYhE6QE9D7O-WGD,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:49:04 PM] [INFO] [🖥️] Resolving SSH host configuration [op:terminal_host_resolve,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:49:04 PM] [INFO] [🖥️] Initiating SSH connection [op:terminal_ssh_connect_attempt,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:49:06 PM] [INFO] [🖥️] Host key verified successfully [op:host_key_verified,user:fSo_LAEYhE6QE9D7O-WGD,host:7]
termix | [2:49:06 PM] [SUCCESS] [🖥️] SSH connection established [op:terminal_ssh_connected,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:49:06 PM] [INFO] [🖥️] Creating shell [op:ssh_shell_start,host:7]
termix | [2:49:06 PM] [SUCCESS] [🖥️] Terminal shell channel opened [op:terminal_shell_opened,user:fSo_LAEYhE6QE9D7O-WGD,host:7,session:dCKdbsnZqsnTkPu7_UFE1]
termix | [2:51:06 PM] [ERROR] [🖥️] SSH connection error [op:ssh_connect,host:7]
termix | Error: Keepalive timeout
termix | at Timeout.sendKA (/app/node_modules/ssh2/lib/client.js:720:23)
termix | at listOnTimeout (node:internal/timers:588:17)
termix | at process.processTimers (node:internal/timers:523:7) {
termix | level: 'client-timeout'
termix | }
termix | [2:51:06 PM] [ERROR] [🖥️] Proceeding with cleanup after error
termix | {
termix | operation: 'ssh_error_cleanup',
termix | hostId: 7,
termix | error: 'Keepalive timeout'
termix | }