Persistent terminal in Linux¶
Page last updated on: 2019-10-22
Jump to Use tmux if you are in a rush.
Problem¶
When you work on a remote server via ssh, you work under a terminal window. The problem is, if you logout or are disconnected when the remote job is running, you lose the job. The terminal and job both quit in such events.
This is because the ssh terminal is a process created by the ssh daemon for this connection, and the commands started in this terminal window are child processes of the terminal process.
PID PPID
root 20283 2365 0 16:23 ? 00:00:00 sshd: jmao [priv]
jmao 20363 20283 0 16:24 ? 00:00:00 sshd: jmao@pts/2
jmao 20364 20363 0 16:24 pts/2 00:00:00 -bash
jmao 20415 20364 0 16:31 pts/2 00:00:00 grep --color=auto ssh
ps -ef | grep 2365
root 2365 1 0 Oct02 ? 00:00:08 /usr/sbin/sshd -D
In the above example, I run a command to demonstrate such a relationship:
PID is process ID and PPID is parent process ID.
SSH daemon with process (pid=2365) was created by the system (1) and it listens to any ssh connections. When I connect to the server, the ssh daemon created two ssh processed with privilege separation (pid=20283, 20363) to take care of my connections. The ssh process 20283 runs on my behalf to start a terminal (pid=20364), and the grep command issued in this terminal has pid=20415 and knows its parent process is id=20364.
If this ssh session is disconnected, or logged out, process 20283 and 20363 will die, and so do its child processes 20364 and 20415.
In order for terminal child process stay alive after disconnection, the child processes must:
- detach from its parent process;
- be linked to another process that is not a child process of ssh connection.
Solutions¶
Detach an existing job¶
This method detaches and attaches the child process.
Summary:¶
- In a terminal with running job, press CTRL-Z to stop job and regain the control of terminal.
- Type
bgto resume the stopped job in background. - Background job is still tied to current terminal process. Run
disown %JOBIDto detach the job. - Now you can exit terminal or disconnect, the job will keep running.
Explanation:¶
Suppose you run a job that takes some time to finish:
sleep 120
In another terminal, check this job's process id and its parent process id:
ps -ef | grep sleep
jmao 238727 91227 0 22:16 pts/3 00:00:00 sleep 120
ps -ef | grep 91227
jmao 91227 22442 0 Oct04 pts/3 00:00:00 bash
We find "sleep 120" is a child process of bash (terminal) with pid=22442.
Now go back to the terminal window where we start sleep 120, press CTRL-Z key combination.
(base) jmao@pc:~/projects/levich-computing$ sleep 120
^Z
[1]+ Stopped sleep 120
sleep 120 and gives terminal control back as shown below:
(base) jmao@pc:~/projects/levich-computing$ jobs
[1]+ Stopped sleep 120
Now, types bg to resume job in the background:
(base) jmao@pc:~/projects/levich-computing$ bg
[1]+ sleep 120 &
(base) jmao@pc:~/projects/levich-computing$ jobs
[1]+ Running sleep 120 &
(base) jmao@pc:~/projects/levich-computing$
As the result, the sleep 120 is running in backgroud. Although we have the terminal control, the job is still a child process of bash. This means killing or quiting bash process 22442 will stop sleep 120 process.
ps -ef | grep sleep
jmao 238727 91227 0 22:16 pts/3 00:00:00 sleep 120
Since the job has id [1], run
disown -h %1
After the job is disowned, the job doesn't receive interruption signal from the shell. This means you can close the window or disconnect ssh without causing the job to quit.
(base) jmao@pc:~/projects/levich-computing$ ps -ef | grep sleep
jmao 239877 21989 0 23:57 ? 00:00:00 sleep 120
jmao 239884 153505 0 23:57 pts/4 00:00:00 grep --color=auto sleep
(base) jmao@pc:~/projects/levich-computing$ ps -ef | grep 21989
jmao 21989 1 0 Sep26 ? 00:00:00 /lib/systemd/systemd --user
The job becomes child of the system.
Run as a nohup background job¶
This method makes independent child process.
Summary:¶
A much simpler way comparing with the previous method is to submit a job with nohup and directly to the background. Essentially the following 4 steps
In a terminal with running job, press CTRL-Z to stop job and regain the control of terminal.Typebgto resume the stopped job in background.Rundisown %JOBIDto detach the job.Now you can exit terminal or disconnect, the job will keep running.
are now:
nohup command &
Example:¶
nohup sleep 120 &
nohupmeans the command is immune to hangup signal.&means the command will run in the background.
When a command is issued this way, the command will keep running even if the terminal window is closed or ssh is disconnected.
Use tmux¶
This method sends child process to 3rd process.
The above two methods have a drawback. That is the screen printout and command history are lost once the terminal/ssh window is closed.
tmux is a terminal multiplexer. A tmux session manages one or more sub-terminals and keeps them alive in detached mode even when ssh is disconnected:
Simple tmux usage¶
To start:
tmux
Run other commands in this tmux window as usual.
To detach tmux window:
- Use CTRL-b, then "d"
To re-attach tmux window back:
tmux a
To list tmux sessions
tmux ls
To attach a specific tmux session
tmux a -t ID
ID is the session number identified by tmux ls.
To close a tmux session:
- Use CTRL-D in an active tmux window.
Advanced tmux usage¶
The advanced usage involves splitting the pane. CTRL-b is key prefix, it is followed by another command key. We learned CTRL-b, d (CTRL-b followed by d) is to detach the pane. To split the pane:
- CTRL-b, % : split vertically
- CTRL-b, " : split horizontally
To navigate through panes:
- CTRL-b, arrow key:
tmux also supports multiple windows. The window IDs are numbers shown at the bottom of tmux session screen:
To create a new window:
- CTRL-b, c
To navigate through windows: * CTRL-b, p * CTRL-b, Window ID
Command summary:
| Goal | command or key | Explanation |
|---|---|---|
| Start tmux session | tmux |
Start a new tmux session |
| List tmux sessions | tmux ls |
List all tmux sessions |
| Re-attach a tmux session | tmux a |
Re-attach the last tmux session |
| Re-attach a tmux session | tmux a -t ID |
Re-attach a specific tmux session as listed in tmux ls |
| Detach tmux | CTRL-b, d | Detach current tmux session |
| Exit tmux | CTRL-d or exit |
Exit from a tmux pane or window |
| Split tmux pane | CTRL-b, % | Split current tmux pane vertically |
| Split tmux pane | CTRL-b, " | Split current tmux pane horizontally |
| Navigate through tmux panes | CTRL-b, ↑ | Move up to another pane |
| Navigate through tmux panes | CTRL-b, ↓ | Move down to another pane |
| Navigate through tmux panes | CTRL-b, → | Move right to another pane |
| Navigate through tmux panes | CTRL-b, ← | Move left to another pane |
| Create a tmux window | CTRL-b, c | Create a new tmux window |
| Navigate through tmux windows | CTRL-b, p | move to previous tmux window |
| Navigate through tmux windows | CTRL-b, n | move to nex tmux window |
| Navigate through tmux windows | CTRL-b, ID | move to a tmux window with that ID |