Processes, Disks and Remote Access

When a server misbehaves, the first questions are usually: what is running, what is using the CPU or memory, and is the disk full? This note covers the commands that answer them, plus archiving files and working on a remote machine over SSH. For what a process is inside the kernel, see Processes vs Threads.

Listing processes with ps#

A process is a running instance of a program. Each one has a PID (process ID) and a PPID, the PID of the parent that started it. ps prints a snapshot of processes, and its flags decide which ones you see.

The aux flags look cryptic, so here's what they mean. In ps aux, a means all users, u adds user-oriented columns, and x includes processes with no terminal, such as daemons.

$ ps -ef | head -3
UID    PID  PPID  C STIME TTY      TIME     CMD
root     1     0  0 09:00 ?        00:00:02 /sbin/init
alice 4211  4210  0 09:12 pts/0    00:00:00 -bash

Common columns#

The output is a table, and column names vary a little between ps formats. These are the ones you read most.

VSZ is usually much larger than RSS, because most mapped memory is never loaded into RAM. See Virtual Memory for why.

Process states#

The STAT column holds a letter that tells you what the process is doing.

Live view with top#

ps is a snapshot. When you need to watch a problem as it happens, use top, which refreshes a sorted process list every few seconds. The header shows load average, CPU use and memory. The table shows PID, USER, %CPU, %MEM, TIME+ and COMMAND.

Many systems also have htop, a friendlier version with colors and mouse support.

Background jobs#

Not every process you care about is a server daemon. Often it's a command you started yourself, and the shell can run several of those from one terminal. A foreground job owns the terminal until it ends. A background job runs while you keep typing.

%1 is a job number from jobs, not a PID.

Signals and kill#

Sooner or later you'll need to stop a process, from top, from a job list or by PID. kill sends a signal to a process. Despite the name, many signals don't end the process.

Signal numbers above are for x86 and ARM Linux. Names are portable, so prefer them.

kill 1234             # SIGTERM: polite
kill -HUP 1234        # reload config, for daemons that support it
kill -9 1234          # SIGKILL: last resort
pkill -f "python worker.py"   # signal by command line

nohup: survive a closed terminal#

SIGHUP from the table above causes a common surprise. When you close a terminal or lose an SSH session, the shell sends SIGHUP to its jobs, and most exit. nohup makes a command ignore that signal.

nohup ./backup.sh > backup.log 2>&1 &

Without a redirect, nohup writes output to nohup.out. For anything long-lived, a systemd service or a terminal multiplexer such as tmux is a safer choice. See systemd.

Zombie processes#

You saw the Z state earlier. A zombie is a child process that has finished but still has an entry in the process table. The entry keeps the exit status until the parent reads it with wait(). A parent that never calls wait() leaves zombies behind.

Zombies use no CPU and almost no memory, but each one holds a PID. You can't kill one, because it's already dead. Fix or restart the parent. When a parent exits, PID 1 adopts its children and reaps any zombies.

# Find zombies and their parents
ps -eo pid,ppid,stat,cmd | awk '$3 ~ /^Z/'

ps aux | grep Z is a common shortcut, but it also matches any line that contains a capital Z.

shutdown#

Sometimes the fix for a stuck system is a reboot. shutdown powers off or reboots the machine and warns logged-in users first.

sudo shutdown -h now                      # power off now
sudo shutdown -r +5 "Rebooting for patch" # reboot in 5 minutes
sudo shutdown -c                          # cancel a scheduled shutdown

Disk space: du and df#

The last question from the opening is the disk. Two commands answer different parts of it. df asks "how full is each filesystem?" du asks "what is using the space?"

$ df -h /
Filesystem  Size  Used Avail Use% Mounted on
/dev/sda1    50G   21G   27G  44% /

A filesystem can run out of inodes (the fixed number of entries that store each file's metadata) before it runs out of bytes. This happens with millions of tiny files. Writes then fail with "No space left on device" while df -h still shows free space. Check df -i.

Archives: tar and zip#

Once you find what's filling a disk, you'll often want to bundle it up to move or back it up. tar bundles many files into one archive. It doesn't compress unless you add a compression flag.

tar -czvf site.tar.gz /srv/www       # create, gzip-compressed
tar -tzf site.tar.gz                 # list contents
tar -xzvf site.tar.gz -C /tmp/site   # extract into a directory

zip creates compressed .zip files, common when you share with Windows users. It needs -r to include a directory's contents.

zip -r project.zip project/
unzip -l project.zip     # list without extracting
unzip project.zip

Remote access: ssh and scp#

Everything above assumes you're already on the server. In practice you reach it from your laptop. ssh opens an encrypted shell on another machine. scp copies files over the same encrypted connection.

ssh deploy@203.0.113.10                  # log in as user deploy
ssh -i ~/.ssh/id_ed25519 deploy@203.0.113.10
ssh deploy@203.0.113.10 'df -h /'        # run one command and exit

scp report.txt deploy@203.0.113.10:~/    # local to remote
scp deploy@203.0.113.10:/var/log/app.log .   # remote to local
scp -r site/ deploy@203.0.113.10:/srv/www/   # a whole directory

Use SSH keys instead of passwords. Create a key pair with ssh-keygen -t ed25519 and copy the public key to the server with ssh-copy-id. Keep the private key at mode 600 (see Permissions, Ownership and sudo).

Reading the manual#

When you forget a flag for any command in this note, the answer is already on the machine. man <command> opens the manual page for a command: its options, behavior and often examples. Press / to search, n for the next match and q to quit. Pages are grouped into numbered sections, so man 5 crontab asks for the config-file page, not the command.

Key takeaways#

• ps aux or ps -ef for a snapshot, top for a live view. RSS is real RAM, VSZ is mapped memory.
• &, Ctrl+Z, jobs, bg and fg manage jobs in one shell.
• kill sends SIGTERM by default. Use SIGKILL (9) only as a last resort. SIGHUP often means "reload".
• Zombies are dead children whose parent has not called wait(). Fix the parent.
• df shows how full a filesystem is; du shows what fills it. Check df -i for inode exhaustion.
• tar -czf to archive, ssh and scp with keys for remote work.