htop explained

For the longest time I did not know what everything meant in htop. I thought that load average 1.0 on my two core machine means that the CPU usage is at 50%. That's not quite right. And also, why does it say 1.0? I decided to look everything up and document it here. They also say that the best way to learn something is to try to teach it. Table of Contents htop on Ubuntu Server 16.04 x64UptimeLoad averageProcessesProcess ID / PIDProcess treeProcess userProcess stateR - running or runnable (on run queue)S - interruptible sleep (waiting for an event to complete)D - uninterruptible sleep (usually IO)Z - defunct ("zombie") process, terminated but not reaped by its parentT - stopped by job control signalt - stopped by debugger during the tracingProcess timeProcess niceness and priorityMemory usage - VIRT/RES/SHR/MEMVIRT/VSZ - Virtual ImageRES/RSS - Resident sizeSHR - Shared Mem sizeMEM% - Memory usageProcessesBefore/sbin/init/lib/systemd/systemd-journald/sbin/lvmetad -f/lib/systemd/udevd/lib/systemd/timesyncd/usr/sbin/atd -f/usr/lib/snapd/snapd/usr/bin/dbus-daemon/lib/systemd/systemd-logind/usr/sbin/cron -f/usr/sbin/rsyslogd -n/usr/sbin/acpid/usr/bin/lxcfs /var/lib/lxcfs//usr/lib/accountservice/accounts-daemon/sbin/mdadm/usr/lib/policykit-1/polkitd --no-debug/usr/sbin/sshd -D/sbin/iscsid/sbin/agetty --noclear tty1 linuxsshd: root@pts/0 & -bash & htopAfterAppendixSource codeFile descriptors and redirectionColors in PuTTYShell in CTODOUpdatesFinal remarksT-shirt htop on Ubuntu Server 16.04 x64Here is a screenshot of htop that I am going to describe. UptimeUptime shows how long the system has been running.

You can see the same information by running uptime: $ uptime 12:17:58 up 111 days, 31 min, 1 user, load average: 0.00, 0.01, 0.05 How does the uptime program know that? It reads the information from the file /proc/uptime. 9592411.58 9566042.33 The first number is the total number of seconds the system has been up. The second number is how much of that time the machine has spent idle, in seconds The second value may be greater than the overall system uptime on systems with multiple cores since it is a sum. How did I know that? I looked at what files the uptime program opens when it is run. We can use the strace tool to do that. strace uptime There will be a lot of output. We can grep for the open system call. But that will not really work since strace outputs everything to the standard error (stderr) stream. We can redirect the stderr to the standard output (stdout) stream with 2>&1. Our output is this: $ strace uptime 2>&1 | grep open ... open("/proc/uptime", O_RDONLY) = 3 open("/var/run/utmp", O_RDONLY|O_CLOEXEC) = 4 open("/proc/loadavg", O_RDONLY) = 4 which contains the file /proc/uptime which I mentioned. It turns out that you can also use strace -e open uptime and not bother with grepping. So why do we need the uptime program if we can just read the contents of the file? The uptime output is nicely formatted for humans whereas the number of seconds is more useful for using in your own programs or scripts. Load averageIn addition to uptime, there were also three numbers that represent the load average. $ uptime 12:59:09 up 32 min, 1 user, load average: 0.00, 0.01, 0.03 They are taken from the /proc/loadavg file. If you take another look at the strace output, you'll see that this file was also opened.

$ cat /proc/loadavg 0.00 0.01 0.03 1/120 1500 The first three columns represent the average system load of the last 1, 5, and 15 minute periods. The fourth column shows the number of currently running processes and the total number of processes. The last column displays the last process ID used. Let's start with the last number. Whenever you launch a new process, it is assigned an ID number. Process IDs are usually increasing, unless they've been exausted and are being reused. The process ID of 1 belongs to /sbin/init which is started at boot time. Let's look at the /proc/loadavg contents again and then launch the sleep command in the background. When it's launched in the background, its process ID will be shown. $ cat /proc/loadavg 0.00 0.01 0.03 1/123 1566 $ sleep 10 & [1] 1567 So the 1/123 means that there is one process running or ready to run at this time and there are 123 processed in total. When you run htop and see just one running process, it means that it is the htop process itself. If you run sleep 30 and run htop again, you'll notice that there is still just 1 running process. That's because sleep is not running, it is sleeping or idling or in other words waiting for something to happen. A running process is a process that is currently running on the physical CPU or waiting its turn to run on the CPU. If you run cat /dev/urandom > /dev/null which repeatedly generates random bytes and writes them to a special file that is never read from, you will see that there are now two running process. $ cat /dev/urandom > /dev/null & [1] 1639 $ cat /proc/loadavg 1.00 0.69 0.35 2/124 1679 So there are now two running processes (random number generation and the cat that reads the contents of /proc/loadavg) and you'll also notice that the load averages have increased.

The load average represents the average system load over a period of time. The load number is calculated by counting the number of running (currently running or waiting to run) and uninterruptible processes (waiting for disk or network activity). So it's simply a number of processes. The load averages are then the average number of those processes during the last 1, 5 and 15 minutes, right? It turns out it's not as simple as that. The load average is the exponentially damped moving average of the load number. From Wikipedia: Mathematically speaking, all three values always average all the system load since the system started up. They all decay exponentially, but they decay at different speed. Hence, the 1-minute load average will add up 63% of the load from last minute, plus 37% of the load since start up excluding the last minute. Therefore, it's not technically accurate that the 1-minute load average only includes the last 60 seconds activity (since it still includes 37% activity from the past), but that includes mostly the last minute. Is that what you expected? Let's return to our random number generation. $ cat /proc/loadavg 1.00 0.69 0.35 2/124 1679 While technically not correct, this is how I simplify load averages to make it easier to reason about them. In this case, the random number generation process is CPU bound, so the load average over the last minute is 1.00 or on average 1 running process. Since there is only one CPU on my system, the CPU utilization is 100% since my CPU can run only one process at a time. If I had two cores, my CPU usage would be 50% since my computer can run two processes at the same time. The load average of a computer with 2 cores that has a 100% CPU utilization would be 2.00. You can see the number of your cores or CPUs in the top left corner of htop or by running nproc. Because the load number also includes processes in uninterruptible states which don't have much effect on CPU utilization, it's not quite correct to infer CPU usage from load averages like I just did. This also explains why you may see high load averages but not much load on the CPU.

But there are tools like mpstat that can show the instantaneous CPU utilization. $ sudo apt install sysstat -y $ mpstat 1 Linux 4.4.0-47-generic (hostname) 12/03/2016 _x86_64_ (1 CPU) 10:16:20 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 10:16:21 PM all 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:16:22 PM all 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:16:23 PM all 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 # ... # kill cat /dev/urandom # ... 10:17:00 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 10:17:01 PM all 1.00 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 97.00 10:17:02 PM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Why do we use load averages then?

$ curl -s https://raw.githubusercontent.com/torvalds/linux/v4.8/kernel/sched/loadavg.c | head -n 7 /* * kernel/sched/loadavg.c * * This file contains the magic bits required to compute the global loadavg * figure. Its a silly number but people think its important. We go through * great pains to make it work on big machines and tickless kernels. */ ProcessesIn the top right corner, htop shows the total number of processes and how many of them are running. But it says Tasks not processes. Why? Another name for a process is a task. The Linux kernel internally refers to processes as tasks. htop uses Tasks instead of Processes probably because it's shorter and saves some screen space. You can also see threads in htop. To toggle the visibility of threads, hit Shift+H on your keyboard. If you see Tasks: 23, 10 thr, it means it they are visible. You can also see kernel threads with Shift+K. When they are visible, it'll say Tasks: 23, 40 kthr. Process ID / PIDEvery time a new process is started it is assigned an identification number (ID) which is called process ID or PID for short. If you run a program in the background (&) from bash, you will see the job number in square brackets and the PID. $ sleep 1000 & [1] 12503 If you missed it, you can use the $! variable in bash that will expand to the last backgrounded process ID. $ echo $! 12503 Process ID is very useful. It can be used to see details about the process and to control it. procfs is a pseudo file system that lets userland programs to get information from the kernel by reading files. It is usually mounted at /proc/ and to you it looks like a regular directory that you can browse with ls and cd.

All information related to a process is located at /proc/<pid>/. $ ls /proc/12503 attr coredump_filter fdinfo maps ns personality smaps task auxv cpuset gid_map mem numa_maps projid_map stack uid_map cgroup cwd io mountinfo oom_adj root stat wchan clear_refs environ limits mounts oom_score schedstat statm cmdline exe loginuid mountstats oom_score_adj sessionid status comm fd map_files net pagemap setgroups syscall For example, /proc/<pid>/cmdline will give the command that was used to launch the process. $ cat /proc/12503/cmdline sleep1000$ Ugh, that's not right. It turns out that the command is separated by the \0 byte. $ od -c /proc/12503/cmdline 0000000 s l e e p \0 1 0 0 0 \0 0000013 which we can replace with a space or newline $ tr '\0' '\n' < /proc/12503/cmdline sleep 1000 $ strings /proc/12503/cmdline sleep 1000 The process directory for a process can contain links! For instance, cwd points to the current working directory and exe is the executed binary. $ ls -l /proc/12503/{cwd,exe} lrwxrwxrwx 1 ubuntu ubuntu 0 Jul 6 10:10 /proc/12503/cwd -> /home/ubuntu lrwxrwxrwx 1 ubuntu ubuntu 0 Jul 6 10:10 /proc/12503/exe -> /bin/sleep So this is how htop, top, ps and other diagnostic utilities get their information about the details of a process: they read it from /proc/<pid>/<file>. Process treeWhen you launch a new process, the process that launched the new process is called the parent process. The new process is now a child process for the parent process. These relationships form a tree structure. If you hit F5 in htop, you can see the process hierarchy.