CPU/RAM Slowness

The three numbers after uptime  i.e load average — 2.03 , 20.17 , and 15.09 —represent the 1-, 5-, and 15-minute load averages on the machine, respectively.

A system load average is equal to the average number of processes in a runnable or uninterruptible state. Runnable processes are either currently using the CPU or waiting to do so, and uninterruptible processes are waiting for I/O.

A single-CPU system with a load average of 1 means the single CPU is under constant load. If that single-CPU system has a load average of 4,there is four times the load on the system than it can handle, so three out of four processes are waiting for resources. The load average reported on a system is not tweaked based on the number of CPUs you have, so if you
have a two-CPU system with a load average of 1, one of your two CPUs is loaded at all times—that is, you are 50% loaded. So a load of 1 on a single-CPU system is the same as a load of 4 on a four-CPU system in terms of the amount of available resources used.

Load can be CPU-bound (processes waiting on CPU resources),RAM-bound (specifically, high RAM usage that has moved into swap), or I/O-bound (processes fighting for disk or network I/O).

For instance, if you run an application that generates a high number of simultaneous threads at different points, and all of those threads are launched at once, you might see your load spike to 20, 40, or higher as they all compete for system resources. As they complete, the load might come right back down.

Typically systems seem to be more responsive when under CPU-bound load than when under I/O-bound load. I’ve seen systems with loads in the hundreds that were CPU-bound, and I could still run diagnostic tools on those systems with pretty good response times. On the other hand, I’ve seen systems with relatively low I/O-bound loads on which just logging in took a minute because the disk I/O was completely saturated. A system that runs out of RAM resources often appears to have I/O-bound load,since once the system starts using swap storage on the disk, it can consume disk resources and cause a downward spiral as processes slow to a halt

 

Look for "wa" (I/O wait) and "id" (CPU idle time):

Checking I/O wait is the best initial step to narrow down the root cause of server slowness. If I/O wait is low, you can rule out disk access in your diagnosis.

I/O Wait represents the amount of time the CPU waiting for disk or network I/O. Waiting is the key here - if your CPU is waiting, it's not doing useful work. Anything above 10% I/O wait should be considered high.

On the other hand, CPU idle time is a metric you WANT to be high -- the higher this is, the more bandwidth your server has to handle whatever else you throw at it. If your idle time is consistently above 25%, consider it "high enough"

I/O WAIT IS LOW AND IDLE TIME IS LOW: CHECK CPU USER TIME:

Use top again -- look for the %us column (first column), then look for a process or processes that is doing the damage.

At this point you expect the usertime percentage to be high -- there's most likely a program or service you've configured on you server that's hogging CPU. Checking the % user time just confirms this. When you see that the % usertime is high, it's time to see what executable is monopolizing the CPU


Once you've confirmed that the % usertime is high, check the process list (also provided by top). Be default, top sorts the process list by %CPU, so you can just look at the top process or processes.
If there's a single process hogging the CPU in a way that seems abnormal, it's an anomalous situation that a service restart can fix. If there are are multiple processes taking up CPU resources, or it there's one process that takes lots of resources while otherwise functioning normally, than your setup may just be underpowered. You'll need to upgrade your server (add more cores), or split services out onto other boxes. In either case, you have a resolution:
if situation seems anomalous: kill the offending processes.if situation seems typical given history: upgrade server or add more servers. 
start by checking important applications for uncharacteristic slowness (the DB is a good place to start), think through which parts of your infrastructure could be slowed down externally. For example, do you use an externally hosted email service that could slow down critical parts of your application?
If you suspect another server in your cluster, strace and lsof can provide information on what the process is doing or waiting on. Strace will show you which file descriptors are being read or written to (or being attempted to be read from) and lsof can give you a mapping of those file descriptors to network connections.

I/O WAIT IS LOW AND IDLE TIME IS HIGH: 

Your slowness isn't due to CPU or IO problems, so it's likely an application-specific issue. It's also possible that the slowness is being caused by another server in your cluster, or by an external service you rely on.

I/O WAIT IS HIGH: CHECK YOUR SWAP USAGE:

Use top or free -m

if your box is swapping out to disk a lot, the cache swaps will monopolize the disk and processes with legitimate IO needs will be starved for disk access. In other words, checking disk swap separates "real" IO wait problems from what are actually RAM problems that "look like" IO Wait problems.

An alternative to top is free -m -- this is useful if you find top's frequent updates frustrating to use, and you don't have any console log of changes.

SWAP USAGE IS HIGH

High swap usage means that you are actually out of RAM. 

SWAP USAGE IS LOW

Low swap means you have a "real" IO wait problem. The next step is to see what's hogging your IO.

iotop is an awesome tool for identifying io offenders. Two things to note:

1.      unless you've already installed iotop, it's probably not already on your system. Recommendation: install it before you need it -- it's no fun trying to install a troubleshooting tool on an overloaded machine.

2.      iotop requies a Linux of 2.62 or above

CHECK MEMORY USAGE

Use top. Once top is running, press the M key - this will sort applications by the memory used.

Important: don't look at the "free" memory -- it's misleading. To get the actual memory available, subtract the "cached" memory from the "used" memory. This is because Linux caches things liberally, and often the memory can be freed up when it's needed. Read here (http://blog.scoutapp.com/articles/2010/10/06/determining-free-memory-on-linux) for more info.

Once you've identified the offenders, the resolution will again depend on whether their memory usage seems business-as-usual or not. For example, a memory leak can be satisfactorily addressed by a one-time or periodic restart of the process.

·        ·         if memory usage seems anomalous: kill the offending processes.

·         if memory usage seems business-as-usual: add RAM to the server, or split high-memory using services to other servers.

·         Shortcuts for top command :

l To display or to hide load average line
t To display or to hide task/cpu line
1 To display or hide all other CPUs
m to display or to hide RAM and SWAP details
s To change the time interval for updating top results(value is in secs)
R To sort by PID number
u  Press u then username to get only that user process details
P To sort by CPU utilization
M To sort by RAM utilization
c To display or hide command full path
r To renice a process, press r then the PID no then the renice value to renice a process.
k To kill a process, press k then PID number then enter to kill a process
w To save the modified configuration permanently.
q To quit the top command.
h for getting help on top command

Isolating High Load Average:

Vmstat

Next I will often run "vmstat 1", which prints out statistics every second on the system utilization. The first line is the average since the system was last booted:

denver-database:~ # vmstat 1

procs ---------memory---------- --swap-- --io--- -system-- -----cpu-----

 r  b swpd   free   buff  cache  si  so  bi  bo   in   cs  us sy id wa st

 0  0  116 158096 259308 3083748   0   0  47  39   30   58 11  8 76  5  0

 2  0  116 158220 259308 3083748   0   0   0   0 1706 4899 22 14 64  0  0

 1  0  116 158220 259308 3083748   0   0   0 276 1435 1490  4  2 93  0  0

 0  0  116 158220 259308 3083748   0   0   0   0 1502 1569  5  3 92  0  0

 0  0  116 158220 259308 3083748   0   0   0 892 1394 1529  2  1 97  0  0

 1  0  116 158592 259308 3083748   0   0   0 216 1702 1825  8  7 84  1  0

 0  0  116 158344 259308 3083748   0   0   0 368 1465 1461  8  7 84  0  0

 0  0  116 158344 259308 3083748   0   0   0 940 1992 2115  2  2 95  0  0

 0  0  116 158344 259308 3083748   0   0   0 240 1906 1982  6  7 87  0  0

The first thing I'll look at here is the "wa" column; the mount of CPU time spent waiting. If this is high you almost certainly have something hitting the disc hard.

If the "wa" is high, the next thing I'd look at is the "swap" columns "si" and "so". If these are much above 0 on a regular basis, it probably means you're out of memory and the system is swapping. Since RAM is around a million times faster than a hard drive (10ns instead of 10ms), swapping much can cause the system to really grind to a halt. Note however that some swapping, particularly swapping out, is normal.

Next I'd look at the "id" column under "cpu" for the amount of idle CPU time. If this is around 0, it means the CPU is heavily used. If it is, the "sy" and "us" columns tell us how much time is being used by the kernel and user-space processes.

If CPU "sy" time is high, this can often indicate that there are some large directories (say a user's "spam" mail directory) with hundreds of thousand or millions of entries, or other large directory trees. Another common cause of high "sy" CPU time is the system firewall: iptables. There are other causes of course but these seem to be the primary ones.

If CPU "us" is high, that's easy to track down with "top".

ps awwlx --sort=vsz

If there is swapping going on I like to look at the big processes via "ps awwlx --sort=vsz". This shows processes sorted by virtual sizes (which does include shared libraries, but also counts blocks swapped out to disc).

Iostat

For systems where there is a lot of I/O activity (shown via the "bi" and "bo" being high, but "si" and "so" being low), iostat can tell you more about what hard drives the activity is happening on, and what the utilization is. Normally I will run "iostat -x  2 5" will print a report every 2 seconds for 5 intervals.

iostat -x 2 5

 avg-cpu: %user %nice %system %iowait %steal %idle

  3.66 0.00 47.64 48.69 0.00 0.00

 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util

 sda 44.50 39.27 117.28 29.32 11220.94 13126.70 332.17 65.77 462.79 9.80 2274.71 7.60 111.41

 dm-0 0.00 0.00 83.25 9.95 10515.18 4295.29 317.84 57.01 648.54 16.73 5935.79 11.48 107.02

 dm-1 0.00 0.00 57.07 40.84 228.27 163.35 8.00 93.84 979.61 13.94 2329.08 10.93 107.02

I'll first look at the "%util" column, if it's approaching 100% then that device is being hit hard. In this case we only have one device, so I can't use this to isolate where the heavy activity might be happening, but if the database were on it's own partition that could help track it down.

"await" is a very useful column, it tells us how long the device takes to service a request. If this gets high, it probably indicates saturation.

Other information iostat gives can tell us if the activity is read-oriented or writes, and whether they are small or large writes (based on the sec/s sectors per second rate and the number of read/writes per second).

Iotop

This requires a very recent kernel (2.6.20 or newer), so this isn't something I tend to run very often: most of the systems I maintain are enterprise distros, so they have older kernels. RHEL/CentOS 3/4/5 are too old, Ubuntu Hardy doesn't have iotop, but Lucid does support it.

iotop is like top but it will show processes that are doing heavy I/O. However, often this may be a kernel process so you still may not be able to tell exactly what process is causing the I/O load. It's much better than what we had in the past though.

RAM :

To calculate CPU usage for a specific process you'll need the following:

1. /proc/uptime
   • #1 uptime of the system (seconds)
2. /proc/[PID]/stat
   • #14 utime - CPU time spent in user code, measured in clock ticks
   • #15 stime - CPU time spent in kernel code, measured in clock ticks
   • #16 cutime - Waited-for children's CPU time spent in user code (in clock ticks)
   • #17 cstime - Waited-for children's CPU time spent in kernel code (in clock ticks)
   • #22 starttime - Time when the process started, measured in clock ticks
3. Hertz (number of clock ticks per second) of your system.
   • In most cases, getconf CLK_TCK can be used to return the number of clock ticks.
   • The sysconf(_SC_CLK_TCK) C function call may also be used to return the hertz value.

Calculation

First we determine the total time spent for the process:

total_time = utime + stime

We also have to decide whether we want to include the time from children processes. If we do, then we add those values to total_time:

total_time = total_time + cutime + cstime

Next we get the total elapsed time in seconds since the process started:

seconds = uptime - (starttime / Hertz)

Finally we calculate the CPU usage percentage:

cpu_usage = 100 * ((total_time / Hertz) / seconds)

Script to find out which single process is consuming how much RAM:

[root@Thinktank ~]# ps aux | grep [c]hrome | awk '{sum=sum+$6}; END {print sum/1024 " MB"}'
1672.14 MB

Script to find out what percentage of Load Avg is there on the server :

[root@Thinktank ~]# cat loadavg 
#!/bin/bash
cores=$(nproc) 
load=$(awk '{print $1}'< /proc/loadavg)
echo | awk -v c="${cores}" -v l="${load}" '{print "relative load is " l*100/c "%"}'