Articles in this section

See more

Investigate issues with disk performance

Emma Lundström
  • Updated
Follow

Follow these instructions:

  1. SSH into the instance
  2. Exit to terminal (Option 7)
  3. Run top
  4. Hit the following keys in sequence: `1 c s 1 enter`
  5. This will present you with a list of processors and processes with a ton of different statistics.

Example:

top - 10:27:21 up 22 days, 20:00, 1 user, load average: 0.68, 0.57, 0.52
Tasks: 131 total, 2 running, 129 sleeping, 0 stopped, 0 zombie
Cpu0 : 35.4%us, 0.0%sy, 0.0%ni, 64.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 2.0%us,  1.0%sy, 0.0%ni, 95.9%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 4.0%us,  0.0%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us,  0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem:  2050564k total, 1751612k used,  298952k free, 24264k buffers
Swap: 1044476k total,    8220k used, 1036256k free, 773284k cached

  PID USER     PR NI  VIRT  RES  SHR S %CPU %MEM  TIME+  COMMAND
19208 unomalyd 20  0  617m 564m 3828 R   34 28.2 0:17.19 python /opt/uno..
19204 unomalyd 20  0 1070m 138m   9m S    6  6.9 2:36.64 /opt/unomaly/un..
19022 root     20  0  129m   9m 2800 S    1  0.5 0:07.83 /usr/sbin/syslo..
19207 unomaly  20  0  562m 9288 6144 S    1  0.5 0:07.52 /opt/unomaly/bi..

 

To identify potential issues with disk I/O - check the processor-list and look in the column with the values appended with "wa" - That is the total % of time since last update that the processor has been waiting for disk access. Any consistent values over 2-3% is a bad bad thing and will lead to decreased performance.

 

Possible solutions?

More RAM

Swapping can cause high IO-load, and will decrease performance of any Linux instance. Check for swapping by checking the row in the top output that says "Swap" - If there is a high amount of used Swap - look into increasing the instances memory if possible. It's also possible to check for swap I/O-usage by running `vmstat 1 10` and checking the output.

Example:

unomaly@unomaly:~$ vmstat 1 10
procs -----------memory---------- --swap-- ---io--- -system-- ----cpu----
 r  b   swpd   free  buff   cache  si  so   bi  bo   in   cs us sy id wa
 2  0  35764 107088  7672  767496  1    1  210  70    4    2  2  1 97  0
 1  0  35764 105756  7680  768056  0    0    0  84 1738 3706  3  1 97  0
 0  0  35808 117092  7680  755924  0   44    0 160 2070 4985 16  8 77  0
 0  0  35808 117872  7680  755952  0    0    0   0 1691 3530  1  0 99  0
 0  0  35808 117840  7680  755964  0    0    0   0 1569 3240  1  1 98  0
 0  0  35808 117872  7704  755952  0    0    0 936 1796 4072  2  1 97  0
 1  0  35808 117872  7704  755984  0    0    0   0 1456 3121  1  1 99  0
 0  0  35808 117872  7704  755996  0    0    0   0 1580 3352  2  1 97  0
 0  0  35808 117872  7704  756008  0    0    0   0 1414 2988  1  0 99  0
 0  0  35808 117872  7704  756020  0    0    0   0 1463 3151  1  0 99  0

 

Note here that the swap usage is low. "si" means swap in and "so" means swap out. It's as easy as that.

 

Try finding possible culprits

You can also run `sudo iotop` to see if you can find anything specific eating using a lot of I/O. Most likely though, it's going to be mongod, the underlying database.

Related articles