Follow these instructions:
- SSH into the instance
- Exit to terminal (Option 7)
- Run top
- Hit the following keys in sequence: `1 c s 1 enter`
- This will present you with a list of processors and processes with a ton of different statistics.
Example:
top - 10:27:21 up 22 days, 20:00, 1 user, load average: 0.68, 0.57, 0.52
Tasks: 131 total, 2 running, 129 sleeping, 0 stopped, 0 zombie
Cpu0 : 35.4%us, 0.0%sy, 0.0%ni, 64.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu1 : 2.0%us, 1.0%sy, 0.0%ni, 95.9%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 4.0%us, 0.0%sy, 0.0%ni, 96.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2050564k total, 1751612k used, 298952k free, 24264k buffers
Swap: 1044476k total, 8220k used, 1036256k free, 773284k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19208 unomalyd 20 0 617m 564m 3828 R 34 28.2 0:17.19 python /opt/uno..
19204 unomalyd 20 0 1070m 138m 9m S 6 6.9 2:36.64 /opt/unomaly/un..
19022 root 20 0 129m 9m 2800 S 1 0.5 0:07.83 /usr/sbin/syslo..
19207 unomaly 20 0 562m 9288 6144 S 1 0.5 0:07.52 /opt/unomaly/bi..
To identify potential issues with disk I/O - check the processor-list and look in the column with the values appended with "wa" - That is the total % of time since last update that the processor has been waiting for disk access. Any consistent values over 2-3% is a bad bad thing and will lead to decreased performance.
Possible solutions?
More RAM
Swapping can cause high IO-load, and will decrease performance of any Linux instance. Check for swapping by checking the row in the top output that says "Swap" - If there is a high amount of used Swap - look into increasing the instances memory if possible. It's also possible to check for swap I/O-usage by running `vmstat 1 10` and checking the output.
Example:
unomaly@unomaly:~$ vmstat 1 10
procs -----------memory---------- --swap-- ---io--- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 35764 107088 7672 767496 1 1 210 70 4 2 2 1 97 0
1 0 35764 105756 7680 768056 0 0 0 84 1738 3706 3 1 97 0
0 0 35808 117092 7680 755924 0 44 0 160 2070 4985 16 8 77 0
0 0 35808 117872 7680 755952 0 0 0 0 1691 3530 1 0 99 0
0 0 35808 117840 7680 755964 0 0 0 0 1569 3240 1 1 98 0
0 0 35808 117872 7704 755952 0 0 0 936 1796 4072 2 1 97 0
1 0 35808 117872 7704 755984 0 0 0 0 1456 3121 1 1 99 0
0 0 35808 117872 7704 755996 0 0 0 0 1580 3352 2 1 97 0
0 0 35808 117872 7704 756008 0 0 0 0 1414 2988 1 0 99 0
0 0 35808 117872 7704 756020 0 0 0 0 1463 3151 1 0 99 0
Note here that the swap usage is low. "si" means swap in and "so" means swap out. It's as easy as that.
Try finding possible culprits
You can also run `sudo iotop` to see if you can find anything specific eating using a lot of I/O. Most likely though, it's going to be mongod, the underlying database.