Cheatsheet: Difference between revisions
→Troubleshooting
(→Misc) |
|||
Line 1,247:
Source: [https://scoutapm.com/blog/slow_server_flow_chart scoutapm.com]
I/O Wait represents the amount of time the CPU waiting for disk or network I/O.
3: IO wait is low and idle time is high▼
Anything above 10% I/O wait should be considered high.
4: IO Wait is high: check your swap usage▼
If your idle time is consistently above 25%, consider it "high enough"
6: Swap usage is low▼
7: Check memory usage▼
2: IO Wait is low and idle time is low: check CPU user time
Look for the %us column (first column), then look for a process or processes that is doing the damage.
If %usertime is high, see which program is monopolizing the CPU
Be default, top sorts the process list by %CPU, so you can just look at the top process or processes.
If situation seems anomalous: kill/restart the offending processes.
If situation seems typical given history: upgrade server or add more servers.
Your slowness isn't due to CPU or IO problems, so it's likely an application-specific issue.
Slowness might be caused by another server in your cluster or by an external services like DB
If you suspect another server in your cluster use - Strace and Lsof
Strace will show you which file descriptors are being read or written to.
Lsof can give you a mapping of those file descriptors to network connections.
Use top or free -m
Cache swaps will monopolize the disk
Processes with legitimate IO needs will be starved for disk access.
In other words, checking disk swap separates "real" IO wait problems from what are actually RAM problems that "look like" IO Wait problems.
5: Swap usage is high
High swap usage means that you are actually out of RAM.
Low swap means you have a "real" IO wait problem
iotop is an awesome tool for identifying io offenders.
Once top is running, press the M key - this will sort applications by the memory used.
Important: don't look at the "free" memory -- it's misleading.
To get the actual memory available, subtract the "cached" memory from the "used" memory.
This is because Linux caches things liberally, and often the memory can be freed up when it's needed.
A memory leak can be satisfactorily addressed by a one-time or periodic restart of the process.
If memory usage seems anomalous: kill the offending processes.
If memory usage seems business-as-usual: add RAM to the server, or split high-memory using services to other servers.
= Flows =
| |||