Home > Linux Command Line, Openvz and Virtuozzo > One-liners for troubleshooting Virtuozzo load issues

One-liners for troubleshooting Virtuozzo load issues

 

I wish to introduce the various one-liners that can be used to troubleshoot, load or performance issues, within a Virtuozzo node.

To begin with, we will discuss the various situations that can degrade performance of a Virtuozzo node. As is obvious, heavy usage of either/all of CPU, Memory, Disk or Network will degrade the performance of node. The same case applies with Virtuozzo nodes too, but with an additional complexity; it could also happen as a result of the processes running inside a container. In this case we need to identify the problem container and deal with the processes inside that container.

I have found these one-liners quite useful for finding out the problem container while troubleshooting Virtuozzo alerts.

 

1) Troubleshooting load issues caused by high CPU activity

 

=> Display the list of containers sorted based on cpu usage,

/usr/sbin/vzlist -o ctid,laverage

OR

/usr/sbin/vzstat -t -s cpu|awk ‘NF==10{print $0}’

=> Sometimes the above one-liners will not show the actual cpu usage inside the containers(possibly due to some delay in updating the stats), but still, the load inside node will be high. In this situation, running the command pasted below will help find out the cpu intensive containers.

for i in `/usr/sbin/vzlist -H -o ctid`; do echo "CTID: ${i} `/usr/sbin/vzctl exec ${i} cat /proc/loadavg`"; done

=> List out all containers for which the status is not in “OK” status. This is quite helpful while troubleshooting load issues when the load average in the node is super-high(above 1000)

/usr/sbin/vzstat -t|awk ‘{if(NF==10 && $2!=”OK” && $1!=”CTID”)print $0}’

=> Lists the top 10 containers based on number of processes running inside the container.

/usr/sbin/vzlist -H -o ctid,numproc|sort -r -n -k2|head

 

2) Troubleshooting load issues caused by n/w activity

 

=> Sorts containers based on socket usage

/usr/sbin/vzstat -t -s sock|awk 'NF==10{print $0}'

=> Sorts containers based on TCP sender buffer usage,

/usr/sbin/vzlist -H -o ctid,tcpsndbuf |sort -r -n -k2

=> Sorts containers based on TCP receive buffer usage,

/usr/sbin/vzlist -H -o ctid,tcprcvbuf |sort -r -n -k2

=> Sorts containers based on the highest inbound traffic(quite useful while troubleshooting n/w related attacks),

/usr/sbin/vznetstat -r |awk '$3 ~ /G/ {print $0}'|sort -r -nk3

=> Sorts containers based on the highest oubound traffic(quite useful while troubleshooting n/w related attacks) ,

/usr/sbin/vznetstat -r |awk '$5 ~ /G/ {print $0}'|sort -r -nk5

 

3) Troubleshooting performance issues caused by memory utilization

 

The ‘dmesg‘ command displays containers which has resource shortages. There is possibility that it could be the abusive one. So there is a need to check the process inside that container.

[root@virtuozzo ~]# dmesg|egrep -v '(SMTP-LOG|INPUT-DROP|LIMIT-PPS-DROP|FORWARD-DROP)'
TTL=64 ID=0 PROTO=UDP SPT=68 DPT=67 LEN=556
[1101732.300833] __ratelimit: 44 callbacks suppressed
[1101732.310531] Fatal resource shortage: kmemsize, UB 12215.
[1101742.294179] Fatal resource shortage: kmemsize, UB 12215.
[1101752.277368] Fatal resource shortage: kmemsize, UB 12215.
[1101752.393226] Fatal resource shortage: kmemsize, UB 12215.
[1105092.458621] __ratelimit: 101 callbacks suppressed
[1105092.468411] Fatal resource shortage: kmemsize, UB 12215.
[root@virtuozzo ~]#

 

4) Troubleshooting load issues caused by high disk I/O activity.

 

You can install the ‘atop’ command and spot problem processes at the top of the list when sorting by disk usage (‘D’). To get more information on using ‘atop’ refer url.

The above one-liners will help you identify the problem CTID or the process(PID) responsible for performance issue. In the second case after finding the process-id, you can use ‘vzpid’ command to spot the container inside which the process is running and either renice or stop that process. And in the first case, you can view the processes running inside the container using either ‘vzps’ or ‘vztop’ command. The usage of which is given below,

vztop -b -c -n 1 -E 

OR

vzps auxfww -E 

So, that is it guys. I sincerely hope you get to take away something helpful from all this.

Happy Hunting 😀

Advertisements
  1. admintriantech
    March 27, 2015 at 6:42 pm

    Thank you for this post ! Really helps.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: