Home > Linux Command Line, Openvz and Virtuozzo > Comprehensive Analysis of /proc/user_beancounters : Virtuozzo

Comprehensive Analysis of /proc/user_beancounters : Virtuozzo

While troubleshooting issues related to a Virtuozzo VPS, we usually come across the ‘user_beancounters’ file in the “/proc directory”. This file is of importance only if we use UBC or Combined SML+UBC memory mode for our Virtuozzo VPS. The resource control information about running virtual environments is contained in this file. So basically, ‘/proc/user_beancounters’ represents your VPS’s allocation of various system resources (memory). It thus is a main indicator of how well our VPS works, how stable it is, or whether there is a resource shortage. So, if you face any trouble while running or installing applications on your VPS, one good way to find the source of the problem is to take a look at this file.

Let’s dig deeper into the details of this file.

In Parallels Virtuozzo containers, virtualization technology resource control settings for each virtual machine are stored in the configuration file “/etc/vz/conf/XXX.conf” (where XXX is the ID of the given CT). These settings are loaded and applied to the containers during the VPS’s startup or on some events such as execution of “vzctl set CTID”. For running containers the resource control settings are exposed via “/proc/user_beancounters”. One such file “/proc/user_beancounters” exists in the node and one inside the VPS too. The file in the hardware node contains the resource control settings of all running VPSs. A pictorial representation of the file “/proc/user_beancounters” inside a VPS is shown below:

image1

A brief description of the various columns are given below,

UID: Indicates the ID of the container. In Virtuozzo each container is given a unique ID for the ease of management.

RESOURCE: This field indicates the primary,secondary and auxiliary parameters in Virtuozzo. In order to get more details of these resources refer url

HELD: Indicates the current usage of the various resources.

MAXHELD: Indicates the maximum usage of the resource since VPS startup.

BARRIER & LIMIT: Gives us the values of the softlimit and hardlimit of the virtozzo resource controls. Resource requests above that particular limit gets denied.

FAILCNT: It shows the number of refused or rather denied resource allocations of VPS right from the start up stage of the VPS. A non-zero value in this column indicates resource shortage and we need to either increase that particular resource or find the process responsible for it and optimize it. Otherwise it can cause weird issues with services running inside the container. Eg: unexpected service down, intermittent website issues,etc.

The following awk script can be used to list out all the containers with a non-zero values for the column “failcnt”. This script will print out all the containers with a non-zero failcnt value, along with their resource name and the corresponding failcnt value. Save the script as “/root/failcnt.awk” or any name that you like.

/root/failcnt.awk

BEGIN{OFS="";i=0;j=1;failcntflag=0;}
{
if(NF==7 && index($1,":") >0 ){
if(failcntflg==1){
printf "\nCTID=%s",arr1[1];
for(j=1;j<=i;j++){ printf " %s ",vector[j];delete vector[j];}
failcntflg=0;i=0;
}
split($1,arr1,":");

if($NF!=0) {
i = i+1;
failcntflg=1;
vector[i] = $2" "$NF;
}

}

 

if (NF==6 && $NF!=0){
i = i+1;
failcntflg=1;
vector[i] = $1" "$NF;
}
}
END{ printf "\n" }

Now run the script from node as follows,

[root@adminahead ~]# awk -f /root/failcnt.awk /proc/user_beancounters

CTID=10592 lockedpages 13
CTID=13917 kmemsize 357 shmpages 4 physpages 5 oomguarpages 1 tcprcvbuf 755
CTID=13904 kmemsize 528 numothersock 1
CTID=13905 kmemsize 73 numothersock 1
CTID=13897 kmemsize 1 shmpages 4 tcprcvbuf 4751
CTID=10000000 numothersock 1986
CTID=10594 kmemsize 27 physpages 7 oomguarpages 1 tcpsndbuf 295136
CTID=12435 shmpages 4
CTID=12437 kmemsize 2 shmpages 2 tcprcvbuf 690
CTID=12441 shmpages 3
CTID=12438 shmpages 1 physpages 712 oomguarpages 73 tcpsndbuf 63
CTID=10651 physpages 15 oomguarpages 8
CTID=10611 physpages 24 oomguarpages 11
CTID=10623 numothersock 14
CTID=10570 physpages 6 oomguarpages 3
CTID=10578 physpages 517 oomguarpages 33
CTID=10603 physpages 49 oomguarpages 40
CTID=10633 physpages 87 oomguarpages 24
CTID=10610 numproc 71 physpages 2250 oomguarpages 472
[root@ adminahead ~]#

As you can see from the above output, container “13917” shows the highest number of ‘failcnt’ for resources. For this VPS, “kmemsize”,”shmpages”,”physpages”,”oomguarpages” and “tcprcvbuf” show non-zero failcnt values and among them the first four resources are related to RAM. Upgrading the RAM inside that VPS is a good suggestion, but that should be considered only after finding out the resource intensive process inside the container and optimizing it.

You can use the following commands to list out the memory intensive processes inside the container.

* Lists top 3 memory intensive processes,

ps -auxf | sort -nr -k 4 | head -3

OR

wget -O /root/ps_mem.py http://www.pixelbeat.org/scripts/ps_mem.py
python /root/ps_mem.py |tail -3

The “/proc/user_beancounters” in the node can be monitored continuously to find out the VPSs that are short of resources and the corresponding VPS owner can be contacted for resource upgrade or optimization.

Advertisements
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: