Back to the Overview

Resource Control in Embedded Linux Systems with cgroups

– ELinOS, Industrial Automation, Linux

For some years now, so-called control groups (cgroups) have enjoyed great popularity among Linux administrators. But not only for them, also for developers of embedded systems cgroups are an interesting technique: With the help of cgroups processes can be structured and managed. You can assign rights and computing time or resources to them and get a better system overview. If used correctly, cgroups can make systems more performant overall. This is important for servers and desktops, but even more so for embedded devices, which are often only equipped with the bare minimum for cost reasons. Especially if the embedded Linux in use has a real-time extension, cgroups can show their strengths. Under high system load, their use can reduce jitter compared to systems in which no grouping is used (Read more). For example, cgroups can be used to address a fixed memory area that must not be used by other general purpose tasks. But not only memory can be reserved with their help, also individual processor cores, which are then only allocated to an application or a group of applications can be addressed. This optimizes the execution of soft real-time applications.


Virtualization with Docker and Resource Allocation

As virtualization (e.g., with Docker) becomes increasingly important in building systems, cgroups take on a new importance in allocating available resources. Here, resources can be used sparingly and wisely without the need for another guest OS with larger resource requirements, while gaining visibility through the reasonably sharp separation provided by Docker containers. A simple example shows the benefit: Let's assume that an embedded device is used as a door opener in a building. At the same time, however, it controls a surveillance camera that is connected to a server via TCP/IP stack and sends recordings at regular intervals showing what is happening in front of the building.

Now, one may argue whether the surveillance camera is important or not: In any case, one would like the sensor responsible for the door to work without any delay or to avoid that it might fail to work at all. In this case, it is possible to outsource the TCP/IP stack and the necessary functionality to a Docker container and allocate fewer resources to it than to the container for door sensor and door opening functionality. The omission of cgroups then at least has what it takes to produce a funny web video.


Battery-dependent embedded Devices

The system resource sparsity provided by cgroups also has implications for non-system resources. If certain tasks are not needed and their use is limited, this results in a reduction of power consumption. Although the contribution should not be overestimated, a conscientious distribution of resources is quite useful. This is an issue with mobile devices, but even more so with the use of embedded space devices. Here, even small contributions are appreciated, especially when Linux systems are needed.


Create Subsystems and cgroups

The most important resources or subsystems that can be controlled by cgroups are cpu (process prioritization), cpusets (processor core selection), blkio (I/O device control), devices (device control), memory (memory control), net_prio (network throughput control), cpuacct (processor time consumption), perf_event (performance analysis).

cgroups can be created manually using the familiar file operation commands via the console and then exist as a file in specific directories where processes and subsystems can connect and resource allocation can be determined here. It is simpler, however, to work with cgcreate and cgconfig, which can also be accessed via the console. For this, the libcgroup-tools package has to be installed. This works on Debian-based distributions with the command (with Feodora you would use yum instead of apt-get):

$ sudo apt-get install cgroup-tools


For simplicity, let's assume we have a shell script called memory_consumer.sh that we want to limit. With the following examples, we could create control groups for the memory and cpu subsystems.

Here, the memlimit group was created and limited with 2 megabytes of memory. As soon as the script requires more than 2 megabytes, the execution is terminated. In this case the Linux Out-Of-Memory Killer is started and the respective script or application is terminated. If you are using ELinOS or another distribution with root, these examples work without sudo command for each line.

Example #1

$ sudo cgcreate -g memory:memlimit
$ sudo cgset -r memory.limit_in_bytes=2M memlimit
$ sudo cgexec -g memory:memlimit ./memory_consumer.sh

The memory_consumer.sh allocates as much memory as possible.

memory_consumer.sh

#!/bin/bash
 
# create some long string
str=$(seq -w -s '' 10000) || exit 1
for index in $(seq 50); do
    # save string as new variable
    eval array$index=$str || exit 1
    echo "Allocate $index"
done

You can check if the memory limitations works by waiting a few seconds after executing memory_consumer.sh. As soon as the script reaches the 2 megabytes limit the Linux Out-Of-Memory Killer is started and kills the script.



The CPU time can be limited in a quite similar way:

Example #2

$ sudo cgcreate -g cpu:cputime
$ sudo cgset -r cpu.cfs_period_us=1000000 cputime
$ sudo cgset -r cpu.cfs_quota_us=200000 cputime
$ sudo cgexec -g cpu:cputime dd if=/dev/urandom of=/dev/null bs=1M

The script can consume a maximum of 20% of the CPU time. In this article, you can learn more about CPU limitation when using the CFS scheduler. You can verify the CPU load by calling top. This will not exceed 20%. Calling dd if=/dev/urandom of=/dev/null bs=1M without using the cpulimit group will use all the CPU time and top shows 100% for the process.


Persistent cgroups

The created cgroups have the disadvantage that they are volatile. To create persistent control groups we need to write a configuration file on the target device /etc/cgconfig.conf and save it with CTRL + D. There we can now insert and merge the two examples as mem_and_cpu_limit and reload them at system startup. Also (and this is interesting with respect to real-time applications) we want to limit the group to a certain processor core which we specify as 2 (which corresponds to core 3 since counting starts at 0).

/etc/cgconfig.conf

group mem_and_cpu_limit {
    cpu {
        cpu.cfs_period_us=1000000;
        cpu.cfs_quota_us=200000;
    }
    memory {
        memory.limit_in_bytes = 2M;
    }
 
    cpuset {
        cpuset.cpus="2";
    }
}

In order for the file to be processed at startup, the following command must be executed by the startup scripts:

$ cgconfigparser -l /etc/cgconfig.conf

In ELinOS you can store the following script in /usr/bin/autostart and enable the autostart directory support:

cgroups.sh

#!/bin/sh
 
cgconfigparser -l /etc/cgconfig.conf


Finally you can execute your application or script with the preconfigured groups:

$ sudo cgexec -g memory,cpu,cpuset:mem_and_cpu_limit memory_consumer.sh &


TL;DR

cgroups are an efficient way to control resources and permissions within a Linux system. With their use, embedded system architects under Linux have means at hand to use the limited capacities of the hardware more efficiently, to gain overview and to harden their Linux. Resource allocation is a way to extend device runtime, especially for battery-bound systems. Also the real-time capability of embedded Linux systems with preempt patch benefits by skillful use of cgroups with the effect that jitter can be reduced. Embedded Linux ELinOS provides embedded system architects with all the means to set up embedded systems in a convenient way and to optimize them with the help of cgroups.

More information at www.sysgo.com/elinos

Back to the Overview

configuration, docker, memory

Useful Links