Analysis of China Mobile's container-customized Linux operating system

General operating systems integrate a large number of software and enable many services by default. Most of these software and services are not required for the container environment. Therefore, deploying container services based on a general operating system will not only increase system overhead, but also lead to environmental instability and an expansion of the security attack surface. Compared with traditional general-purpose operating systems, container operating systems are deeply tailored and optimized for container applications, providing a lightweight minimum running environment for containers. This article introduces some of China Mobile’s attempts at containerized systems and some of its achievements.

China Mobile launched the research and development of a container-customized operating system in 2017, deeply customized it based on the Big Cloud operating system BC-LINUX, and officially released version 1.0 in May of that year, named "Big Cloud Containerized Operating System". BC-LINUX is an enterprise-level general-purpose Linux operating system independently developed by China Mobile based on the CentOS open source community and leveraging the openness advantages of open source technology through customized means. Currently, nearly 20,000 units have been deployed within China Mobile. On the basis of the general system, the Dayun containerized operating system provides a streamlined container operating environment through kernel optimization and system tailoring and other technical means, improves the system's operating speed, achieves system minimization and performance optimization. as the picture shows.

To strike a balance between system ease of use and simplicity, Dayun containerized operating system cuts out irrelevant software packages and services while retaining the basic functions of the system. On the basis of providing the minimum operating environment for containers, Dayun containerized operating system ensures that common services and functions of the operating system are not missing, reduces system overhead, and reduces the difficulty of system operation and maintenance. Compared with general systems, Dayun containerized operating system The number of system software packages has been reduced from 3723 to 376, the number of services has been reduced from 254 to 143, and the installation image size has been reduced from 4.31G to 770M, as shown in the figure.

Dayun containerized operating system integrates Docker components and provides 11 mainstream open source middleware container images for out-of-the-box use. We provide version updates, security warnings, vulnerability fixes and technical support services for these 11 open source components, and regularly scan and update to fix security vulnerabilities in container images to ensure that there are no security issues in container images, as shown in the figure.

For container usage scenarios, Dayun containerized operating system provides an optimized customized kernel. The customized kernel is customized and developed based on the latest long-term support version 4.9 of the kernel community. The kernel is tailored for the container business and adds many function enhancements and performance optimizations for XFS, Btrfs and Overlayfs. The Dayun container operating system supports the overlay2 storage driver. Compared with overlay, the overlay2 of Dayun containerized operating system is more efficient in terms of inode usage. In addition, China Mobile's multiple patches for containers are added to the customized kernel, which realizes the separation of some network configuration parameters of the container and the host system, and meets the tuning needs of the container business system in high network concurrency scenarios, as shown in the figure.

The big cloud containerized system reduces the security attack surface of the system by cutting out unnecessary services. At the same time, the system has built-in security hardening software independently developed by China Mobile, which can comprehensively scan the system for security vulnerabilities and security configuration issues, provide security assessment results and repair suggestions, and can harden the system with one click and turn on the system security mode.
The customized kernel is based on the 4.9 kernel, and higher versions of the kernel have fixed many security vulnerabilities, such as the kernel privilege escalation vulnerability Dirty Cow (CVE-2016-5195). A system with this vulnerability can bypass the system's security policy in the container and obtain root permissions of the host system, and then can view, modify or even delete any files in the host, thus posing security risks to the host and other containers.

In response to the problem of business interruption caused by dynamic library and kernel upgrades in traditional upgrade methods, Dayun containerized operating system has launched hot patch technology. Hot patch technology is an online defect and vulnerability repair technology that does not affect the business. It can achieve online upgrades of dynamic libraries and kernels without interrupting services or restarting the system. It does not affect system performance and significantly improves business performance. System stability and availability.
Specifically, dynamic library hot upgrade solves the problem of dynamic library upgrade of business programs. It is suitable for dynamic library upgrade of all processes. It is simple and convenient to operate, has high reliability, and supports multiple re-entry and reverse operations, as shown in the figure. .

Kernel hot upgrade technology, based on the kernel's ftrace mechanism, dynamically adds detection points to realize online replacement of function-level execution processes. This technology allows kernel upgrades without restarting the system, minimizing system downtime. For important security vulnerabilities, Dayun containerized operating system can respond quickly. At the same time, the system supports rollback operations and can quickly restore the kernel to the state before the upgrade.

For containerized operating systems, Dayun can provide continuous system updates and technical support services, track security vulnerabilities in the operating system, especially Docker components, and issue security warnings and vulnerability update patch packages, as shown in the figure.

Since its release, the Dayun containerized operating system has been commercially promoted within China Mobile. The current deployment scale has reached nearly 200 nodes. It uses the Kubernetes container management platform and has been running stably for 6 months, supporting 5,000 containers. The product’s Safety, stability and reliability have been fully verified in the project.

The above is the detailed content of Analysis of China Mobile's container-customized Linux operating system. For more information, please follow other related articles on the PHP Chinese website!