What’s new in Singularity 2.5 and why it affects everyone using containers

The Sylabs team recently discovered an exploit vector to all container runtimes, that allows a malicious user to gain additional privileges within a container on hosts running kernels that do not support the PR_SET_NO_NEW_PRIVS feature.

This discovery triggered the 2.5.0 release of Singularity and prompted a ‘high severity’ security alert to both Singularity and Singularity Pro users. With this release, Singularity will refuse to run if the underlying kernel cannot provide the mechanisms required to prevent privilege escalation.

There are no existing mitigations or workarounds for these issues outside of updating the kernel to support PR_SET_NO_NEW_PRIVS.  Singularity is not the only container platform affected; this vulnerability can be exploited using any container runtime on a vulnerable kernel.

There have been many questions both on Slack and the Singularity Google Group about the 2.5.0 release, so we decided to dig deeper into what has changed and why in a Lab Note.

 

A bit of background on the security issue

With Singularity 2.5 we drop support for running Singularity on hosts that do not support the prctl() function PR_SET_NO_NEW_PRIVS.

The NO_NEW_PRIVS bit is a mechanism to make it safe for a process to modify its execution environment in a manner that persists across execve() calls. It is inherited across fork(), clone(), and execve() and cannot be unset.

The execve() system call can normally grant a newly-started program privileges that its parent did not have.  This is what happens when we run a setuid/setgid program or use file capabilities.

With the NO_NEW_PRIVS bit set, it’s impossible to grant privilege to do anything that could not have been done before the execve call.

Kernels that do not have this feature cannot implement container runtimes securely and should be considered inherently insecure for this purpose (and others).

 

Who is affected

Sylabs engineers have been able to reproduce this exploit and gain root privileges on other container runtimes using exactly the same attack vector that led to the discovery of the vulnerability on Singularity.

This problem is NOT specific to Singularity, so even if you are not using Singularity, you should take care and review what impact this could have in your environment.

The PR_SET_NO_NEW_PRIVS feature was added to prctl() in the Linux 3.5 kernel. Various distributions have since backported this feature to currently maintained kernels (for example, Red Hat added this feature to RHEL 6.7 with the 2.6.32-504.16.2 kernel).

 

What steps we took and why

Because no mitigations exist,  it is safest to put blocks in place preventing Singularity 2.5 from building or running on any Linux kernel that lacks the NO_NEW_PRIVS feature set..

In the past, pains have been taken to maintain support for legacy kernels, and the decision to abandon this effort is not made lightly.  But it is really the only sane thing to do because security is paramount for Singularity users.

Sylabs has not provided details about this exploit because there is no workaround short of upgrading the kernel or uninstalling Singularity.  So giving more information will only help malicious parties.

As of now, Sylabs will not request a  CVE for this issue because it only affects old kernels and CVE’s associated with PR_SET_NO_NEW_PRIVS have already been provided and resolved.

Check if you are affected and upgrade

As a convenience Sylabs provides a script to allow you to automatically check whether your Linux system implements PR_SET_NO_NEW_PRIVS properly via the following command:

 

curl -s https://www.sylabs.io/privtest/no_new_privs.sh | bash

 

This script compiles a small test program to test the prctl() calls for the PR_SET_NO_NEW_PRIVS feature. Please verify your system using this script before running Singularity.

If you find that your system does not support NO_NEW_PRIVS properly, the only option is to upgrade your Linux version or to request from your upstream Linux vendor that they add support for PR_SET_NO_NEW_PRIVS to their maintenance kernels.

Red Hat, CentOS, Scientific Linux have already done so in their currently maintained distribution releases.

Tags: