The last few years have been an amazing whirlwind within the high performance computing (HPC) community thanks to a buzzword that has finally moved from the enterprise world to scientific research. And that word? Containers. But, first, as Mick Jagger says, please allow me to introduce myself….
I’ve spent the last near 20 years at the US Department Of Energy Labs as an HPC systems architect, where I designed solutions for scientific and data centric computational workflows. During that time, I also dove deep into the open source community and founded several very successful projects including CentOS Linux and Warewulf, which has become the most utilized stateless HPC cluster provisioner. More recently, I founded Singularity, an open source container technology for HPC and which has had the fastest uptake of them all.
All this has led me to today. I am very proud to announce Sylabs, a company that will introduce Singularity to the enterprise and HPC commercial market and advance the fields of artificial intelligence, machine/deep learning, and advanced analytics.
Singularity in HPC
While most container platforms are designed for microservices, Singularity was designed for HPC and scientific use cases. Founded in late 2015, Singularity’s adoption rate has been staggering. It’s now in its 13th release (version 2.4.2) and runs over a million containers a day, via the Open Science Grid alone (a consortium that provides distributed computing resources for scientific research). According to a voluntary registry, it has an estimated user base of more than 25,000 spread around the world at major universities and government institutions including Stanford University, the National Institutes of Health, San Diego Supercomputer Center, the Texas Advanced Computing Center, as well as multiple US national labs.
The features that facilitated Singularity’s growth appeal to scientists as well as system providers. Some of these include:
- Security: Unlike other container technologies, Singularity enables users to run containers without the security implications of granting users control of a root-owned daemon process or kernel features which do not exist in legacy operating systems
- Single File Format: Singularity containers consist of a single file (SIF) that encapsulates the runtime environment. Benefits of this format enable extreme user mobility, security controls compliance, archive, reproducibility and trust via cryptographic signing of the runtime container
- Support for high performance hardware: Singularity supports GPUs, Infiniband, and Intel Phi natively which are necessary resources for accelerating HPC and enterprise computing workloads
- Compatibility: Singularity is compatible with Docker Hub and, when applicable, direct host IO and integrates seamlessly with resource managers, Message Passing Interface (MPI), batch job workflows, etc.
Singularity in the Enterprise
Already the container platform of choice by academia and commercial HPC centers, Singularity’s features also make it the ideal container technology for artificial intelligence, machine / deep learning, compute-driven analytics, and data science — areas that we characterize as enterprise performance computing™, or EPC™. These applications carry data-intensive workloads that demand HPC-like resources, and, as more companies leverage data to support their businesses, the need to properly containerize and support those workflows has grown substantially.
Singularity was designed for native use on GPUs, which enables users to go beyond microservices enterprise deployments and create “build once, run anywhere” applications that move seamlessly between HPC, enterprise, and cloud resources. To this point, Microsoft’s Azure Batch announced its support of Singularity in November. We have also been working to support enterprise orchestration platforms like Kubernetes and Mesos and align with the Open Container Initiative.
Open for Business
Creating business models out of open source software can be tricky when not done properly, so we look towards models that have proven to stand the test of time. We’re putting all of our development efforts and engagement of the community into the public GitHub repository for Singularity. We also plan to make commercially available Singularity Pro, which will be available via a subscription license.
Sylabs is a fast-growing team of 12 that is engaged with companies of all sizes and in different industry verticals for a private beta. If you’re interested in learning more, contact us here. Sylabs is privately funded by RStor Inc., a startup in stealth mode.
I’ll also sneak in a brief announcement: we’re hosting a Singularity meetup and hackathon on February 22-23 in Berkeley, CA. Stay tuned for more details!