Today is a very special day for all of us at Sylabs and, by the end of this post, we hope you’ll agree that it’s also a very special day for Singularity.
Sylabs at 1
One year ago today, Sylabs officially emerged from stealth mode aiming to provide the market with solutions and services based upon Singularity containers. From the outset, Singularity founder as well as Sylabs founder and CEO Gregory Kurtzer emphasized compute-driven workloads:
Already the container platform of choice by academia and commercial HPC centers, Singularity’s features also make it the ideal container technology for artificial intelligence, machine / deep learning, compute-driven analytics, and data science — areas that we characterize as enterprise performance computing, or EPC. These applications carry data-intensive workloads that demand HPC-like resources, and, as more companies leverage data to support their businesses, the need to properly containerize and support those workflows has grown substantially.
No stranger to founding open source software projects (e.g., CentOS Linux and Warewulf), even Greg himself has been impressed by Singularity’s rate of adoption – now numbering some 40,000 users who run many millions of containerized applications per day.
Founded originally in late 2015 when Greg was an HPC Systems Architect and Technical Lead at the Lawrence Berkeley National Laboratory, Singularity witnessed 13 updates in arriving at the version 2.4.2 release when Sylabs emerged a year ago. The last major update to this original code base was the version 2.6 release – a compelling and solid Singularity offering that remains in use at many sites worldwide.
Much of the recent history beyond this 2.6 release of Singularity has been captured in our year-end white paper. Briefly, Sylabs lead two strategic initiatives in arriving at the 3.0 milestone for the software:
- The Singularity core was completely reimplemented in a combination of Go and C
- The Singularity Image Format (SIF) was introduced to encapsulate the entire container runtime into a single, immutable, and cryptographically signed file
Additionally, the same release included an alpha preview of the Sylabs Cloud – a Singularity-centric portal hosted in the cloud that enables key signing and verification, hosted containers, and remote builds.
Although the software continued to receive accolades and awards in 2018, its proudest achievement is the ever-growing list of sites that, with similar pride, openly convey their use of the software. One of these sites is the Argonne Leadership Computing Facility (ALCF):
a national scientific user facility that provides supercomputing resources and expertise to the scientific and engineering community to accelerate the pace of discovery and innovation in a broad range of disciplines.
In a very recent HPCwire article, the benefits of Singularity were expressed quantitatively by the ALCF:
The researchers were able to run the ATLAS software on Theta inside a Singularity container via Yoda, an MPI-enabled Python application the team developed to communicate between CERN and ALCF systems and ensure all nodes in the latter are supplied with work throughout execution. The use of Singularity resulted in linear scaling on up to 1,024 of Theta’s nodes, with event processing improved by a factor of four.
Not only do these results validate Greg’s original vision for the software plus Sylabs’ focus on enterprise performance computing (EPC), they amplify the value of reproducible science and mobile runtimes at the leadership scale of an 11.69 petaflop supercomputer.
Singularity 3.1 RC1
Another critical component of that one-year old press release was the following promise:
Sylabs remains committed to the open source community. All of its development efforts and engagement of the community will reside in the public GitHub repository for Singularity.
In keeping with this spirit upon which the company was founded, the latter part of this post is all about giving back to the community. Specifically, we are delighted to share that the first candidate for the version 3.1 release of the software is now available on the project’s GitHub site. Included in Singularity 3.1 RC1 are the following enhancements:
- Introduction of the “oci” command group to support a new Open Containers Initiative (OCI) compliant variant of the Singularity runtime – an enhancement that complements existing support for the OCI image specification introduced in version 3.0.0 of Singularity
- Introduction of the “cache” command group to inspect and manage cached images
- The ability to build the Singularity CLI in the Darwin operating environment for limited functionality on macOS platforms
- Support for additional bootstrap agents “scratch” and “zypper”
- Improved support for The Scientific Filesystem
- Significantly expanded Continuous Integration (CI) unit and end-to-end testing
For the complete list, please refer to the Singularity 3.1 RC1 changelog. Because this is only a first candidate for the 3.1 release of Singularity, it is important to note that enhancements delivered in the generally available (GA) release may differ from that which has been shared here.
Singularity 3.1 RC1 can be found on Git Hub here. We look forward to your feedback, as together we progress this candidate towards a GA release.
Our year-end white paper included a roadmap. From OCI compliance to integration with Kubernetes and more, we are collectively striking off items from this to-do list. We can assure you that there’s more to come! In fact, the best opportunity to take the pulse of the community, and find out what to expect next, will be the inaugural meeting of the Singularity User Group (SUG). Brought to you by the San Diego Supercomputer Center (SDSC) and Sylabs, SUG provides the perfect opportunity to share and to learn. Our CFP is open for another week, as is registration at the event site, for this March 12-13, 2019 event. Whether or not you make it to SUG, the path forward is straightforward: get involved! To get started with Singularity, please click here.
Finally, if you’re already an actively engaged Singularity user, developer, or provider, we extend our heartfelt appreciation your way as well. Simply put, Singularity and the ecosystem that embraces it would not be what it is today without our collective stewardship.
Onwards, with mobility … to the next reproducible and secure orbit.