The story behind gLinux, Google’s discreet distribution

The best known Google operating system is Chrome OS. But the American giant uses its own Linux distribution for workstations: gLinux. Its main characteristic: reducing the work and stress of the teams in charge of its deployment and operation.

If you look around Google’s offices in Mountain View, California, you’ll see Windows machines, Chromebooks, Macs, and gLinux desktops. G what, you ask? Well, in addition to relying on Linux for its servers, Google actually has its own desktop Linux distribution. But impossible – damn it! – to download it for more than a decade. Initially, it was called Goobuntu, not without hiding a relationship with the famous distro Ubuntu. But in 2018, the American firm moved its development to another Linux distribution, named gLinux, this time based on Debian. Why ? Google explained that going to the LTS version of Goobuntu meant upgrading every system in its fleet of more than 100,000 devices before the OS’s end-of-life date. A titanic project. Add to that the tedious need to fully customize engineers’ PCs, and Google decided it was too expensive. In addition, the effort to upgrade the fleet of terminals to Goobuntu would have taken a good part of the year, so much time bitten on a support spread over two years before having to start the same process again for a next version. LTS.

“This whole process was a huge stressor for our team, as we received hundreds of bugs with requests for help with critical cases,” explained Google, who had had enough and therefore switched on Debian Linux. The company then created a Debian distribution in continuous development: GLinux Rolling Debian Testing (Rodete). The idea? That users and developers take advantage of latest updates and fixes as they are created and deemed ready for production. These distributions include Arch Linux, Debian Testing, and openSUSE Tumbleweed. For Google, the immediate goal was to get out of the two-year upgrade cycle. As the move to continuous integration/continuous deployment (CI/CD) showed, these incremental changes work well. They are also easier to control and undo if something goes wrong.

Life-Changing Complete Automation in the Pipeline

To make it all work without a lot of blood, sweat, and tears, Google created a workflow system, Sieve. Whenever Sieve spots a latest version of a Debian package, it starts a build. These packages are bundled into package groups and once they’ve been created, Google runs a virtualized test suite to ensure that no core components and developer workflows are broken. Then each group is tested separately with a full system install, boot, and run of the local test suite. The package builds in minutes, but testing can take up to an hour. Once done, all packages are merged with the latest gLinux package pool. Then, when the vendor decides it’s time to put it into production, the team then takes a snapshot of that pool before deployment and uses “reliability engineering” (SRE) principles such as incremental canarying to make sure nothing goes wrong.

Today, thanks to Sieve, gLinux development boils down to a single release engineer position that rotates between team members. There are no major modernization efforts to be made. No multi-stage alpha, beta, and general availability (GA) releases to manage. Even better, with the rolling release schedule, Google can quickly fix security flaws across the entire fleet without compromising stability. Previously, security engineers had to carefully review each Debian Security Advisory (DSA) to ensure that the patch was present. Additionally, “Google’s improved test suite and integration testing with key partner teams that run mission-critical development systems also resulted in a more stable experience using a Linux distribution that provides the latest Linux kernel releases.” Our strong desire to automate everything in the pipeline has greatly reduced the toil and stress within the team. It is now also possible for us to report bugs and incompatibilities with other versions of the library while ensuring that Google tools work better within the Linux ecosystem.

An expected public distro but for the moment in dream

Going forward, the Google team said they will work more closely with Debian upstream and contribute more to their internal patches to maintain their package ecosystem. All this sounds good but leads to two remarks. First, for some organizations, LTS releases still make sense. Don’t need the latest and greatest programs for your business? So a Linux Ubuntu or Red Hat LTS still makes sense. Second, and most importantly: Sieve sounds like a rallying cry. Who doesn’t want a program that can automate a streaming delivery production pipeline to the point where it only takes one engineer to maintain it on over 100,000 seats? Nobody… So what is Google waiting for to release the Sieve code to start producing Linux desktop builds continuously? Come on, a little more effort.

Leave a Comment