GitHub made a Codespace, easy development environment. Codespaces provides software teams a faster, more collaborative development environment in the cloud. Read more on our Codespaces page.
The GitHub.com codebase has been around 14 years old. Rails were just two years old when the first commit to GitHub.com was pushed. AWS was the only one. Azure and GCP were not yet available. It might not seem like much in COBOL time but it is quite significant in internet time.
Over the past 14 years, GitHub.com’s core repository (Github/GitHub has received over a million commits. Most of those commits are from developers who test and build on macOS.
But our development platform is evolving. We have moved from our macOS model to Codespaces over the last months for most of our GitHub.com development. This was a significant shift in our development workflow. This has made Codespaces’ product more robust and better positioned to support the future of GitHub.com.
The status quo
We have spent a lot of time and effort over the years to make local development easy. Our scripts-to-rule-them-all approach has presented a familiar interface to engineers for some time now–new hires could clone GitHub/Github, run setup and bootstrap scripts, and have a local instance of GitHub.com running in a half-day’s time. Most cases were successful, but if they weren’t, the bootstrap script would open a GitHub Issue connecting the new hire to internal support. Our #friction Slack channel–staffed by helpful, kind engineers–could debug nearly any system configuration under the sun.
Despite all of our efforts, local development has remained fragile. Even the most innocuous changes can make a local environment unusable and require hours of development time to repair. Mysterious breakage was so common and catastrophic that we’d codified an option for our bootstrap script: --nuke-from-orbit. The script will delete as much as it can to restore the environment to a good condition when invoked.
This is, of course, a classic tale that any software engineer will immediately recognize. Local development environments can be fragile. Even when they function perfectly, local development environments can feel increasingly out of sync with the instant-on, anywhere-anytime world in which we operate.
It was difficult to collaborate on multiple branches in multiple projects. It was common for us to be stuck in a 45-minute bootstrap every time a branch added new dependencies, changed the schema, or branched to a different SHA. This was due to the rapid changes in our codebase (we deploy hundreds of changes each day), and it was often a source of engineering friction.
We weren’t alone in noticing this. Codespaces was built with the help of several top-of-the-line engineering companies that had created Codespaces-like platforms to address similar problems. This type of productivity loss can be eliminated quickly on any scale.
Development infrastructure
Servers are still a commodity in the infrastructure industry. This is consistent with industry best practices. It is believed that every server is not unique, indispensable, or irreplaceable. Without much fuss, any piece could be removed and replaced with a similar piece. It’s okay if a server breaks down. It’s okay to tear it down and put it back.
Each local development environment is unique and has its own quirks. They require close monitoring to ensure their maintenance. You don’t want to be building software when your environment is being degraded by the next git pull, bootstrap, or another occurrence. It’s not a convention to have a warm laptop on standby.
There are many things to be proud of when we treat development environments like ours. They’re where we spend most of our time. Our workbench is not only a tool for productivity, but it’s also an expression of us.
Codespaces was a way for us to think of our dev environments as infrastructure. It is a commodity that we can churn, but we still have the ability to curate and manage our workbench. Visual Studio Code extensions, settings sync, and dotfiles repos allow us to bring our environment into our computer. A broken workbench in this context is minor; we can now provision a new codespace at the known good status and get back to our work.
Adopting Codespaces
Migration to Codespaces addressed our shortcomings, motivated us further to push the product, and gave us leverage to improve our overall development experience.
While our migration story is happy, the initial stages of our transition were difficult. The GitHub.com repository has almost 13 GB of disk space; cloning it takes only 20 minutes. Bootstrapping a GitHub.com Codespace takes upwards of 45 minutes when you add dependency setup. The application would not run if had a successfully mounted repository into a codespace.
All 14 years of macOS-centric assumptions that were ingrained in our bootstrapping process would have to be rewritten.
These challenges were a great opportunity to showcase the best of GitHub. We had many contributors from all over the company who helped us to review past decisions and question long-held assumptions. They also worked at the source level in order to separate GitHub development from macOS. We finally were able to provide working GitHub.com codespaces for Linux hosts. We had to figure how to make it all work.
45 minutes to 5 minutes
Codespaces aims to create a model in which development environments can be provisioned as needed for each task. This is roughly a 1:1 mapping of codespaces and branches. We need to support task-based workflows by providing instant-on. Although 45 minutes was not enough time to accomplish our task-based goal, we could still see low-hanging fruits, rich in potential optimizations.
First, Codespaces will change how they clone GitHub/Github. Codespaces will now perform a shallow clone after being provisioned. Then, Codespaces will create a codespace with the most recent commits and unshallow repository history. This reduced the time it took to clone from 20 minutes down to 90 seconds.
Next, we had the opportunity to cache the network of software and services that support GitHub.com. This includes traditional Gemfile-based dependencies, as well as services written using C, Go and a custom Ruby build. A GitHub Action was created that would run nightly to clone the repository and bootstrap dependencies. Finally, it would build and push a Docker Image of the result. The published image was then used as the base image in GitHub/Github’s dev container–config-as-code for Codespaces environments. Now, our codespaces were 95%+ bootstrapped.
These changes along with a few app and service level optimizations reduced the time it took to create GitHub.com codespaces from 45 minutes down to five minutes. But five minutes is still quite a distance from “instant-on.” Well-known studies have shown people can sustain roughly ten seconds of wait time before falling out of the flow. We had yet to make great strides.
5 minutes to 10 seconds
Although five minutes was a significant improvement over the previous time, these changes were tradeoffs that suggested a larger product need.
Although our shallow clone method is useful for quickly launching into Codespaces, we still had to pay the full cost of a full clone at some points. The unshallow post-create created load with distracting side results. A similar problem would be faced by any large and complex project. Cloning and bootstrapping create contention for resources.
What if we could clone the repository and bootstrap it ahead of time so that when an engineer asks for a codespace, we had already done most of the work.
Prebuilds is a pool of codespaces that have been fully cloned, bootstrapped, and waiting for a developer to connect them. Prebuilds has proven to be a great engineering investment. We can create preconfigured codespaces that are ready for GitHub.com development in just 10 seconds.
In less than the time it takes to install Slack, new hires can move from zero to a functional development environment. Engineers can create new codespaces to support parallel work streams with minimal overhead. Our engineers can quickly set up a new environment if an environment is not working properly.
Increased leverage
We had some real problems with Codespaces. It eliminated the fragility of local development environments and gave us strong new leverage to improve the developer experience at GitHub.
This wedge allows us to perform additional setup and optimization work in environments that are not local. The cost of these optimizations is too high (both time and patience). Prebuilds allow us to prime our language server cache, gem documentation, run pending data migrations, and enable both GitHub.com Enterprise development modes. This is a task that typically requires yet another loop through bootstrap.
Codespaces allows us to upgrade all engineers’ machine specs in one configuration change. We used 16GB RAM, 8 core VMs in the initial stages of our Codespaces migration. Those machines were adequate, but GitHub.com has many services and will happily consume any core or RAM that we provide. We moved to 64GB RAM, 32 core VMs. We upgraded all engineers’ machines by changing one line of configuration.
Codespaces also began to take business from our “review laboratory” platform, which is a production-like environment in which we can preview changes with internal colleagues. In order to share their work, GitHub engineers had to commit to Codespaces and then deploy to a review laboratory instance. This often required peer review. Friction. We can now ctrl+click to grab a preview URL and email it to a colleague. No commit, no push, no review, no deploy — just a live look at port 80 on my codespace.
Command line
Visual Studio Code is a great tool. Visual Studio Code is the main tool GitHub.com engineers use for interfacing with codespaces. It’s not as easy to ask Vim and Emacs users for a commit to a graphic editor. We had to invite everyone if Codespaces were our future.
We could easily support shell-based colleagues by updating our prebuilt image. This update initializes sshd using our GitHub public key, opens port 22, and forwards port 22 out of the codespace.
There, GitHub engineers have the ability to run Vim, Emacs, or even if so desired.
This worked extremely well! This has worked out exceedingly well, just like Docker image caching leads to prebuilds. The obvious next step is to take what we did for the GitHub.com Codespace and make it a top-class experience for all codespaces.
Reception
It is difficult to change, especially when it comes down to developing environments. Fortunately, GitHub engineers have a tendency to be curious and kind. They are quickly becoming Codespaces Superfans.
Codespaces now serve as the default development environment at GitHub.com. That #friction Slack channel that we mentioned earlier to help debug local development environment problems? It will be archived.
Every day we are adding more services to GitHub and more engineers. We also discover new stories about the benefits Codespaces can bring. You’ll find a common theme at the heart of every story: I found a better program, I’m more productive, and I won’t go back.
The GitHub.com codebase has been around 14 years old. Rails were just two years old when the first commit to GitHub.com was pushed. AWS was the only one. Azure and GCP were not yet available. It might not seem like much in COBOL time but it is quite significant in internet time.
Over the past 14 years, GitHub.com’s core repository (Github/GitHub has received over a million commits. Most of those commits are from developers who test and build on macOS.
But our development platform is evolving. We have moved from our macOS model to Codespaces over the last months for most of our GitHub.com development. This was a significant shift in our development workflow. This has made Codespaces’ product more robust and better positioned to support the future of GitHub.com.
Each local development environment is unique and has its own quirks. They require close monitoring to ensure their maintenance. You don’t want to be building software when your environment is being degraded by the next git pull, bootstrap, or another occurrence. It’s not a convention to have a warm laptop on standby.
There are many things to be proud of when we treat development environments like ours. They’re where we spend most of our time. Our workbench is not only a tool for productivity, but it’s also an expression of us.
Codespaces was a way for us to think of our dev environments as infrastructure. It is a commodity that we can churn, but we still have the ability to curate and manage our workbench. Visual Studio Code extensions, settings sync, and dotfiles repos allow us to bring our environment into our computer. A broken workbench in this context is minor; we can now provision a new codespace at the known good status and get back to our work.
Read More: How to rank SEO content
