1
Engineers typically have a fixed number of cloud environments, including production, staging, and maybe some additional pre-production environments. They use these to preview features, share work with other team members, do manual QA/automated testing, and more. As the number of engineers grows, having a fixed number of cloud environments becomes a bottleneck. Teams end up queuing for access to staging and so on. Creating additional environments and keeping them in sync, though, is a major headache, especially when the environments are reasonably complex.
When this problem reaches a boiling point, teams either have to slow down feature development, test on production, or deal with building their own on-demand platform internally. That’s where we come in. We enable on-demand spinup of unlimited environments for quicker development. Engineers use these as environments-per-feature—in parallel—to share previews with QA/other engineers/product people, to create demo environments for clients with features that aren't in production, to run automated testing on ephemeral environments as part of their CI processes, and more.
Vignesh and I are engineers with experience in infra/research at Facebook/Apple/Microsoft. We met as roommates at Caltech. We got into YC with a very different idea (a way for stores to ask customers questions and offer instant recommendations/promotions based on their responses) but soon abandoned it (it was more exciting to us as a technical challenge than to stores as a practical solution) and found ourselves testing different ideas as quickly as we could.
During this period, right before every launch without fail, Vignesh and I would collide on staging: features that worked locally would break on prod infra. We tried to provision our own individual staging environments, but setting them up and keeping them in sync sucked up valuable bandwidth. The gold-standard workflow, in my mind, was the one-click “On-Demand” environment provisioning I’d had at Facebook, wherein I could click a button and instantiate a live feature preview. We found solutions that promised on-demand environment spin-up, but none of them worked for us; they were either incompatible with our stack (mostly built around App Engine), didn't interface with our cloud provider, or required too much overhead and finessing to set up.
This was the seed for our idea. Talking to larger startups, we realized that the inconveniences we faced were just a few of the major pains faced by bigger engineering teams with nascent DevOps; they often had brittle, scattered workflows around managing their environments and growing queues for features. They needed a solution to flexibly create, destroy, and replicate environments, through both ad-hoc and developer triggers like PR’s and CI builds. While there are a number of existing services that aim to simplify DevOps and/or provide easier workflows around environments, engineering teams we spoke to were either unable or unwilling to use them due to incompatibility with their specific setups.
We realized that a better approach would be to abstract away the specifics of each environment by using infra-as-code as middleware. Infra-as-code solutions like Terraform, CloudFormation, etc. can represent infrastructure thoroughly in a standardized fashion. The majority of people we talked to were maintaining their infrastructure using Terraform or something similar, and it was easy to build out pipelines for those who didn't by using Terraform under the hood. This was the key to what became Gallery.
Since every company manages their environments differently (cloud provider, use of K8s or infra-as-code, cloud-managed services, security/privacy layers), creating a product that can work with them all out-of-the-box is a challenge. Unlike other solutions on the market, Gallery isn't a Kubernetes orchestrator that replicates containerized environments; rather, we can represent entire cloud projects, including managed services (e.g. App Engine, Elastic Beanstalk, etc.), spin them up in sandboxed cloud projects, and automate their teardown as well.
To create cloud resources on your GCP/AWS accounts, we use Terraform as a middleware. For select cloud services, we can understand an existing project, generate the Terraform corresponding to it, and use it to spin up your resources. We allow users to link their own Terraform code to allow spin-up of more complex environments as well. When you trigger the creation of a new environment, we spin up a worker that applies the Terraform template and performs any post-deploy actions to seed the newly generated infrastructure (for example, copying over initialization data into databases). We store your Terraform state file and manage tearing down environments when they outlive their TTL, or deletion is triggered by a specific action (e.g. merging a pull request).
Once we spin up infrastructure, we pull application code from your app repositories, and allow you to specify build and deployment commands that target the newly created environment. This lets you use your current deployment scripts on Gallery almost as-is, for fast and straightforward onboarding. We have integrations with Github/Gitlab around the Pull Request/Merge Request flow, letting you create environments that track a branch whenever a PR is created.
We're a SaaS product, and we have pricing tiers based on the number of concurrent environments. We’ve prepared a live demo account that you can play around with as a read-only team member here: http://a.getgallery.co/teams/14. I've also prepared a demo video of how to set up ephemeral environments per-PR in just a couple of minutes (https://www.loom.com/share/26165ea69f0d4b7b974019bdf72e5d11). You can see our docs here: docs.usegallery.com.
Thank you so much for reading! We’d love to hear your thoughts, ideas, and experiences around the problem we’re tackling and the solution we’re proposing!