Working on Self-Managed Software vs. Working on Cloud Managed Services

I started my software engineering career at MemSQL (now called SingleStoreDB) and our product was 100% self-managed back then. This meant that our customers had to download, install and then maintain the installation of our database product on servers that they managed. Then in late 2019, we launched SingleStore Cloud and since then I've had the experience of mainly working on fully managed software. As an engineer, the experience of working on both is very different.

(In)Consistent Underlying Infrastructure. When building software that runs on our customers' infrastructure, it's really important to test and support different base operating systems, Linux kernel versions, and other diverging hardware configurations. This can be a real pain, because it's unlikely that as the provider, you'll have the leverage to push for modern infrastructure from all of your customers. And this means that tests take longer and longer to run, which means that the software development lifecycle naturally has to be slower. On the other hand, if you manage the infrastructure, most likely, you have to support a single base Docker image, or a very narrow set of VM instance types. And you can totally push to modernize these over time. This, in turn, allows you to use more modern tooling.

Rate of Software Delivery. When working on self-managed software, for some of the reasons I already mentioned above, but for many other reasons too, the rate of delivery is typically slower. You're most likely not going to be able to make very frequent updates to the software, and the engineers who build this type of software won't be learning as quickly. Obviously, customers of self-managed software usually see this as an advantage (the less updates, the less often things break!), but not being able to iterate quickly can mean that we're not learning as fast as we could about what works and what doesn't work. It also means that quick improvements and/or bug fixes can take longer to be in customers' hands. And I find that as an engineer, there's few better feelings than being able to ship something quickly and seeing customers use it shortly after that!

Open Canvas for Innovation. I have found that it's much easier to innovate on the set of features you can provide if you're not constrained by the hardware limitations of your customers' infrastructure. As an engineer, this has been very beneficial to my career, as I've been able to push the envelope on what our product can achieve.

• If we need a blob store (S3), it is right there for the taking.
• The latest and greatest load balancers? You got it.
• Managed Kubernetes? Also available.
• ...

And so on and so forth. We've been able to leverage these and many more cloud-native services to expand upon what our product can do. Finally, we're free to explore multi-tenancy and shared pieces of infrastructure, which result in lower costs for both ourselves and our customers.

Oncall Support. When I transitioned to mostly working on fully-managed software, I was a bit fearful of having to be part of an oncall rotation (we actually already had a very lightweight oncall before having a managed service, but it was very limited). I had heard the horror stories from other companies, and I wasn't looking forward to that. However, after some time, I've learned that carrying a pager is actually not all that bad. And, in fact, it is one of the absolute best ways to grow as a software engineer. Being exposed to how the software you build breaks in all sorts of unexpected ways over time teaches you to build better software. And it also helps us gain empathy for both our customers and our support teams, which only makes us more aware of what to prioritize. So, all in all, I highly recommend software engineers to join their oncall escalations and put themselves in a position to grow and become more impactful through these.

Security and Privacy. Obviously, if you run the infrastructure, you're primarily responsible for its security. When I worked on self-managed software, security was not top of mind as it is today for me. And I'm very glad that I've been forced to learn so much about security in SaaS and Cloud over the years. All this experience will surely be very useful for the rest of my career. And it can be very fun to think about ways to break these systems, and learn from the many hacking attempts that you end up being subject to over the years of running these services.

Final Note: Going Hybrid

Some providers are now going hybrid, with some sort of BYOC (Bring Your Own Cloud) architecture (and Databricks is the most common example). This comes with some of the best, and worst, of both worlds. I personally really enjoyed this blog post about BYOC from Jack Vanlightly, who works at Confluent. These hybrid solutions are very interesting, and can be used as a way to give more conservative customers a more "cloud-native" experience. At the same time, I've found that the types of discounts that large companies get with cloud providers are almost too good to be true. For instance, Snap famously had a five-year, $2 billion deal with Google Cloud. This greatly incentivizes them to only purchase software that can make use of the discounts they must be getting. However, there's more to be said about this topic and it probably warrants its own blog post.

And then of course, companies like SingleStore, where I work, or MongoDB and GitLab and others, have both a self-managed and a fully-managed offering. If you work at a company like this, you're going to be exposed to all of the different challenges, which is arguably even better than just being exposed to one set of them.

To finalize, building self-managed services and building fully-managed present interesting challenges to the engineering teams behind both. However, I would guess that it's more appealing for most engineers to work on fully-managed services. The increased rate of software delivery, and a larger canvas for innovation are much more enticing. While I am personally fortunate enough to work on both, I find that the freedom and control over more layers of the system in fully-managed services allows me to think of more ways to provide value to customers.

Feel free to reach out on Twitter!