Infrastructure Engineering

Book

Mon, 16 Jan 2023 07:00:00 -0700

Suggestions? Take a look at ‘Want to help?’ section on About.

This is the in-progress version of Infrastructure Engineering.

Mon, 16 Jan 2023 07:00:00 -0700

Hey folks! I’m Will Larson, sometimes known by Lethain, and this is the Infrastructure Engineering. Infrastructure software engineering impacts the professional lives of every software engineer deeply, and subtly shapes the products and platforms our companies build, but relatively little is written about running an effective infrastructure engineering organization.

Hopefully these interviews and guides will do a bit to help with that!

Tech Spec Review

Mon, 16 Jan 2023 07:00:00 -0700

As the organization starts to write more Technical Specifications, you’ll eventually want a forum to discuss the key decisions. At most companies, that meeting is the Tech Spec Review.

The Tech Spec Review is a forum to review feedback on new Tech Specs, resolve open points of discussion, and flag new context to be considered before finalizing the design. Secondarily, it’s a valuable forum for keeping the wider organization aware of new and upcoming technology changes.

Related tools

Tech Spec

Related meetings

Incident Review

Other approaches to Tech Spec Reviews

Goals

Drive consistent technical decision making. Much of the value from your technology strategy comes from its consistent application, and this meeting should support consistency. The review is a particularly valuable source of problems to inform your technology strategy
Role model good technical decision-making and discussion. Your organization will learn what good technical decision-making looks like from this meeting. Proactively coach folks giving feedback in both good (“keep doing that!”) and ineffective (“in the last meeting, …”) feedback
Prevent teams from pursuing local maxima in ways that are misaligned with the company. For example, a given project might benefit from introducing a new database, but the cost to the company to support business continuity, privacy auditing, and so on might outweight the project’s benefits
Avoid the Tech Spec Review anti-patterns. Don’t be a domineering review, bottlenecked review, status-oriented review, or an inert review. As a key forum for resolving technical disagreement, there are many ways for Tech Specs Reviews to fail. Avoiding these anti-patterns requires ongoing, proactive attention from the Tech Spec Review’s sponsoring leader

Agenda, Scheduling, and Scaling

The default approach here is to run them on a weekly cadence, sending out Tech Specs for discussion two days ahead of the meeting, requiring all attendees to read the specs before the meeting, and canceling meetings ahead of time when there are no specs to review.

That said, most organizations end up with a fairly custom approach to this meeting. When your organization is small, you can likely do on-demand reviews for each Tech Spec. This allows the team to get comfortable reviewing and being reviewed without the risk of “running over time” and preventing another spec from getting discussed.

As your organization grows, it will typically become hard to schedule all stakeholders into a on-demand meetings, and you’ll typically move into a standing meeting. Each standing meeting should discuss one to three reviews, depending on the size of open decisions. You can experiment a bit with format here: you might be able to review five specs in five minutes if it’s just a matter of approving unless there are any additional concerns to flag.

There are many ways to scale this meeting. Some organizations rely on asynchronous review for most specifications, and only bring “controversial” specs to the synchronous review. Some organizations hold multiple Tech Spec Reviews, sharded by area: one for Product Engineering, one for Infrastructure Engineering, and so on. Ultimately, I recommend actively experimenting with your approach based on the specific issues you’re running into with the meeting. There are general solutions, but each company uses this meeting in a somewhat different way, so adopting the standard solution may not work well for your needs.

Roles & Attendance

There are four key roles in the Tech Spec Review:

Facilitator who coordinates the agenda and the conversation. This is generally either a Staff Engineer, a Technical Program Manager, or a partnership of the two
Presenter who has written the Tech Spec being discussed
Notetaker who ensures notes from the discussion are captured
Attendees who share context, ask questions, and participate in the discussion. Some companies restrict attendance because too many folks attend and want to “demonstrate value” by asking questions, or unconstructively inject their personal preferances rather than prioritizing the organization’s perspective. Generally, I think it’s better to allow open attendance and give direct, firm feedback to those who attend unconstructively. If folks feel like they must attend to avoid bad decisions impacting their team, then you should probably consider creating more visibility into Tech Specs outside of this meeting, via either chat or email
Sponsor who provides organizational weight to the meeting through their participation, this is generally either the head of engineering, a Staff Engineer serving as the head of engineering’s right hand, or a manager reporting directly to the head of engineering

Is it working?

Some questions to ask when considering if your current Tech Spec Review is working:

Do you have Tech Specs coming in for review? If not, is it because the review isn’t useful? Is the review too intimidating? Are folks not sure how to submit new specs?
Are too many reviews coming, such that feedback is slowing down execution? Is there a set of category-wide decisions you could make that would reduce the need for certain kind of Tech Specs (e.g. auto-approve specs that use the common storage and compute tiers)?
Are reviews generally getting to the right decisions? Are the right concerns being raised, but getting rejected because the presenters don’t engage with feedback? Conversely, is it because the review lacks the necessary authority to succeed in your company?
Are discussions generally on topic? Do some participants routinely derail discussion? How could you prevent that pattern from reoccuring?
Do attendees enjoy attending?

Meetings

Mon, 16 Jan 2023 07:00:00 -0700

Incident Review

Thu, 05 Jan 2023 07:00:00 -0700

I’ve never heard of a company that has a business, that doesn’t also occasionally have things go wrong. Something going wrong might turn into a support ticket, an angry email, or an alert popping up on an on-call engineer’s phone. If there is user or business impact, and an engineer might need to respond, then it becomes an incident.

After the incident, the folks involved in mitigation write an Incident Review Template, and the that document is discussed in this meeting, the Incident Review.

Related Tools

Incident Review Template

Other Approaches to Incident Review

PagerDuty

Goals

Incident Reviews are a cultural carrier meeting for most engineering organizations. They are a rare meeting where you will see a wide mix of teams and seniority-levels arguing about something that the business cares about deeply: customer and employee impact. A well-run Incident Review helps new employees quickly understand how your culture works when things really matter.

An effective Incident Review facilitates these goals:

Foster and socialize learning about what caused an incident: incidents have a certain inherent rhythm, and the only way to change it is to ensure others are aware. The most valuable thing this meeting does is create awareness of what has actually happened in a given incident, which is the precursor to preventing a repeat
Surface missing context across teams and functions: customer success might mention an impact to users, an infrastructure engineering team might mention that the incident had a wider impact than initially recognized, a product engineering team might explain the business cost of delayed message processing
Inform investments on work that will best contribute to increased reliability: broaden an ongoing investment project to support a new edgecase, cancel a previous mitigation effort based on improved understanding of the underlying issue, recognize that similar issues are repeating without being successfully addressed

Anti-goals

Because of this cultural significance, Incident Reviews also have a predictable tendency to become ideological arenas, and to attract participants with ideological goals about the right way to foster reliability, run reviews, etc. Your goal as the senior leader who owns this meeting is to prevent it from becoming an open ideological discussion forum, and to instead focus it on the specific agenda at hand.

Several patterns to be wary of:

Ensuring adherance to documented process: some review meetings become focused on driving adherance to the specified incident response or review process. That is valuable work, but ineffective to conduct in a large, learning-oriented forum. Instead, drive adherance before the meeting
Pedantic or status-oriented: a surprising number of incident discussions end up orienting around policing correct nomenclature rather than encouraging learning and growth. Effective reviews are progress-oriented, with practioners who explain important context when additive, but don’t orient around policing correctness
Public performance of a one-person play: effective learning meetings don’t spend much time reading materials or reports out loud. The entire time should be devoted to discussion, perhaps with a short initial window for attendees to read the report. Learning is a group activity, wbhereas readouts as a solitary performance
Public performance of two-person play: some meetings adopt a consistent chorus across sessions. A certain set of questions, e.g. “How did you first become aware of this issue?”, will be asked and answered at each session, consuming much of the time. That feels useful, but it implicitly silences the wider group, who are not able to contribute their context and encourage group learning

Finally, like any important, large meeting, there may sometimes be individuals who are more focused on their personal ideological goals rather than the meeting’s goals, and it’s your responsibility to either anchor them on the meeting’s goals or get them out of the meeting so work can be done.

Agenda, Scheduling, and Scaling

The agenda for every incident review is discussion of one to two individual incidents or a cluster of related incidents. The agenda should be decided one to two days ahead of the review, and shared out with attendees to allow them to prepare. Because most learning occurs in discussion, I recommend against trying to include more than two incidents (or one batch of related incidents) in a given session.

Run these on a weekly cadence, canceling ahead of time when there are no incidents to review.

If you start to have backlog of incidents to review, then you have three options:

Batch related incidents if you have a cluster of incidents with shared contributing causes. For example, you might have a streak of incidents related to database instability caused by unindex queries, which would benefit from one curated, joint discussion rather than treating each as an independent incident
Extend review time for one week to have more incident review bandwidth. This works best when you have a short-term spike in incidents. Generally speaking, it is an organizatonal smell to permanently extend incident review beyond an hour a week for a large audience, as it’s an expensive investment of time
Stop discussing lower severity incidents in the review. For example, only discuss incidents with “significant” customer or internal impact, coupled with a simple definition of what incidents would fall beneath the line

Roles & Attendance

There are five key roles in an Incident Review:

Facilitator who coordinates the agenda and the conversation
Presenter who filled in the Incident Review Template for a given incident
Notetaker who ensures notes from the discussion are captured
Attendee who share context, ask questions, and learn from the discussion
Sponsor who provides organizational weight to the meeting through their participation, this is generally either the head of engineering or the head of infrastructure. It is reasonable for the Sponsor to occasionally miss, but I believe it’s essential for them to attend the majority of incident reviews

The Incident Reviews goals, particularly around learning and surfacing missing context, encourage a wide audience of attendees. I recommend allowing anyone to participate so long as they read–and abide by–the meeting’s goals and anti-goals. Ensuring folks act in accordance with the meeting’s goals is a joint responsibility of the Facilitator and the Sponsor.

Is it working?

Some questions to ask yourself if you’re unsure if your meeting is useful:

Are they getting scheduled? If that’s because you’re truly not having incidents, great! Conversely, if it’s because folks are not filling in the template, then dig into why not. Often these templates get overloaded with many questions to please many stakeholders, and consequently become difficult to use
Are key personnel attending? Particularly the sorts of folks who have important context to bring into the discussion. If the meeting is working, these should be an exceptionally high-leverage opportunity to grow the organization
Are the discussions resulting in a modified reliability strategy or roadmap? If these discussions are driving learning, then they should alter the shape of your roadmap
Do you enjoy attending?

Tools

Thu, 05 Jan 2023 07:00:00 -0700

Matthew Clarke

Wed, 11 May 2022 08:15:00 -0700

Interview in May, 2022. Learn more about Matthew on his blog, twitter, and linkedin.

Tell us a little about your current role: where do you work, your title and generally the sort of work you and your team do.

I work at Spotify as a Senior Backend Infrastructure Engineer. My team builds and maintains the tools that enable Spotify engineers to deploy safely and quickly whenever they need to.

We work a lot with Kubernetes, which Spotify uses to deploy and manage most of its websites and backend services. Spotify runs some of the largest multi-tenant Google Kubernetes Engine (GKE) workloads in the world, so this is a large responsibility.

My team builds tools on top of Kubernetes to simplify and create a great developer experience. These tools involve developing and maintaining our deployment tools, aggregating error messages from different Kubernetes resources and displaying them through Backstage (our internal developer portal), supporting developers on Slack with questions they have or problems they’re running into with Kubernetes, and working on our Kubernetes plugin for open source Backstage.

How did you start doing infrastructure engineering work? How have the companies you joined, your location, or your education impacted your path?

I actually started as a software engineer focused on e-commerce; while there are a lot of interesting problems to solve in this space, I found the lack of direct interaction with end-users frustrating. People don’t really care how “well” their online payment is accepted as long as it goes through, so you don’t get valuable feedback often.

My role at the Financial Times was my first real taste of infrastructure engineering. It was a DevOps microservice role focused on identity and e-commerce. My team was responsible for provisioning cloud resources, writing applications, deploying them and monitoring them. There I learned a lot about AWS, Kubernetes, and Cassandra. We used lots of different languages so that we could experiment with what worked for us, including Python, Java, Scala, Node, Go and Elixir, but we mainly settled on Java and Go.

However, throughout all of my roles, I found I gravitated towards building developer tools. Whether that was integrating two different build platforms at Cybersource/Visa, adopting Kubernetes at the Financial Times or changing to my current team at Spotify. One of the great things about infrastructure engineering is that you are sitting beside your users everyday, they’re your colleagues and you get to help make their life easier and get instant feedback about what they like and don’t like.

I have always wanted to have a big impact at the companies that I have worked at, and there is no better way to have an impact than to help increase the productivity of all the other developers at the company. This is also why I love to contribute to open source. By contributing to open source, you can make an impact not just at your company but throughout the whole industry.

What dashboards and metrics do you personally use to stay aware of your software and team’s work?

I use Backstage a lot to keep track of the current state of my team’s services. Backstage provides integrations with monitoring, deployment, CI and tech docs all in one place.

Other than that, I keep track of the various deployment features we provide, such as test environments and automated canary analysis, to get a good idea of what features users find useful.

Recently we have been making the effort to try to quantify and visualize deployment toil, so that we can see if we are moving things in the right direction with our platform offerings.

What would happen over next month if your infra org were all pulled away onto a secret project and couldn’t do their day to day efforts? Where would things slow down?

I think most things would continue along but probably not very efficiently, trending downwards.

We help developers at Spotify every day by answering their infrastructure questions, helping them get their services set up or debugging production issues, so there would be a lot of unanswered slack conversations! We are also continually scaling our systems out behind the scenes to continue to support an ever-growing number of users and artists.

Infrastructure engineering organizations have a lot of priorities. A few years ago I tried to define an overarching set of infrastructure priorities and came up with: security, reliability, usability, leverage, cost and latency. Of course, folks immediately started arguing I’d defined the scope too narrowly. How do you figure out what to prioritize working on?

This is a very interesting question, we get a lot of feature requests and feedback, but it is impossible to do everything. We try to focus our time on work that will have a wide impact, usually defining this on how much “toil” we can prevent. Toil for us is usesrs making infrastructure changes or tweaks that should be automated or happen behind the scenes without their interaction. An example of this would be our effort to automate migrations, make it clear to users the goal, and provide the tools to perform a migration with as small overhead as possible.

Related to priorities, one topic that I’ve had come up a few times recently is the idea of “Shadow IT”, where other organizations bootstrap an infrastructure project without your knowledge, and then ask you to take over running it once it becomes a burden. How do you deal with other teams asking infrastructure to take over their projects once they’re no longer fun (or often when the original implementer leaves the company)?

Something my team has been struggling with recently is the sheer number of systems and tools we own. Some of these might have been transferred to us like you mention above, but we give the benefit of the doubt and assume it was the best decision the implementor could have made given the information they had at the time.

Still you can’t support a limitless amount of systems and tools. Therefore the questions my team ask are:

Do we really need this tool? / What value is it bringing?
Are we the best people to be supporting this tool? / Instead of supporting this tool could we be doing something more important?

If we can’t justify the tool existing then it is a good candidate for deprecation. If it is valuable but we aren’t the best people to support it or could be working on something more important then perhaps we need to find a new owner, either another team internally or a managed version of the tool.

What’s the single most impactful project you’ve heard of an infra engineering org doing? Why? Was it obviously impactful beforehand?

I would say Backstage fits the mold for this. Before Backstage, different infrastructure teams at Spotify would create their own user interfaces, this work was not very efficient. Engineers had folders full of bookmarks and infrastructure engineers would toil away solving problems that other teams already had solutions for.

When Backstage came along the benefit was clear: developers had one portal for all their infrastructure needs, they could search Backstage for docs, datasets, teams, services and runbooks. Infrastructure engineers could embed their interfaces in Backstage and benefit from the large library of utilities and React components the Backstage maintainers had created for common use.

This lightened the load for all the engineers at the company and ultimately improved developer productivity, which is the ultimate goal of an infrastructure engineer.

Your current work focuses heavily on Kubernetes. This is a technology that has an outsized impact on the technology industry, and over the last six years has grown from something perceived as a toy into something widely used at scale. Where do you see the future of Kubernetes going?

Kubernetes is an open-source success story. It is great to see the industry rally around it as a project, including building incredible tools on top of it. Initially, it seemed like the only benefit was container orchestration. However, now we can see the additional benefits of extensibility, which has pushed Kubernetes beyond just containers.

In the future, I’m excited to see where the community goes with handling multiple clusters and whether some patterns emerge there. I also think there will be an emerging trend of workload clusters vs infrastructure-as-code clusters; some Kubernetes clusters will be used to manage your infrastructure through tools like Crossplane, and others will be where your services run.

I also hope we continue to see Kubernetes tools evolve to address the needs of service owners who have services running inside multi-tenant clusters and not just the administrators of the clusters.

Ok, excluding Kubernetes, but are there other technologies or tools that you see advancing the field in a similar way? What about technologies or tools, other than Kubernetes, that you believe will meaningfully advance the field over the next decade?

While I am a contributor, I do think Backstage has the potential to change how developers interact with their infrastructure and allows them to better focus on their code. Backstage has grown from an internal tool at Spotify to an open source CNCF Incubating project with hundreds of adopters and contributors, dozens of tool integrations and several commercial ventures using it as the basis of their products. The ability for a developer to have a single view of the entire software ecosystem at their company, including monitoring, docs, CI/CD and runtime, has been incredibly valuable at Spotify, and I think other organizations are discovering this too.

I am also very excited about eBPF; quite a few different tools are emerging that could enable language-agnostic service-mesh-like features in a microservice environment built on top of it. I like the idea of a service mesh that doesn’t require a sidecar proxy, which has latency and cost overheads. However, I think it still has a pretty steep hill to climb to rival some of the proxy-based service meshes out there.

What are some resources (books, blogs, people, etc) you’ve learned from? Who are your role models in the field?

I learned a lot from Sarah Wells when we were at the Financial Times; we embarked on a Kubernetes migration fairly ahead of the curve; Sarah gave a great talk on our migration (which is probably why it has been on the Kubernetes homepage for four years now!).

I love to read; some of my recent highlights have been: Network Programming with Go by Adam Woodbeck, Effective Python by Brett Slatkin, A Philosophy of Software Design by John Ousterhou and, of course, Staff Engineer by Will Larson.

I follow quite a few blogs, but the most valuable personally has been Last Week in Kubernetes Development. It can be tough to follow the current development of the Kubernetes codebase as it is such a moving target; this blog summarizes the interesting: PRs, merges, deprecations and news which makes that task a bit easier.

Manager

Wed, 11 May 2022 08:15:00 -0700

Interview

Wed, 11 May 2022 08:15:00 -0700

Suggestions? Take a look at ‘Want to help?’ section on About.

Folks who have shared their infrastructure engineering wisdom:

Mahdi Yusuf

Tue, 03 May 2022 14:00:00 -0700

Written interview in May, 2022. Learn more about Mahdi on his website, linkedin, and his StaffEng podcast interview.

Tell us a little about your current role: where do you work, your title and generally the sort of work you and your team do.

I am currently a Senior Staff Engineer at 1Password, leading the Server Architecture team. We are implicated in our systems’ overall design while pushing for the modernization of legacy systems.

The work encompasses everything from our overall system reliability and a few core components like queues, workers, and data stores. We also spend a decent chunk of time maintaining foundational libraries and service scaffolds that are used throughout the company.

Generally, this includes most of the non-product engineering work.

What dashboards, metrics, and forums do you personally use to stay aware of your organization? Is there a different answer that you would be more proud of? What’s preventing you from that answer being the current answer?

Currently, we are a Datadog shop for our dashboard and metrics. We now use collaborative Datadog notebooks when discussing/investigating new initiatives. We also use Kibana for logging and Bugsnag for error tracking.

I would like to see something that could cut across all those three places to get a real sense of what is happening entirely across our system. Without having to jump from platform to platform. One tool to rule them all. The more data sources you can synthesize, the better your understanding of your system can be.

I have been an avid user of Grafana, which delivers on the premise above. It integrates metrics, logs, and traces all in one clean interface.

There were various considerations around sticking with Datadog. In addition to the cost of moving, there was the idea of who would keep this running. I am happy to see there is a managed Grafana being offered by Amazon now. So we may revisit this, when we have more time.

What would happen over next month if 1Password’s infrastructure org were all pulled away onto a secret project and couldn’t do their day to day efforts? Would the company still run?

Depending on when you ask that, it can vary. But, honestly, as much as I would like to say, things would grind to a halt. It’s a constant effort, but we are always trying to make sure no team is in a position to bring things to a complete halt.

If the infrastructure organization were utterly gone, progress on tasks that have payoff farther in the future would lag behind the rest of the organization’s efforts, eventually impacting the broader organization.

The way I like to look at this work is as necessary investments we need to make today for the future progress of the entire engineering organization. So it’s a constant trade-off with many factors that come into play.

I have never seen this effectively work without dedicated teams focused on issues in the production systems. A new product feature usually trumps fixing something that isn’t a problem…yet. How long and at what speed the company would still run are probably more pertinent questions.

This is something I have been thinking about lately. If I was pressed to really get into my gut and define these prioritizations, it would be tricky but let me try here. Frameworks are great general guidelines when you don’t have context. Still, most of these decisions depend on the organization’s willingness to make said priorities happen and stick with them to see them through to completion.

That being given, I primarily focus on desired outcomes and slowly put problems behind us. Some of these classes of issues come back in various forms (see: scaling and migrations).

Also, knowing you can’t solve them all quickly, let’s get to the actual job of prioritization.

The first thing you need to identify is the severity of these problems. There are classes of problems that you can live with and others that, if left alone, will only get worse if they aren’t given the attention they need. The problems in the latter group aren’t usually a problem today, but being left alone can be limiting in some way in the future.

Keeping the organization as agile as possible is essential in this regard. I might be conservative, but I always pay off the compounding debt first. Software systems change, but teams always build on top of what is there today.

If a problem has more or less the same impact on the organization six months from now as it does today, it goes down my list of importance. However, suppose it gets worse as time goes on, the higher on my list of importance. This is when compounding is working against you.

Now let’s talk about when compounding is working with you. If I fix something that makes each of my engineers lose an hour a week–just one hour. If I eliminate that, I just saved the company 200 hours a week and reduced toil in the process. These classes of problems aren’t the ones that usually get worse with time; these are typically focused on developer velocity and usability.

So there you go, another framework.

You can always say no. Use this one sparingly often, you will eventually be working with that team in the future, and the road to fame and riches is long.

Frequently these systems are necessary but not in active development. It usually isn’t that bad if you can have some time for hand-off and transition it slowly. Documentation here can be worth its weight in gold. Knowing where the bodies are buried is helpful when things eventually go wrong.

There are always teams that get overburdened with these services with no owners. The burden is much like peanut butter: it’s better when spread around. There is always a team that is the best fit for said service.

Like Spike Lee said, “Do the Right Thing.” If the team is overburdened, you can always assign more headcount to the team.

I will say that leaving these services without clear ownership is a poison pill for your organization. People will shirk responsibility, and zero effort will be put towards these services, sometimes out of mere spite. It is better to assign the service to a team that won’t prioritize than to give it to no one.

At times I have run into a belief that infrastructure necessarily conflicts with productivity: e.g. we have to reduce productivity to increase reliability. Have you seen a tension between infrastructure and product engineering productivity? Are there ways to reduce that tension?

Absolutely! Measuring it is something you should try to be doing. For example, can you measure how long it takes merge requests to get through review? How long are RFCs in the review state? How many regressions are we seeing after deploying a new piece of infrastructure?

These things can worsen if infrastructure engineering is too prescriptive without understanding the underlying product work. Embedding infrastructure engineers into product teams can help here. But, again, it’s mostly balancing priorities/perspectives and communicating clearly.

The benefits of embedding can be twofold and can help infrastructure engineers get a first-hand experience of what is slowing down product engineers. They can take that back to the team to improve things, and product engineers can get some visibility on how these processes improve reliability in production.

I believe in supporting product engineers to deal with (read: empower to resolve) most of the issues their code causes in production and support them if they need help. But unfortunately, overzealous product engineers create debt faster than they develop products.

Most of this tension usually comes from not getting feedback in the correct stages if you cannot embed engineers into product teams. Writing design specifications can be fantastic and let’s most of the discussion occur before the rubber meets the road.

Ideas are quickly redrawn, maybe even code, architecture, and infrastructure, not so effortlessly. Where would you want to give constructive feedback?

There’s a tendency for infrastructure engineering to be invisible when nothing is going wrong. How do you articulate the value of your organization’s work?

This is true, but data shall set you free. So it’s essential to capture why you are doing something and what you think it will improve. Then follow up with either data or people you impacted with that change.

It’s all about outcomes. If you can’t track those with either data or teams you impacted positively through your efforts, you should probably rethink them. If you are doing this effectively it shouldn’t be too hard to articulate. You are often left to synthesize where engineering has under invested and figure out if anyone cares.

It’s important to understand that as software systems grow and more people start working on them, they become more complex. Unfortunately, you can look at these like a thousand cuts over time, so they are easy to miss and overlook.

Making sure you don’t succumb to these changes is essential. But unfortunately, I am sure most infrastructure engineers have been in the position where something they wanted to work on was minimized and deprioritized to have things quickly change when things go splat. Understanding risks and tying those to straightforward trade-offs is vital to communicating with leadership.

What are some resources (books, blogs, people, etc) you’ve learned from? Who are your role models in the field?

I have found Twitter in general to be a great resource throughout my career. I have met tons of people and learned so much. I often recommend LeadDev to new leads because they have outstanding resources. I am also a big fan of Neal Ford’s works around software architecture. I am also working on something new here called architecturenotes.co where we breakdown system design with the people that built them. I think this audience would get a kick out of it.

Shawn Wang / swyx

Mon, 11 Apr 2022 08:00:00 -0700

Interview occurred in February, 2022. Read more from Shawn on his blog, twitter, and his book, The Coding Career Handbook.

Tell us a little about your current role: where do you work, your title and generally the sort of work you and your team do.

I’m currently Head of Developer Experience at Temporal.io, an open source workflow engine for long running, durable processes powering companies as small as 2-person YCombinator startups, to enterprises as large as Stripe, Snap, Datadog, Netflix, Doordash, etc. We are generally responsible for improving the experience of “front line” individual contributor developers, covering their end to end journey from first contact (DevRel) to learning (Docs) to API Design (SDKs) to ecosystem (Community).

The basic insight is that companies ship their org charts (Conway’s law), but developers don’t care what team shipped which when they go through your product, so it makes sense to have someone whose job it is to coordinate and build out developer-facing efforts cohesively.

In Developer Exception Engineering, you wrote a bit about the slipperiness of defining “developer experience,” and how it often varies significantly across companies. How would you explain developer experience to someone unfamiliar with the role?

Developer Experience (DX) is a buzzword at this point, so naturally everyone is co-opting it to represent their particular view of the world, making it extra confusing for anyone that just wants a straight answer. But I’ll give you my best shot here.

At the highest level, the basic dichotomy to be aware of is “internal” vs “external” DX. Most people who come up within one of these two branches may be completely unaware of the other, which contributes to the confusion when people discuss “DX”.

Internal DX teams focus on developer productivity within a company (sometimes called “dev infra”). The math is simple - if you have 50 engineers, and you think it’s possible to improve their productivity by >1% a quarter, then you would be silly not to invest in 1-2 engineers who don’t work on product, but just focus on making everyone else more productive. Scale this up to a 1000 engineer company and you now have a whole Internal DX org to play with. They can span a wide range of deliverables, from build/test automation to dev environment to code quality. The clearest mental model for identifying Internal DX opportunities I’ve come across is Netflix’s Productivity Engineering team, which is responsible for three major components - from new hire to productive local dev (their Bootcamp and bootstrapping tool, NEWT), from local dev to production (their “build-bake-deploy paved road”), and then from production back to dev (their observability tools like Atlas). The other popular taxonomy to work with are the four Accelerate metrics. Both of these approaches essentially divvy up the software development lifecycle into meaningful chunks, which can then be independently and tangibly improved by internal DX teams.

External DX teams focus on improving developer adoption/mindshare/productivity at other companies. Where almost any software company can have Internal DX, it only really makes sense to have External DX if you make something for developers. This means it is a natural fit for devtools companies, but you might be surprised at what companies invest in this. Are Spotify, Notion and Slack devtools companies? No… but they all offer APIs for developers! So they all have DevRel teams. The distinction between DevRel (also known as Dev Advocacy) and DX is another common question. On one hand, traditional DevRel is very heavy on content creation (blogging and speaking, basically, but also demos and workshops), whereas DX has more of a mandate to write (non-core) code and docs to solve problems. I first transitioned from DevRel to DX at Netlify, where eventually it formally covered Advocacy, Integrations, and Documentation. The exact coverage will naturally differ based on the product - for example, Netlify is a closed source SaaS platform, so Advocacy plays a bigger role, whereas Temporal is an open source client-server system, where equal love needs to be given to Community and Docs.

A quick aside for those who often hear DevRel vs DX conflated: DX is supposed to be the superset, but frankly, the lion’s share of DX is still DevRel, for both economical and historical reasons. Economical, because most developers know how to build product, but are terrible at building distribution, so a DX team often contributes the most value by reaching developers despite all the other things on its plate. Historical, because the DevRel to DX transition is a once-in-a-lifetime career upgrade for Dev Advocates to have more impact, just like the Sysadmin to DevOps transition. It all makes sense once you consider that Dev Advocates speak the most to users, but usually have the least power to make fundamental changes to solve their pain, particularly those I term “Developer Exceptions” in that blogpost. Blogposts and talks have a half-life far shorter than docs and tooling/product improvements.

Once you’ve marinated in the various aspects of DX enough, the distinctions start to re-blur once you consider that Internal DX just serves internal customers (and needs to invest in docs and advocacy too), and External DX serves external ones (and needs to tangibly improve productivity too). Both roles require a great deal of empathy with developer problems, and an expansive mental catalog of ways to solve them. Yet the ultimate relevance of either to the outside world matters only to the extent of a typical build-vs-buy decision. Don’t get too hung up on precise definitions in an inherently fuzzy and still-moving field.

When we first discussed this interview, you asked if your experience would be interesting to folks focused on infrastructure engineering. I’ve increasingly come to believe that Developer Experience is a core competency for all folks developing infrastructure software or working on infrastructure projects like large-scale migrations. Should infrastructure teams consider Developer Experience as a core engineering competency? Any ideas why they often don’t?

It’s funny, even though I do DX at a company that serves Infra engineers now (and Temporal enables Infra engineers to offer a dramatically better developer experience to product teams by providing “reliability on rails”), I had never ever viewed it as something Infra engineers themselves should regard as core. For sure, the Derisk-Enable-Finish cycle in that article on Migrations leans on many of the same skills as DX teams - advocacy, docs, tooling. But I’m loath to recommend that it should be “core” in all contexts, because (as we discussed earlier) DX is so broad and hard to define, and I’m always skeptical of people hawking their pet topic as mandatory. A bloated definition of “core” defeats the purpose of defining a “core”.

What I will say is that I think most Infra Engineers could do with more developer empathy, which in most situations simply means putting themselves in the shoes of people with less context and knowledge than them and proactively helping them out by any means necessary. If you do it right, then yes, the developer experience of your users will be better because you took the effort, but it should be done not for altruistic “let’s make them happy” reasons, but rather, selfish ones: your efforts will be more successful if they feel more successful.

Why don’t more infra teams invest in Developer Experience? Honestly, probably because there’s no cultural expectation for them to. It’s common for infrastructure teams to get consumed by the loudest issues surrounding them like incidents and infrastructure costs such that they end up much more focused on their obligations to computers than their obligations to other engineers.

What are the top three tools or techniques that you use in Developer Experience that infrastructure engineering teams should consider adopting?

Journey Mapping: exhaustively enumerate every concept, system, or API capability your user should know (thereby letting people know what they don’t know). Pick 2 main axes of concern and map them out in 2D space - clustering related concepts together. Draw a small core of “must know” concepts where everyone should start (letting people know what they don’t need to know). Identify and highlight FAQs. Then let them find their way based on their needs. This contrasts with a “one size fits all” linear path. (see example)
Pitch Sizing: Be prepared to explain/define your system in one sentence (pique interest), one paragraph (by desired requirements or by pitching the problem), a 10 minute presentation, or a 30 minute demo. Logical/technical arguments are best supplemented with Cialdini persuasion principles. Practice this when you don’t yet need it (eg at internal demo day/lunch) because you will be called upon to do it at the most unexpected times for the most high leverage reasons.
Two phase commit: Knowledge is transferred as both discrete particles and continuous waves. Concretely, some of your users will want a monolithic organized reference, and others will just want diffs. One example rule that implements a “two phase commit” of knowledge - Every feature update should be communicated via a changelog and a doc/wiki update (and, for more impactful updates, a tweet, slack message, blogpost, talk…).
(Bonus) Events: Learning to throw events that people look forward to and enjoy participating in is a huge multiplier on existing DX efforts. (see Community Annealing)

Infrastructure engineering organizations have a lot of priorities. A few years ago I tried to define an overarching set of infrastructure priorities and came up with: security, reliability, usability, leverage, cost and latency. I imagine this is at least equally true for DX teams, how do you figure out what to work on given the wide range you could prioritize?

I think about DX work in terms of concentric circles radiating out from the core product, matching the maturity of the product:

When the product is still being shaped, there is no better time to give feedback on API design.
As the product approaches fully baked, I shift my attention to Docs.
After shipping the product with a complete set of docs, I shift to Content (Advocacy) to get users and to spell out and elaborate whatever doesn’t tonally or structurally fit in docs.
Users come for the content, and stay for the Community, so I start investing in getting to know them, helping them in their adoption, and find/build for/hire each other.

On and on pushing outward when we can, but looping back inward whenever a new feature or product is launched or a new problem is found.

All of these efforts should be coordinated with the same “map” I described above - shared terminology, shared understanding of core concepts, and a shared reality of neighborhoods and landmarks. However they are not equal in all contexts, because inner circles tend to have higher long term impact (the best docs are the docs I don’t have to read because the product teaches me as I go, the best blogposts are the blogposts I don’t have to look for because the documentation was good enough, etc.), but outer circles have more reach.

What I’ve described is from my experience in my sweet spot at early stage, Series A-C devtool startups, where each program is usually a singleton, but there are advanced versions of this at the larger companies too:

Every SDK can always have more languages and devtooling.
Every conference can be replicated across the major continents.
Every docs effort eventually morphs into a “University” or a certification/education program.

At AWS scale, we also layered the DX circles with language, geographical, and business vertical dimensions. If you wanted a Chinese speaking Telcos specialist in Australia, we had someone for that.

Folks working on infrastructure engineering often have a specific dashboard they look at every morning to get a sense of how the software, system, and organization is operating. Do you have a similar dashboard? What’s on it?

We use a mix of internal BI tools for lagging indicators (active clusters, SDK version adoption) and Common Room for leading indicators (open source and community activity). As long as everything is trending up on a trailing 2-3 month basis I’m not too concerned about checking it every day. Considering that it takes >10 touches for the average person to go from first contact to seriously interested, the natural frequency of consideration cycles make for extremely long feedback loops.

This is further confounded by the extremely non-ergodic nature of the open source enterprise customer, where one large customer can be worth 5 orders of magnitude more than the median, and take anywhere from a month to two years to convert to a customer.

Most DX metrics are better regarded as a health check that things aren’t broken, rather than proof positive that things are actually working well. If you need more specifics, I’ve received very positive feedback on my piece on Measuring Developer Relations.

OK, I’m going to start turning the conversation towards Temporal for a bit. In every infrastructure team I’ve worked on there’s a team focused on supporting services that offer an API, but it’s often only much later that there’s any support for workflows (by which I mean scheduled, periodic or event-driven tasks) outside of something like Airflow for batch processing. How did Temporal decide to focus on a workflow engine?

There was no decision so much as it was a lifelong obsession borne out of decades of distributed systems experience at scale, and solving the same problems over and over again. My basic insight is that everyone converges on the same requirements for reliability, observability, and scalability in their systems, but the tools we have are too low level, so everyone handrolls (poorly) their own distributed system out of these tools. Eventually, large-enough companies build their own workflow engines to slow the wheel-reinvention.

Our cofounders had been working on various iterations of messaging services and workflow engines for the prior ~20 years, at AWS, Google, Microsoft, and finally Uber. They created Temporal’s precursor at Uber, which became a full-time job as the number of applications using it ballooned to 300 in 3 years. This work was open sourced and similar growth was seen at Hashicorp, Coinbase, Airbnb, Doordash, etc. Finally, demand for a hosted solution was so strong, and the Uber-specific tech debt was so high, that they forked the project to start Temporal. So at every step of the journey the market demand drove the next phase of adoption, rather than any one decision.

Temporal is at once a 2 year old startup and a 20 year old team in this sense; and having that much big tech and open source validation gave us a lot of conviction that the industry is hugely underappreciating the use cases of a workflow engine beyond simple scheduled jobs. There’s a bunch of hypey hyperbole thrown around: “distributed system in a box”, “reliability on rails”, “distributed application state platform”, “a new computing primitive”, “service mesh for long running operations” - all of which are true depending on your point of view.

In listening to Stripe’s talk on Temporal and Netflix’s talk on Temporal, both mention writing their own SDK wrapper on Temporal’s SDK. Is it a good or bad sign when your users routinely wrap your SDK?

It’s easy to map out the pros and cons:

It’s good in that it validates that we solve a hard enough problem that people wrap us rather than build us… for now. And it gets us users that closed source SaaS and inextensible “No Code” platforms would not.
It’s bad in that it means our users have a built in facade that makes it easier for them to move off us in the future
It’s good in that both Stripe and Netflix talk about their wrappers solving company specific problems and providing good defaults for their intended users, that we can later absorb into core once validated enough in “userland”
It’s bad in that we don’t do some things for them out of the box… yet.

Ultimately I view “being wrapped” as a natural, net positive outcome of any valuable enough devtool. The best thinking I’ve come across on this is Kevin Kwok’s view of platforms vs their ecosystems:

Usecases that are high impact and generally useful should be solved by us, whereas usecases that are lower impact and very specific should be solved by wrappers. We would look to our growing ecosystem to help solve high impact, high niche usecases, and investing in an open source community directly contributes towards this long term advantage.

Since Stripe moved to base generating SDKs off their OpenAPI spec, I’ve started to suspect that SDKs are better interfaces to expose to users than APIs themselves. Are SDKs better user-facing interfaces than APIs?

This is very close to my heart! The simple response is that yes, if you can afford to, offering an SDK (or a CLI, by the way) generally provides a better developer experience than just the raw API. The basic argument is that if you don’t provide an SDK, the developer will eventually have to build one for themselves for anything of sufficient complexity. There are a number of problems that can only be solved at the SDK level, including providing more specific types or type inference, inline documentation/autocomplete, and mocking out the API for testing.

However a poorly implemented SDK can also introduce an extra layer of potential bugs and performance issues, constrain advanced users, cause uncertainty about exposed classes and data structures, and add complexity to versioning/upgrades. In scenarios like these, being able to “drop down” a layer to the underlying API is crucial and the platform should not actively obstruct that.

One should also distinguish between “Fat” and “Thin” SDKs. “Thin” SDKs are simple, 1:1 language wrappers over APIs, the kind that can be generated from OpenAPI. “Fat” SDKs do more, often managing state (e.g. AWS’s AppSync SDK creates a local replica of your DynamoDB backed database, and handles offline sync and merge conflicts), or allowing plugins, or as Temporal’s SDK does, offering a deterministic sandbox which can replay events through your code for failure recovery and durable async functions.

In short, the opportunities for “Fat” SDKs to improve developer experience well beyond simple RESTful APIs are greater, at the cost of more engineering (and docs, and support…) to maintain them. Tradeoffs everywhere!

In The Self Provisioning Runtime, you modify Alfred North Whitehead’s quote saying, “Developer Experience advances by extending the number of important problems our code handles without thinking of them.” That quote gets at the long-term promise of cloud providers, which are slowly making important problems invisible for many users, e.g. my experience is that general awareness of networking is significantly lower than it was a decade ago, which I attribute largely to cloud adoption. In some ways I see Temporal as competing with cloud providers’ own workflow engines. How do you think about competing with the cloud?

I’m excited by it. We certainly have to take it seriously, because Temporal is MIT-licensed, and there is nothing stopping Amazon or Azure from hosting us as a service tomorrow. But at this point dozens of open source companies have faced that threat and survived - by relicensing, and by serving their customers better than the big clouds can. On one hand, this is intimidating, because Amazon theoretically has infinitely more resources to crush us. On the other - I’ve worked at Amazon and seen how hard it is to push through the absolute mountain of conflicting priorities and legacy tech to get anything done compared to tiny startup teams with a fraction of our funding.

This is why topics like developer experience are so important - there are so many more dimensions to building a successful developer infra business than just the commodity operation of software - but I am actually most excited about outcompeting the big clouds by better product strategy and better network effects as those are sustainable and compoundable wins.

I can’t be too specific here but consider how Snowflake made an independent case for itself by being the “Data Cloud”, Cloudflare is doing the same for the decentralized cloud, and Stripe being payment rails for ecommerce. All are justifiably market positions that the big clouds will not/can not tackle given their current strategies. Temporal happens to occupy a very nice space:

managing lightweight application state, not egress-heavy data
having a small well defined contract with every mission-critical microservice in your company and others’, and
being generally agnostic as to whether humans or machines complete tasks in a given workflow.

I think every startup that competes with big clouds (read: every ambitious Infra startup) will need to carve out a space on which they are the undisputed independent source of truth, at least until the $100billion valuation stage when the metagame changes once more.

What’s the single most impactful project you’ve heard of an infrastructure engineering team working on? Why? Was it obviously impactful beforehand? Stripe’s Sorbet is an example of a discrete project that I found surprisingly impactful.

Probably the sharding system that became Vitess at YouTube. YouTube is quite simply the biggest social video platform on Earth today, but it faced a horde of well funded competitors in 2005-2010. YouTube was experiencing 2 outages a day due to the extreme load, and could have gone the way of Friendster if those performance issues continued. No Vitess, no YouTube.

Vitess made MySQL scalable for YouTube, then its open sourcing helped Hubspot, Slack, Pinterest, GitHub, Square and more. If database infra counts, then I’d be hard pressed to think of a more impactful project.

What are some resources (books, blogs, people, etc) you’ve learned from? Who are your role models in the field?

I keep a list of resources for DevTools and Dev Rel/Dev Community in my own space because the list of resources is long for such a young field. Special shoutouts to Beyang’s Guide to Devtools and Mary Thengvall’s Devrel Resources, and to Kelsey Hightower for getting me started Learning in Public and going down the Developer Advocate career path. Scott Hanselman is also a huge mentor to me, being an early reviewer of my book and with his inclusivity and ability to make anything in the Microsoft ecosystem interesting, and ability to cross over into newer platforms like Tiktok!

Developer-Experience

Mon, 11 Apr 2022 08:00:00 -0700

Smruti Patel

Sun, 10 Apr 2022 07:00:00 -0700

Written interview in early February, 2022. Learn more about Smruti on linkedin, twitter and on LeadDev.

Tell us a little about your current role: where do you work, your title and generally the sort of work you and your team do.

I currently lead the Data Platform group at Stripe – we operate the centralized data lake, and the bigdata, async & stream processing infrastructure for Stripe’s mission-critical business, while ensuring security, reliability and efficiency. Essentially, supporting Stripe’s core money movement & storage, financial reporting & analytics products for our merchants, and empowering ML infra to build credit, fraud & risk intelligence.

Prior to my current role, I also led the LEAP organization, which stands for Latency, Efficiency, Access & Attribution and Performance - my vision here was to take those small steps needed to unlock the giant leaps for both our engineering organization internally, and our users using Stripe. To enable that, we developed cross-functional strategies and tools for optimizing our cloud spend, and lowering Stripe’s end-to-end latency through performance tuning.

What dashboards or metrics do you personally use to stay aware of your organization’s work? How often do you check these?

I am one of those personality types, who is facts-oriented, analytical, and leverages data to draw patterns and drive decisions. So yes, metrics are my jam!

I’ve been leading teams for over a decade now. In this period, I’ve learnt that engineers don’t lack motivation. They are here to do their best work. Intrinsic motivation talks about a healthy balance of autonomy, competence and purpose. Let’s assume you have solved the hiring problem. You’ve built an inclusive team of highly skilled engineers with the right domain expertise. We’ll also assume that your management and leadership practices lean toward a healthy culture. A culture which provides the right blend of growth mindset, radical candor and psychological safety for individuals to thrive. So competence and autonomy are more or less solved, but how do we as leaders, then address purpose, the northstar, the why?

That’s where it’s important to think about opportunity cost! We have finite resources: and doing X implies not doing Y. For any software-driven company, our engineering talent, their productivity, efficiency and impact is our highest leverage. Hitting the right product-market fit can be extremely time sensitive. The opportunity cost, therefore, of going down a potentially wrong path, can be significantly high.

And so, you need a high fidelity OODA loop to observe, orient, decide, act and react to feedback! And that’s where I leverage metrics heavily to measure and debug engineering velocity:

Precision - What are you shipping and why?
Speed - How frequently are you able to ship?
Quality - What is the failure rate or quality of your software?
Impact - How well does it achieve business goals?

For LEAP: our impact metrics were around measuring overall cloud spend as a function of business volume. Or the tail latency - the p99.9– of the most important ChargePath API.

For Data Platform: some aspects are easier to measure than others. So here, we have 3 categories - starting from the outer loop of Stripe users, to the inner loop of our direct engineering cohorts and the bridge between the 2 - our executive leadership:

Non-functional requirements to measure strong guarantees of security, reliability, and performance of our systems.
Functional requirements to democratize access to data to enable rich insights for various cohorts that work with data - data scientists, data engineers, ML engineers or Product engineers. This is generally the hardest to measure!
The efficacy of operating Stripe’s business through data efficiency, compliance, and rigorous financial accounting.

I personally look at most of these system & business metrics weekly, to determine overall health of our systems and the broader investment within the organization.

In addition to these, I also look at team health metrics (monthly & quarterly) - like employee engagement, hiring ratios, attrition or transfers, #uplevel readiness.

Several of the areas you’ve worked on, especially efficiency (e.g. infrastructure spend) and performance (e.g. CPU utilization and user-facing latency) are areas of distributed accountability. A system’s efficiency is heavily dependent on the individual parts within the system. How do you set goals for areas of distributed accountability? What have you found effective for reducing the challenges of diffused accountability?

I love this question, and especially your reference to Thinking in Systems, a book which blew my mind a decade ago. Here’s how I’ve come to approach these problems.

Frame the problem. The why

For both Efficiency & user-facing latency, the first thing I did was own the whole problem, from the farm to the table- the reason this was a key, fundamental step was it helped provide a unified direction and sense of purpose for the narrative. I established myself as the accountable and authoritative subject matter expert in framing the problem for the company, through trust and verifiable, clean data.

I had learned from past experience that accountability without authority was a kitchen sink at best and a dull knife at its worst. So I secured executive sponsorship to back this key impactful initiative for Stripe, aligning on the outcomes through charter metrics (eg: overall cloud spend as a function of the business and p99.9 latency of the most popular ChargePath API), and setting expectations on the relative agency of a centralized team in driving those outcomes.

While this was necessary, it was far from sufficient. And that brings us to identifying the key elements of this system - the movers & the shakers.

Identify the elements. The what / who

In order to determine whom to hold accountable, we had to invest a few quarters in doing the gardening - creating a M.E.C.E. attribution our total cloud spend down to the last dollar to _a team. _This required navigating the notion of organizational hierarchies, supporting reorg workflows, re-attributing and backfilling to support error handling. This can feel toilsome, and be the valley of slow death – but here’s where I’d recommend persisting through, cause it will pay dividends when done right, and well.

Once we had attributed every dollar or every time slice, we then used Pareto’s 80-20 rule to focus on the top 5-10 product or platform teams, which provided the highest leverage.

Identify the interconnectedness and the flows. The how

Drucker said, culture eats strategy for breakfast. And a key aspect to changing behavior, especially when accountability is diffused, is to motivate culture. And culture is nothing but the behaviors the system incentivizes or disincentivizes.

We saw that the Hadoop platform team allocated its resources to teams through statically assigned queues, which led to local fragmentation and overall dropped system utilization when those jobs weren’t running. We needed the platform team to implement elasticity and the job runners to release resources – but both needed to be made aware, and then incentivized to prioritize this work.

And here’s where I’ve found it immensely useful to implement and operationalize the E.A.S.T. framework for behavioral economics:

Easy to self-serve costs through attribution, rich cost observability tooling and automated customized Nudges providing insights and recommendations on ways to meet their goals
Attractive to incentivize and reward Efficiency efforts by tracking wins, providing badges or company-equivalent means of public recognition
Social by driving ownership and accountability through cohort analysis, leaderboards, public Ops Reviews, and
Timely by introducing LEAP in Eng101 onboarding classes.

Lastly, there are systems where the carrots work better, or the stick. Depending on the urgency of the problem, some levers to drive the latter are setting explicit budgets (eg: cloud spend or headcount, spend budgets for org size of 25+), ensuring that teams have the right company level prioritization for related work, enforcing capacity governance processes or ring-fencing engineering bandwidth to drive centralized optimization.

Are there any processes or forums (like a “quarterly business review” or whatnot) that you’ve found valuable for inspecting execution within your team or across the many teams that share some responsibility for performance and efficiency?

In addition to diffused accountability, the other biggest challenge with inspecting execution for areas of performance and efficiency is, realized impact.

Let’s say, a data team decides to build a resource request portal, to automate away the static allocation and under-utilization of its compute resources. They ship this feature and move on to solve other problems. However, a few months in, they don’t see any change in the overall spend on the Hadoop infrastructure.

These are especially common in the performance and efficiency space, as the evaluation of the problem is based on several hypotheses, and there’s underlying complexity in the causal chain of dependencies. In the above case, we see that resources are under-utilized and waste is high. We concluded that most waste occurs from queue fragmentation in statically assigned compute resources, dynamic allocation will thus reduce fragmentation, hence cost savings! If we’d looked deeper at the data, we might have identified that the issue wasn’t so much in the fragmentation, but in the release of unused resources – a similar but different problem, begging for a different engineering solution.

Given this complexity in diagnosis, I’ve found it extremely useful to establish a contract with relevant teams (or my own) – anchor around invariants that need to be true at the end of a certain timeline, or around quantifiably, verifiable metrics. Eg: No product engineering team will miss their p99.9 latency service level agreement for over 48 hours, and beyond that, open an incident to follow due protocol. Or, team X will spend no more than 2% over their monthly allocated spend budget; any variances beyond this will need explicit approval from executive leaders.

Whether the teams decide to solve problem X or Y, or engineer a solution Foo or Bar, then, is secondary. We shake on the outcomes and invariants- and this forsters both agency & autonomy for the teams to drive results, and also creates owned accountability.

Speaking of accountability, I am a firm believer of ‘Trust _and verify’. _It is crucial then, to create the right ~real-time alerting and feedback loops, to catch early drifts - and I’ve found the weekly Ops reviews to be at the right cadence for these. This is where we want to leverage the exec sponsor for this program, who’ll recognize the right behaviors we want to see amplified, or facilitate deep dives into the incorrect outcomes to dampen their spread.

Lastly, QBRs are a great way to formally view trends in resource management, and related impact. This is also a great time to strategize and prioritize future investment, in line with the organization’s broader goals.

On that same theme, one particular challenge I’ve encountered is the perception that infrastructure efficiency is less important than developer productivity. To the extent that is true, some would argue that it’s illogical to prioritize things like performance and efficiency. How have you dealt with this tension between efficiency (or performance) and developer productivity?

For me, the joy of engineering lies in the solving of constraints, similar to those linear programming problems in Math. Given a system, and some non-functional requirements ( eg: availability, reliability, security), how do we seek equilibrium in the system? How do we make the right tradeoffs to sustain that?

At the macro level, it goes back to the opportunity cost for the business. When does it make sense for the business to invest in efficiency or performance? When a company is in its growth stage, its engineering talent is the highest asset and finding the right product-market fit is its highest priority. At that time, and at that scale, developer productivity is higher leverage than efficiency.

But as the business matures, and its organization and the engineering systems evolve, the balance shifts. 4YPs and discounted cash flows also start expecting to yield economies of scale- especially given the compounded nature of money. The CFO is likely to assess marginal revenue per net new employee, or overall margin for the business. And for most SaaS companies running infrastructure on the Cloud, their OPEX is the second largest spend.

At Stripe, I intimately witnessed our burgeoning cloud costs, and thanks to your foresight in investing early, we were largely successful in bending the curve along multiple dimensions of our overall spend. In order to justify and equip engineering teams with the agency to drive their investment, we laid down a generally applicable decision-making framework to translate engineering time to cost savings. For example: Invest 1dev-week of effort for $10K/month savings. For our own centralized Efficiency team, we placed a high premium on opportunities worth pursuing: eg: 5X cost savings per IC. These help address some of the tension between investment in dev prod efforts vs those catering to Efficiency.

However, at the micro level, depending on the problem you’re solving, you could either improve both systems efficiency and developer productivity, or face situations where “going faster” necessitates spending more. Eg: Take CI costs: if we were to improve and finetune our selection set of which tests to run, we’d reduce the dev time spent on running tests and reduce CPU hours, thereby being more efficient. But take build times- let’s say throwing 15% more instances to generate builds, reduces average build time from 25mins to 15mins. Is it worth it? Yes. But at what point is it not- how about when going from 15mins down to 12mins?

In Staff Engineer’s Manage Technical Quality, I argued that folks should focus on pursuing quality through improving hot spots, best practices, and so on. The least recommended solution was running an organization program that requires coordination across the entire engineering organization. This is a point of view that I developed in part during our time working together based on how hard it is to coordinate moving an organization. Do you think I came to the wrong conclusion in recommending folks avoid running organizational programs as much as possible? Any advice to make running organizational programs effective?

I especially love that article on Managing Technical Quality and I wholeheartedly agree on your assessment!

I started my career as a Quality Engineer around 2 decades ago, testing key features like distributed resource scheduling and linked clones for VMware’s control plane management solution to manage VMs. It was prior to the DevOps movement, and most enterprise companies ran these through centralized teams. There were several downsides to that model, stemming primarily from misaligned incentives, which arose due to lack of end-to-end ownership in shipping a high quality product to users. The developers were responsible for checking in code, and the QE for identifying defects and performance bottlenecks. This adversarial engagement created tension, as opposed to a joint commitment to delivering value. There was also a downward spiral of brain drain, due to the system perpetuating implicit second-class citizenship, in its hiring, compensation and talent management frameworks.

Fast forward to recent times, the core tenets of DevSecOps place a high value on end-to-end ownership of engineering – from code deployment to managing maintenance and operations. Systems which embrace this model heavily benefit from your recommendation in the article - which is to focus on the hot spots, drive practices, find leverage points and so on.

As cliche as it is, It all comes down to people! People are at the heart of every engineering problem, and its solution - be it more engineering, practice, process or program. I am of the firm opinion that people want to do the right thing, but they are optimizing for the constraints they are given. The most expedient way to then drive change is to provide awareness of the problem, align incentives, and give them the time and space to prioritize the fixes. For example, if a business leader is pushing their org to release product features at a breakneck pace, it will lead to technical debt or low code quality.

Also, running a program has extremely high overhead– sustainable metrics, weekly executive sponsorship and commitment, ongoing program evaluation. A program, its related scoring or goals evaluation, and associated leaderboards, also create a sense of foreboding - it is akin to being called into the Principal’s office– and shift the balance from the program owners being medics and dependable consults to cops who must be dealt with.

But there are times when a technical program is indeed the right solution – factors here range from the scale of the engineering organization (eg: tracking cloud spend for a group of 1000+ vs 200), to bootstrapping baseline shifts in your overall posture (eg: driving least privilege access to all data) or requiring immediate change to uplevel the entire organization simultaneously (eg: compliance needs like GDPR, India data locality).

I’ve had fair success leading such programs, focusing on:

Early (and often) alignment with key stakeholders on defining the goals and soliciting their buy-in.
Fostering trust and autonomy: trust in the data you leverage to guide ongoing decisions, trust in your intention to meet the mutually beneficial goal, and trust in being an equal, supportive partner throughout the journey. Trust _and verify. _
Effective communication and tight collaboration: create feedback loops to ensure information flows at the right cadence, at the right zoom factor for the right audience.
Giving credit liberally; publicly recognizing the good citizens, or the early adopters.

What are some of the most impactful projects or tools that your teams have rolled out to improve performance or efficiency that were impactful without requiring mass-coordination across many teams?

Efficiency, Performance and to that extent, even Reliability and Security are horizontal programs. For each of these, I’ve found it valuable to establish the right balance of tooling, education & practices to drive organizational behavior and simultaneously land direct improvements by solving real engineering problems. Anchoring on either end of the spectrum disproportionately impacts the end outcome. For example, if you index heavily on laying down patterns and practices for the org to adopt, but don’t build critical infrastructure or land impact by fixing existing systems, it erodes trust and credibility. If you are making point fixes, and landing impact a system at a time, you’re likely not evolving fast enough in a rapidly scaling company.

Keeping that balance in mind, and similar to macro-economic cyclicality, I developed our Efficiency strategy around 3 dimensions:

Pay Less (optimize procurement) ,
Use Less (optimize utilization) and
Need less (optimize performance).

Early on, optimizing procurement was the single biggest lever in reducing our cloud spend. Automating Reserved Instances & Saving Plans purchasing, implementing storage tiering for hot/warm/cold data accesses and centrally leading vendor discount negotiation (in collaboration with F&S) significantly dropped the spend/business volume bps.

We then focused on the second bucket - use less- improving utilization. This involved auditing unused/unclaimed stuff, automating brownouts to those unaccessed resources and then releasing the resources to prevent future spend.

Similarly, on the latency side, we rolled out an incident-free Ruby GC optimization without needing to coordinate with Product teams. This change dropped the tail from 4.6 seconds down to 2.9 seconds.

I’ve often considered Efficiency to be an “obvious spot” to partner with a Technical Program Manager (TPM), because it’s such a cross-organizational effort and there’s no finish line: the work just keeps going further. Do you agree? How would you approach involving TPMs in areas like efficiency and performance?

There are 3 key pillars to navigate when running an effective Efficiency & Performance program - the engineering, the organization and the Finance & Strategy.

Engineering comprises the centralized team which drives the execution of the strategy, and related projects serving the end outcomes.
Organization involves the product and other infrastructure teams within engineering, their organizational leaders and the executives leading the business.
Finance & Strategy leads the overall capital allocation at the macro business level, often reporting into the CFO.

A solid TPM can serve as _the _glue and the singular force operationalizing the strategy and seamlessly bridging all 3 pillars:

Identifying technical inefficiencies in product & infra engineering and creating the Efficiency portfolio of opportunities. This could involve: a. Tracking big scale up, scale down and swings in cloud spend, and enforcing capacity governance processes. b. Tracking platform rate cards (eg: avg cost per vcpu for a Hadoop job) and quantity of resources consumed (eg: #vcore-hours for team X). c. Creating effective feedback loops to bridge the utilization with consumption and budgets.
Enablement & education to motivate change bottom up and shift left the culture of efficiency & performance - Facilitate prioritization conversations across various stakeholders and leadership to unlock resourcing for highest leverage work items. Partner with the centralized Efficiency team, Education and other Infrastructure teams to develop best patterns and practices for developing systems and services efficiently.
Operationalize budget tracking and drive high forecast fidelity by organizing monthly spend budget reviews for: a. Identifying the right set of teams and tracking org-wise budgets vs actuals. b. Evaluating engineering plans for new investments. c. Accounting for budget variances due to delayed execution (eg: Team X budgeted $Y for the month of March for a new feature launch, but came in lower cause they encountered issues), or overspend (and identifying critical remediations). d. Enabling identification of potential cost saving opportunities.

Lastly, a TPM is a core partner to the Engineering Manager and F&S, in identifying and unifying KPIs to tune the OODA loop (observe, orient, decide & act), to make macro or micro refinements to the overall strategy.

There’s a tendency for infrastructure engineering to be invisible when nothing is going wrong. How do you articulate the value of your organization’s work?

I am very glad you brought this up! With infrastructure, when something’s going wrong, there’s nowhere to hide. But the key challenge is when nothing is going wrong, how do you know it’s actually going right? So, when I think of infrastructure problems, I think of ‘great power comes with great responsibility’. And here’s why.

The beauty of Infrastructure is in its whittling down of essential complexity, through simplified abstractions which bring joy to its users. And through that, key leverage for the business.

Most mid-sized companies looking to scale, start investing in infrastructure engineering teams; with typical hiring ratios being 7-8 Product engineers to every infra IC. This makes it imperative that every infra-eng-week of effort be dedicated to high leverage work. Infra problems also take more rigor to solve, and get right, to avoid thrashing the rest of the engineering organization. Imagine building out a cloud compute abstraction, which changed every quarter, and fanned itself out to 20+ product engineering teams doing daily deploys. It’d be a nightmare!

This combination of complexity, rigor and the expectation of high ROI, makes Infrastructure engineering a very high stakes endeavor :)! Teams which romanticize or idealize the tech, over its customers or business value tend to languish – either due to missing the mark on realized impact, ceased investment due to lost credibility or due to internal employee burnout. And so articulating value, within and without, at every stage of software development is extremely crucial to leading a high performing, value delivering Infra team.

The 3 tenets I’ve found useful are 1. Know your customer 2. Bring in a product mindset, whether it involves doing initial market study (eg: evaluating build vs buy options), customer analysis & segmentation (eg: focus on data scientists over business analysts ), or even developing a go-to-market strategy (eg: white-glove migration workshops to facilitate Data Locality needs). 3. Measure what matters, not what’s easy to measure (and do the early work to identify what this is!)

At the planning stage, drive precision through alignment and prioritization — are you focusing on the right problem? And for whom? Here, you need to be grounded in the why, before the what. Do the 5 whys exercise, especially if embarking on multi-half infrastructure investments (eg: migrating a monolith to a SOA). And to seek alignment and early feedback, I’ve found the PRFAQs practice from Amazon, quite useful to build trust and credibility with your stakeholders and executive leadership.
At execution, drive focus, speed & quality, leveraging the SPACE metrics whenever applicable. Be extremely paranoid about scoping the problem just so, go deep before you go broad, and aim for vertical slivers of delivering impact vs all-or-nothing. I recently led the data security strategy for Stripe, and the biggest win we had was in the underlying approach. We identified a data access metric, and pivoted from securing one data system at a time, to incrementally driving value and moving the needle. Depending on the culture of your organization, communicate early, and often, through shipped emails, company all-hands or demos. This is a great avenue to seek feedback with your beta users, confirming the validity of your approach.
Finally, ensure that you’re maximizing overall impact: Are folks using what you delivered? Are you actually seeing movement toward your northstar metric? This is when we hone the “what” and validate the “why”. Quite recently, we shipped some work expecting to see change, and moved onto solving other problems. Looking back retrospectively, we realized that we had needed to build adjacencies to the shipped work to actually capitalize on all the effort. Sometimes, you need to evaluate what an additional 5–10% looks like to realize the most impact; this could be a marketing strategy, a small UX improvement, a small optimization (e.g. making load times much faster), or in many cases, better documentation. _Take that time. Bring it home. _

On March 15, 2013, 1,200 Japanese workers converted the Shibuya Station train Line from above ground to underground, in just 3 hours, before the first morning train the next day! I have worked on Infrastructure for nearly 2 decades now – when I think of Infrastructure, I think of this. It is this behind-the-scenes symphony of dedicated, resilient and talented people, working together to keep the masses moving, with zero friction or downtime- THIS gives me joy. And pride.

Asking the same question again but from a different perspective: how does working on something like efficiency or performance impact someone’s career, particularly in terms of getting promoted?

There’s the finite game of uplevels and promotions, and the infinite one of constant learning, development and evolution.

For the former, when we think of individual career impact, there are 3 systems at play – individual career aspirations, the engineering ladder and expectations for different levels/roles, and the business need/team opportunity.

Efficiency/Performance-shaped problems tend to be both broad and deep (eg: spark tuning for bigdata computation or improving your Kafka publish tail latency). To navigate such problems, some key traits and competencies are: highly motivated, proactive problem-solvers who can move with urgency and focus, while balancing critical thinking, comfort in dealing with data-driven diagnosis, hypotheses and analyses, and ability to work cross-organizationally, collaborating with different teams, systems and organizational dynamics. Let’s assume that individuals working on efficiency or performance-shaped problems are inherently motivated and excited about solving such problems.

That brings us to ensuring that there is indeed a strong business need to be solving these problems. A company, in its initial phases, might not want to invest in efficiency and performance, and rightly so as we discussed earlier. If we’ve secured the need and the buy-in, it comes down to demonstrated value - results, results, results! And once you’re secured results, identify the narrative for the impact driven - what’s the before/after story? What got better? What gets worse, if left unsolved? Understand and evaluate your organization’s leveling rubric, to assess if the complexity or realized impact, are in line with the system’s expectation from someone at that level and role. Eg: At Stripe, we’ve intentionally introduced the “Fixer” archetype for Staff engineers, to create room for, and acknowledge the value of associated impact to the business.

Some things to keep in mind:

Going back to your earlier bit about diffused accountability, we also need to ensure that the individuals working on these problems are well equipped to navigate this aspect of the system (especially, because ICs tend to avoid situations involving potential conflict).
If in the discovery phase of evaluating which problems to solve, identify a rubric for effort and impact (eg: 1 eng-quarter for $X million in annualized savings), and stack rank those opportunities to avoid missing the forest for the trees.
Balance driving value with incoming interrupts, when driving change through the rest of the org – ICs want to code, and solve problems, so leverage partners like the EM and TPM to help ICs get focus time.

Lastly, speaking of the long game, and my own experience : I’ve developed some key strengths through my journey from quality engineering to leading efficiency/performance programs - ability to seamlessly operate & diagnose varied distributed systems, strong business communication skills, and effectively drive influence without authority across cross-functional organizations.

What are some resources (books, blogs, people, etc) you’ve learned from? Who are your role models in the field?

You and yours - infraeng is a great resource to hear from other practitioners, operators and builders!

I love understanding how complex systems at scale fail - for these, I’ve found it extremely valuable to study the AWS Post-Event summaries! I also routinely look forward to shared content from Cindy Sridharan, Liz Fong-Jones and Corey Quinn. I found Designing Data Intensive Applications to be an excellent resource for the basics around dealing with Data.

In terms of engineering management, I’ve learnt a lot from playbooks, frameworks and resources shared by Lara Hogan, Kim Scott, and Brene Brown. My favorite book on leadership is, I too had a dream by Dr. Kurien.

Engineering Leadership at the higher levels starts getting fuzzier in time and space, and has lesser structured content. Here, I do find myself leveraging a lot from my Macro & Micro Economics - and leaning on more holistic modeling of the world around us (Thinking in Systems: A Primer), and our own interpretation and response of it (Mindset: The New Psychology of Success, Switch: How to Change Things When Change Is Hard, Unlocking Leadership Mindtraps: How to Thrive in Complexity).

Contract Negotiation Checklist

Mon, 04 Apr 2022 07:00:00 -0700

Fork this template on Google Docs

Negotiating contracts is an important part of managing costs, but it’s also something that you only do infrequently. Particularly in an earlier stage company, you might only negotiate one large contract a year. It’s quite hard to get better at something that you do so infrequently, but using a checklist is one way to be consistent in your approach, and to ensure learnings from one negotiation carry over into the next.

Contract Negotiation Checklist

There’s no perfect checklist: you should customize the checklist for your company based on your process, preferences and the experience level of those who are involved. If this feels too heavy, then by all means remove some steps.

More Readings On Vendor Negotiations

How to Use

Fork this template on Google Docs
Follow the template’s checklist
Link your template into an internal repository of all negotiations so folks can find it the next time you’re negotiating this or related contracts
Now you’re done!

Tips

Negotiating contracts is a learned skill, and you’re probably not going to be good at it for the first few. That’s ok! Find someone else with more experience to partner with on your first few, even if it’s just asking a more experienced friend at another company to brainstorm with you through the process
Whenever possible, negotiate knowing how much other companies are paying the vendor you’re talking to. This creates a clear price ceiling to negotiate towards
If the vendor hasn’t hit their quota and is approaching their financial year or quarter, you can almost always make significant progress on terms if you’re willing to move quickly
Everyone should know their role in each negotiation. Sometimes your role is being the difficult, inflexible person! Sometimes your role is being the person who thinks they’re the final decider but is infact mistaken when your manager “take overs” later “much to your chagrin”
Not all contracts are equal. Sometimes the absolute number of dollars isn’t high enough to follow a time consuming process. On the other hand, sometimes the numbers are genuinely massive and are worth pulling in even your most senior leadership to get the best possible deal

Efficiency: Managing Infrastructure Costs

Sun, 03 Apr 2022 07:00:00 -0700

In my early career roles, I worked at companies that never worried about their infrastructure costs at all. They were simply too low a cost and growing too slowly for the Finance team to pay much attention to it. This “ignore it until it’s too large to ignore” approach served me well.

Until it didn’t.

Working at Uber, I was caught me off guard when a new Director joined and overnight infrastructure costs were recategorized from insignificant to requiring urgent, detailed review every month. Adding the instrumentation and accountability for these costs retroactively was a difficult retrofit. Although I was surprised that time, I’ve come to appreciate that all successful companies go through the transition from ignoring to setting goals on infrastructure costs, and an early focus during my time at Stripe was ensuring we were ready ahead of that shift.

Your job as an infrastructure leader is diagnosing the right mode of operation for your company’s infrastructure costs today, understanding when you’re likely to switch modes, and ensuring you’ve done the prework to make the transition relatively painless.

We’ll explore this topic by digging into:

three distinct operating modes for infrastructure costs: early-stage, growth, and late-stage
concrete tools and tactics such as managing infrastructure costs with cloud-specific reductions, including costs in your Business Review Template, and using a Contract Negotiation Checklist
whether you should spin up a dedicated team working in this area

When you finish reading this, you won’t have your entire efficiency plan worked out, but you will have the high-level pieces, know where you need to dig in, and have a clear approach to communciate to anyone who has been pushing you for a documented approach around infrastructure costs.

Related Interviews

Smruti Patel: Head of Engineering for L.E.A.P. at Stripe

Should you prioritize infrastructure costs?

Before diving into the mechanics of managing infrastructure costs, the first question to answer is whether it’s a valuable use of organizational time to make your current infrastructure spend more efficient. How you think about this will vary a bit depending on whether your company is early-stage, prioritizing growth, or focused on profitability in late-stage.

Early-Stage

Generally speaking, very early-stage companies shouldn’t spend much time thinking about infrastructure costs. You should instead be focused on finding product-market fit for your first product.

Here are two checks you can run to determine if it’s worth reducing your infrastructure costs:

If you were to reduce your infrastructure costs to $0, and it still doesn’t increase your runway by at least two months, then it’s not worth focusing on
If you’re spending less than $2,000/month per employee on infrastructure costs, then it’s probably not a significant priority because your headcount spend will be so much higher

If you’re not violating either of those checks, then keep on ignoring infrastructure spend. If you are exceeding one, and infrastructure costs are a significant part of your overall burn, then invest a sprint into reducing spend, and then resume ignoring it once these checks resume passing.

The one notable exception is if you’re building a low-margin product or product where cost efficiency is a pillar of your long-term strategy. For example, if you’re operating a metrics collection and dashboarding product like Datadog, then efficiency probably is worth considering earlier than usual.

Growth

When you’re prioritizing growth, the primary focus of the engineering organization in a technology company is creating, operating and advancing the products that support the business. Managing costs is important, but even immaculate cost management won’t make your company a success if enough energy isn’t being invested in your product.

The fundamental question to ask is whether infrastructure’s share of cost of goods sold (COGS) is increasing as a percentage of revenue? (The simplest way to think COGS is all your non-headcount costs, although a slightly better definition would be all costs to operate your software.)

Start answering this question by plotting revenue and infrastructure costs on a chart to get a sense of how these two numbers are moving. Although logarithmic scales often generate more confusion than they’re worth, in this case it’s usually the only way to see both lines closely enough to understand their slopes within a single chart. You particularly want to understand if either line has experienced an inflection over the past few quarters. If costs have started accelerating without corresponding acceleration of revenue, that’s worth digging into.

Once you’ve looked at the two lines independently to understand their movement, simplify your first chart into a chart showing infrastructure costs as a percentage of revenue. This chart hides some detail but is easier to parse for folks further away from the details. As long as the ratio is going down and your company is focused on growth, then this data should be sufficient to justify your current level of investment into efficiency: if growth is key, and infrastructure costs are not getting in the way, why should you slow down growth to reduce them?

Late-Stage

Even the best business lines stop growing at some point. Facebook is one of the most valuable businesses in the world, but even they at some point ran out of new users to attract to their platform. Once growth slows, a business naturally starts focusing more on costs, including infrastructure spend.

In those scenarios, the easiest approach is to work with the business to align on two numbers:

Dollars spent on infrastructure overhead per engineer: this includes things like development environments, testing tools, and so on. Determine your starting point by bucketing vendors and non-production infrastructure costs into a chart and plotting them over time divided by headcount. Pick a reasonable point on that line as your target. Refine it by reaching out to industry peers to get a sense of how this number compares to theirs (be sure to pick industry peers in companies that are currently focused on profitability, otherwise their answers won’t be very helpful to you)
Infrastructure dollars spent per N product operations served: anchoring on cost of operating the product. This will vary a bit depending on your product or business, but it might be “$1.00 in infrastructure costs to powering every 10,000 searches”, “$2.50 for every 10,000 payments processed”, or “$3.00 for every 10,000 trips scheduled”

In both, the key thing is moving away from anchoring on a percentage of revenue and instead setting a target against the fundamental operations that you support. Thinking of costs as a percentage of revenue works well when you’re growing, but is too abstract and hides too many details once you’re focused on reducing costs.

If you find yourself exceeding those targets, then it’s time to dive into reducing them.

Tools for Managing Infrastructure Costs

What I’ll introduce here is the fairly common playbook for managing infrastructure costs. As you work through these approaches, your goal is to do as few of them as possible while meeting your efficiency goals. I’ve prefixed a few particularly high return-on-investment tools with a “⭐, if you’re debating where to start, consider starting with them.

⭐ Use cloud vendor’s cost optimization tools. Every cloud vendor has a program along the lines of AWS Savings Plans or AWS Reserved Instances. These plans allow you to trade usage or spend commitments for reduced pricing. If you aren’t already using these, you can usually reduce your infrastructure costs by 20-40% in a few weeks of work
⭐ Standardize your vendor negotiation process. Beyond a core cloud vendor, many companies have five or six additional large vendor contracts for things like observability, security, or developer productivity. Introducing a structured process for negotiating and renegotiating, like using a Contract Negotiation Checklist, will significantly improve your pricing (as well as visibility into costs)
⭐ Run periodic deep dives on cost. Until you have a dedicated team actively looking at your infrastructure costs, you can usually identify significant cost reductions by periodically taking a week to dig into your biggest infrastructure costs and prioritizing low-hanging fruit. These will usually be accidents, like storing unused data, development environments not getting retired, etc. The key thing is scoping the opportunity to work that the infrastructure team can take on themselves
Update your Tech Spec template to include a section that estimates costs. Many engineers will be unfamiliar with that process, so make sure the template links to examples of how a few representative services estimated their costs. A great example will be onerously detailed, including links to the specific queries and tools to estimate their costs. A template that requires cost estimation without guiding folks through that process will inevitably trend towards make-work rather than a useful discussion
Find the executive sponsor who really cares about infrastructure costs and is willing to push inefficient users to spend less. This is usually your CTO or your CFO. Without an executive sponsor willing to prioritize this efficiency work, you’ll find progress further down this list difficult. If you can’t find a sponsor, that’s usually a good sign that you’re already doing enough to prevent infrastructure costs from becoming a top priority
Find product cost optimizations. There will be significant opportunities to reduce costs by changing how your product works, e.g. improving your data model, changing storage technologies, moving workloads between streaming and batch. However, product changes have a much wider set of stakeholders, which makes these sorts of improvements harder to prioritize. Generally, only try to pursue these if there is a massive opportunity
Pursue a cloud vendor contract discount. Negotiating your cloud contract to include a discount is very doable after you reach a certain level of spend, or are a sufficiently strategic partner, but before you reach that level of spend it’s quite difficult to get a meaningful discount. Is it worth spending three weeks and making multi-year financial commitments to get a six percent discount on your cloud spend? Maybe! It depends on your priorities and your confidence in future spending estimates, but it certainly isn’t worth it to everyone. Conversely, at a certain spend level–think, tens of millions of USD per year–your discount can get much higher without requiring any product-level changes
Set coarse goals on infrastructure costs. Partner with your company’s Finance team to coarsely attributing costs across teams, then set and monitor goals against those costs. Fine-grained goals and cost attribution requires a deeper investment into tooling, but most companies can split costs across their production environment, development environments, and data engineering. Once you done that split, you can set a goal and assign that goal to appropriate teams (respectively, something along the lines of infrastructure, developer productivity, and data engineering). This will provide some visibility and pressure on costs without requiring much attribution prework
Review costs in Business Reviews. Once you’ve set those coarse goals around infrastructure spend, ensure your company’s Business Review Template includes a section on their costs. If you run Business Review Meetings, then make sure someone is showing up to ask questions about costs for teams whose costs are missing goal or otherwise accelerating
Expand metadata to facilitate fine-grained goals on infrastructure costs. Implement an approach to Ownership Metadata such that you can assign all usage and storage costs against a specific team. Once you have that ownership metadata maintained, you can go further by generating proactive nudges to teams on following best practices, prioritizing high costs, and helping them identify accelerating spend early

If doing all of these sounds overwhelming, it should! Few companies do all of these, and those that do either operate in a business that is unusually margin sensitive or are spending many millions a year on their infrastructure costs.

Should You Have a Dedicated Efficiency Team?

Generally, the way I think through spinning out any given area into a dedicated team is described in Trunk and Branches Model, and that applies for the efficiency as well. That said, let me add a few caveats to that general approach as it applies here.

Much like managing technical quality, efficiency is an area where you can make significant progress with one-off initiatives. Improving how you use AWS Reserved Instances or renegotiating your vendor contracts can reduce your spent by 30-40% in a week or two. Product-level improvements to your architecture can reduce your spend even more, although they’ll probably take a bit longer.

Because you can make significant progress through one-off initiatives, the default is to wait until late into a company’s growth to spin out a dedicated team, and in most cases that’s the right decision.

The three factors to consider as you think through whether postponing a dedicated team is the best solution for you are:

Is infrastructure efficiency a fundamental strategic pillar for your business?
Are your infrastructure costs, today as an absolute cost, 10x more expensive than a team working to reduce them?
For the past year have you had pressure to reduce costs but an inability to prioritze the work because other critical work continues to displace efficiency efforts?

If you answer yes to any of those, then you may want to spin out a team earlier than the Trunk and Branches Model suggests. As you start sourcing candidates, it’ll become apparent that this is a bit of a custom role with folks who specifically enjoy working on the problem. Recruiting one or two folks with siginficant preexisting experience will save you years!

Strategies

Sun, 03 Apr 2022 07:00:00 -0700

Business Review Template

Wed, 30 Mar 2022 07:00:00 -0700

Fork this template on Google Docs

As your company gets larger and more complex, it’s easy to become embroiled in supporting incoming asks from other teams. That’s important work, but it’s also important that your team is operating effectively and prioritizing your goals in addition to the goals of other teams making requests.

If you’re getting mixed signals on whether your team is doing the right work, the Business Review Template can help cut through the confusion. This written document facilitates an operational review of your team, and even more importantly creates an opportunity for you, your team, and your stakeholders to discuss if you’re focused on the right work.

Example using the Business Review Template

Most companies wind up using a variation of this template by the time they reach a thousand employees, with some starting much earlier. Even if there’s no structure business review process, it’s helpful to start writing them periodically for the area you’re responsible for: think of them as your area’s performance review.

Related Meetings

Business Review Meeting

Other Approaches to Business Reviews

The Kool-Aid Factory’s The Shipping Great Work Issue

How to Use

Fork this template on Google Docs
Find examples of previous business reviews at your company, and if possible ask the authors what was and wasn’t well received in their most recent review
Fill in the template for your team’s area
Iterate on your draft with feedback from your team and manager
Identify the key groups you want feedback from, and create copies for each of those groups. Transparency is important, but transparency too early often mutes the direct feedback that helps you succeed. Give these groups a week or so to provide feedback, including running a Business Review Meeting if that’s something your company finds valuable
Widely publish a clean, readable copy into wherever business reviews are collected, and let anyone who hasn’t gotten a change to see it so far know where to find it and how to share feedback on it
Now you’re done! (At least until the next one.)

Tips

Writing an effective business review depends first and foremost on understanding the audience you’re writing for, and what that audience cares about. If you’re not sure about the answer to either of those, ask!
Many companies and many teams try to use their business review to solve too many different problems. Your business review should focus on answering only two questions: how well is your area of the business operating? What do you need to do for it to operate better?
Good business reviews are focused on what the reviewers need from the review. Bad business reviews are comprehensive, capturing everything that someone on the team wants reviwers to know
Every metric you include in a business review should be a well-formed metric that includes the current value, the goal, and the trend over time
Avoid delegating the writing of your business review to multiple different folks. Short documents with disjoint authors are hard reads
The Amazon Way of Writing is a helpful set of rules for writing these sorts of documents

Infrastructure Engineering

Categories

Book

Tech Spec Review

Goals

Agenda, Scheduling, and Scaling

Roles & Attendance

Is it working?

Meetings

Incident Review

Goals

Anti-goals

Agenda, Scheduling, and Scaling

Roles & Attendance

Is it working?

Tools

Matthew Clarke

Manager

Interview

Mahdi Yusuf

Shawn Wang / swyx

Developer-Experience

Smruti Patel

Contract Negotiation Checklist

How to Use

Tips

Efficiency: Managing Infrastructure Costs

Should you prioritize infrastructure costs?

Early-Stage

Growth

Late-Stage

Tools for Managing Infrastructure Costs

Should You Have a Dedicated Efficiency Team?

Strategies

Business Review Template

How to Use

Tips