Devops for Growth

There are three common lists or methodologies:

• From the Google SRE book: Latency, Traffic, Errors, and Saturation
• USE Method (from Brendan Gregg): Utilization, Saturation, and Errors
• RED Method (from Tom Wilkie): Rate, Errors, and Duration

You can see the overlap, and as Baron Schwartz notes in his Monitoring & Observability with USE and RED blog, each method varies in focus. He suggests USE is about resources with an internal view, while RED is about requests, real work, and thus an external view (from the service consumer’s point of view). They are obviously related, and also complementary, as every service consumes resources to do work.

For our purposes, we’ll focus on a simple superset of five signals:

• Rate — Request rate, in requests/sec
• Errors — Error rate, in errors/sec
• Latency — Response time, including queue/wait time, in milliseconds.
• Saturation — How overloaded something is, which is related to utilization but more directly measured by things like queue depth (or sometimes concurrency). As a queue measurement, this becomes non-zero when you are saturated, often not much before. Usually a counter.
• Utilization — How busy the resource or system is. Usually expressed 0–100% and most useful for predictions (as Saturation is probably more useful). Note we are not using the Utilization Law to get this (~Rate x Service Time / Workers), but instead looking for more familiar direct measurements.

Start here

Distributed Systems Observability – Cindy Sridharan

What the hell is observability? How is it any different than monitoring? Is it just the “devops” vs “sysadmin” debate all over again? This article answers these and so much more.

Monitoring Isn’t Observability – Baron Schwartz

This quote from the article sums it up quite well: Monitoring tells you whether a system is working, observability lets you ask why it isn’t working.

Monitoring, Analytics, Diagnostics, Observability, and Root Cause Analysis – Baron Schwartz

Now that we’ve introduced some nuance into our world, our terminology is getting overloaded. This post sets out some definitions of what everything means and how they differ.

How do I make my applications more observable?

Monitoring and Observability with USE and RED – Baron Schwartz

USE and RED are two methods for deciding what to instrument why. This article walks you through their meaning and usage.

Hierarchical Observability with RED – Baron Schwartz

One of the best parts about the RED Method comes when you instrument all of your services to emit the same data: it becomes soooo much easier to spot the troublesome service in a microservice/distributed architecture.

Best Practices for Observability – Charity Majors

The list of best practices at the end is worthwhile reading.

How to Monitor the SRE Golden Signals – Steve Mushero

Another method for deciding what to monitor is the Four Golden Signals, popularized by the Site Reliability Engineering book. This article series by Steve Mushero walks you through what the signals mean and how to gather them.

Who is actually doing observability?

There are a number of teams out there doing observability-like things, and some of the larger, more engineering-focused companies have more mature Observability teams that are focused on providing expertise and a platform to other teams.

Twitter

Observability at Twitter

Dating from 2013, Twitter was one of the first companies to work toward solving the problem of monitoring high-scale, distributed monitoring. For more details on their architecture (from 2016), see these posts: Observability at Twitter: technical overview, part I, Observability at Twitter: technical overview, part II

SRE advocates addressing problems blamelessly. When something goes wrong, don't try to determine who is at fault. Instead, look for systemic causes. Adopting this approach has many benefits, from the practical to the cultural. Your system will become more resilient as you learn from each failure. Your team will also feel safer when they don't fear blame, leading to more initiative and innovation.

Learning everything you can from incidents is a challenge. Understanding the benefits and best practices of analyzing contributing factors can help. In this blog post, we'll look at:

• A definition for root cause analysis
• A definition for contributing factor analysis
• How to choose between RCAs and contributing factor analysis
• Best practices for contributing factor analyses
• How to incorporate learning from analyses back into development

Il est éclairant de voir comment Duncker voyait la «résolution de problèmes».

Le processus de résolution de problèmes, développé par Duncker

1. Si un objectif ne peut être atteint immédiatement par ses actions évidentes ou habituelles, il devient un problème. Selon les mots de Duncker: "Un problème survient lorsqu'une créature vivante a un but mais ne sait pas comment ce but doit être atteint. Chaque fois que l'on ne peut pas passer de la situation donnée à la situation souhaitée simplement par l'action, alors il doit y avoir recours à la pensée (Par action, nous comprenons ici la performance d'opérations évidentes.) "
2. La résolution de problèmes comprend des phases, chaque phase étant une reformulation du problème. Duncker décrit cette étape comme suit: "... la solution d'un nouveau problème se déroule généralement en phases successives qui (sauf la première phase) ont, rétrospectivement, le caractère d'une solution et (sauf la dernière phase), en perspective, celle de un problème."
3. Le point ou la fonction d'une solution est également sa définition de «solution». "La valeur fonctionnelle d'une solution est indispensable pour comprendre qu'elle est une solution. C'est exactement ce qu'on appelle le sens, le principe ou le point de la solution."
4. La définition du principe de la solution est, en général, la première étape du processus de résolution. "La forme finale d'une solution individuelle n'est, en général, pas atteinte par une seule étape à partir du cadre d'origine du problème; au contraire, le principe, la valeur fonctionnelle de la solution, survient généralement en premier, et la forme finale de la solution en question ne se développe que lorsque ce principe devient de plus en plus concret. "

Note: This is the culmination of years of work managing and optimizing the practice of technical operations groups/DevOps at scale. These are proven tactics and techniques that can be applied across any technical value delivery organization of any size to increase efficiency, satisfaction, and enterprise agility. While I had hoped to write a book on this eventually (see the outline for what that would have looked like), I do not have the time to do so, and yet these topics are extremely relevant especially as the cloud native revolution takes hold. This is not a replacement of DevOps, but instead the overarching framework that DevOps is a part of.