apenwarr

thundersnap 0.01: an undo button for everything

Sat, 04 Jul 2026 04:24:21 +0000

Happy July 4th! For those of us around the world contemplating independence, it's a good day to think about how we came to rely on expensive cloud infrastructure for our fundamental computing needs.

With that in mind, here is my latest toy project: an open source tool that makes replicating, forking, sharing, and running container snapshots fast and easy across cloud and personal devices.

It's fun to play with, especially on bare metal hardware you run at home, or rent from a provider like Hetzner or OVH. Or, because it uses Tailscale, why not all of them in a single mesh?

There's a lot more to say but I don't have time right now. Details are in the README.

I will say this: humans and AI agents both want the same things when they're trying to get work done. Ephemeral containers aren't really it. But how about unlimited disk space, fast CPUs, an undo button, and the ability to move to whatever provider offers the best hardware at the best price? That's more like it.

Go visit thundersnap on github and tell me what you think!

The software industry: annealing, but wrong

Sun, 31 May 2026 20:24:22 +0000

In recent months I've heard of several teams with an interesting policy: each pull request should be no more than a few files, and no more than a certain number of lines (say 500). And do just one thing and do it well. And be easy for a human to review. And be fully tested by the test suite.

All those are good requirements, right? Surely this is quality software engineering.

And often, the results are good. Sure, splitting a single 6000-line feature or fix into twelve 500-line PRs is more work, but each of those PRs is surely easier to review. And you can git bisect them when there's a bug! And maybe revert the individual change that broke something.

...and also cause 12x as many context switches for your reviewers as they review each one sequentially.¹ But that's just the cost of software quality! Right?

Mostly, yes. My analogy here is simulated annealing. In that process, you start your problem solving with a high energy -- making big changes to move quickly through the problem space -- and then slowly reduce the energy level so that your "hops" get smaller and smaller. In real physical annealing (used eg. for metallurgy), the result is stronger, more stable, more crystalline structures. In simulated annealing, you use it to find solutions that aren't obvious, by rapidly exploring the solution space and then zooming into the areas that look most promising.

In software the analogy is clear: sure, you might start with big jumps, but once your system is more mature, you should make smaller jumps. Big jumps break the crystalline structure. They cause bugs.

Fear of breaking the crystalline structure sounds cooler than fear of change

The main problem with annealing-driven intuition happens when things do need to change quickly. It's not made for that. You usually don't build a hammer and then decide one day you want it to be a different shape. But every day, there are compelling-sounding reasons to make your software a different shape. Annealing is the enemy of change.

Modern AI-driven coding (ironically, with LLMs trained using a process quite similar to annealing) does not care about your annealing and your risk management and your fear of change. It produces changes as big and interconnected as you want, jumping all over the solution space as quickly as you can prompt. And it has all the outcomes the math would predict: the output is less strong, less coherent, more likely to fail. LLMs have no fear of change because the LLM instance will be long gone before the consequences materialize.

But, it's a new and special feeling to suddenly be able to take a large, mature code base and suddenly explore any kind of large change you want. Most of those changes turn out to be bad ideas... and it's nice to be able to discard bad ideas quickly. But some turn out to be good ideas. Then what?

Well, follow your development processes. Break the big changes into 500-line patches. Review them one by one. You already did the research! You know it's worth it.

Not every big step is made of small steps

But it's not about being worth it -- some changes simply don't lend themselves to small steps.

In the early development of Aperture, I wanted to implement dollar-based spend quotas: across all your LLM backends, let a given team or person or node spend up to $x per unit time. But to do that, we first had to add pricing information (it's mysterious how LLM vendors don't to tell you how much your queries cost), which meant assigning prices to provider definitions, and then we had to assign quotas to particular identity+model+session combinations. And quotas are one of the first key value propositions of Aperture. We had to have them, but we had to have all that stuff.

So, I made a giant change that included three major areas: first, the Grant syntax for applying attributes to sessions; second, a query cost approximator that combined multiple sources and a messy heuristic; third, the actual quota enforcement system. Each of these parts was imperfect, but we needed all three parts in order to make anything work at all, before we could refine them. That's the high-energy big-jump part. It came out to something like 12000 lines of code.

Now, I'm not a monster. After I made it all work, I split it into three parts: the grants, the pricing, the quotas.² Otherwise it really would have been an unreviewable mess. But also, I could not have developed the quotas feature in real life in that artificial order. The grants structure evolved as my understanding of pricing and quota enforcement evolved. The original quota semantics sucked, so I rewound back to the data structures, which affected how the pricing got imported, which changed how the quotas were stored. The code reviewers didn't have to worry about that but I did.

Mercifully, because Aperture was new, everyone on the team understood that three 4000-line patches were better than twenty-four 500-line patches when implementing this series of feature. There was even some forgiveness when it came out later -- inevitably -- that each of those parts was not quite right and needed more bugfixing. That's how new software gets made. That's the annealing stage.

But the hard part was the philosophical difference between that and, say, core Tailscale. Tailscale has 7+ years of maturity behind it. It's been annealing for a long time and it has a reputation for extreme quality, hardening, durability, whatever you want to call it. If you start pulling stunts like that in core Tailscale, stuff absolutely will break and its millions of users will absolutely not be impressed. Which is why, for the most part, we don't.

But the feeling of moving fast again is such a wonderful feeling. Some people devolve the analysis to "founder mode" and call it a personality thing, but it's not. It's using the right tool for the right job at the right time. Sometimes you need to go fast, sometimes you need to go slow.

Pain does not cause gain, it's just frequently correlated

That feeling of moving fast again reset my brain a little. It reminded me that some changes to mature products can become impossible because we commit so hard to the math of annealing that we fall forever into a local optimum. Sometimes, when the well is too deep, you can't escape from it without a bigger jump.

We're entering a world where it's cheap to produce bigger changes, but that doesn't make it any safer. Or, it's cheap to ask an LLM to artificially break your change into a dozen rule-compliant PRs but then you just stuck on tedious neverending code reviews instead.

On the other hand, it's also possible to fork your own project a dozen different ways, add huge compliance test suites you never could have afforded to invest in before, rewrite your project in Rust in a week just to see what happens.

Sturgeon's Law says 90% of your big changes will be crap because 90% of everything is crap. When your changes were 500 lines long and you had to reject them, that didn't feel like a huge sunk cost. But now, it's okay if your 12000 line changes are crap and you have to reject them; it's the same cost to write³ as the old 500-line change.

You still have to figure out how to efficiently review, reject, and refine these big jumps. You definitely need a much heavier investment into CI/CD automation, specifications, UX testing, all of it. But also, all those things just got cheaper.

I wouldn't recommend overdoing it. The other thing is, customers don't like it if you change your product out from underneath them too often. But sometimes, you're just stuck in a rut. Sometimes you have to use a higher-energy jump to get unstuck. That doesn't mean you abandon smaller steps. Use the right tool for the job.

Footnotes

¹ The reviews only need to be sequential because Github's code review system doesn't support stacked diffs, 18+ years later, leading us into this false dichotomy in the first place.

² That's a slight oversimplification since there were a couple of other parts first. I had to define the data structures for the quotas before I actually added the quota system, so that I could use the data structures in the grant syntax, and so on in a big circle.

³ A 12000-line AI-driven patch might take as much time to write as a 500-line human-written patch, but by default it's much more work to review. In fact, so much work that people give up trying, and rightly so. Rather than abandon hope, I continue to think we need to invest more into (and will gain more from) non-annoying AI-assisted review workflows than AI-assisted development workflows. Imagine for example an automated pre-human-review step that says "no, this sucks, fix these 25 things first" and closes the pull request. Is it rude? Not really, if it's good quality advice that comes back fast. In a world where reviewing code is hard and writing it is easy, put more demands on the writers.

Web Summit Vancouver 2026: AI and the next generation of engineers

Tue, 26 May 2026 00:00:00 +0000

AI isn’t killing engineering. It's making it less meditative.

There’s a lot of talk about AI killing engineering jobs. Some jobs will change, and some will disappear. But we’ve been automating engineering work for decades, and somehow we keep finding more engineering to do.

At Web Summit 2026, I talked about how AI can raise the floor for developers, and what that means for juniors trying to break into the industry.

If you'd rather read than watch, the full transcript is below.

Transcript

0:03 Hi. Hello. Thanks so much for joining us and I'm so delighted to have this opportunity to speak with Avery because you know, I hear from young developers all the time as well as employers who are trying to figure out what it even means to build a tech career or to manage a team of developers in this world where

0:29 you know, I'm a I myself have no CS training and I now make software and of course there are lots of companies all over the world where people are vibe coding in ways that change what it means to be a developer. So Avery, you obviously have had a whole career. You're an engineer yourself. You've managed engineering teams and now you're in the position where you're I assume well, I went and looked at Tailscale's

0:52 current job ads. How how do you think the role and experience of building a career as an engineer has changed over the course of your career and in particular with the advent of AI? So yeah, you talked you talked about young developers. I'm I'm clearly not a young developer. I'm an old developer. So I've seen a lot of stuff. When I was

1:14 when I was going to university in the late 1990s, there was a program that I didn't take called software engineering that was competing with computer science. I actually ended up taking computer engineering, but we had a lot of discussions about what what is software engineering? This is like a new thing. Yeah. And we're in Canada and it's like engineering is actually a protected legal term in Canada. If you work for a company and you don't have an

1:36 engineering license, they have to call you a software developer. You can't be a software engineer in Canada unless you like meet these certain criteria. And so like at the time the joke was like look, there's no such thing as software engineering. Like if you're an engineer, right? And you're building a bridge and the bridge falls down, people are going to sue you, right? And if you're selling software, you just put in the license agreement. Yeah, sorry, it's not my fault. Haha,

1:59 you know, as is. Yeah. Right? And that that's the difference, right? But over time, we have actually figured out what real software engineering is in the intervening years, in the last like 25 years since then. Like, engineering is taking responsibility for your work and understanding that like everything's going to break eventually. And like, monitoring that and deciding when is it going to be okay for it to break and what are you going to do about the fact

2:20 that it's going to break. And so, to me, like software like software engineering is new. Right? What's different now is is suddenly the part of the job that was used to be called you called computer science or used to be called software developer, like a lot of that's disappearing, right? But the engineering part is exactly the same as it always was. Like, that's what people want to buy. They want to buy the guarantee that this thing is is when it if it and when

2:42 it falls apart is not going to kill people. So, I mean, I feel like that's a distinction that a lot of people are missing, as you can see if you use five-coded software. The it won't break thing seems to be a loosely held goal. Um you know, for folks who are are building and managing teams of developers now, where people come into the field, um you

3:08 know, how how much do you think developers still need to prioritize I almost want to call them like it's weird to describe coding as an old-timey skill, but it's starting to feel that way. I mean, do you still think that when you're hiring, you want somebody who knows how to write the lines of code or is the guarantee that it won't break about a different sort of a different

3:32 lens? So, I should say that the guarantee that it won't break is not is more like, you know, deciding how likely you want it to be to break, right? But my first year engineering class, I remember we did a uh they made us do this experiment where they gave us like a bunch of paper clips and you had to just bend it back and forth and then write down how many times it took before it snapped. And then you did that with like 20 paper clips, and then you had to like plot it on a curve,

3:55 right? And then and the guy like combined all of our answers, and he was like this like beautiful Gaussian curve. He was like, "Hey, this is reality. There is no paper clip that doesn't break. Some of them break in one bend, right? You have enough of them, like some of them are going to break in one bend, and some of them like didn't break until like 20 or 30 bends." And he said like, "You need to understand, they don't make paper clips that won't break until 100 or 1,000 bends because nobody

4:18 would buy them cuz they would be too expensive, right? The paper clips people want to buy are the ones that break, right? And so, that's what engineering is is understanding those constraints. And so, as somebody who's like building software, it's like, "Okay, like it's okay if I write code some crappy stuff that barely works if that meets the specification. If I'm writing an app for myself, right? No, it doesn't

4:42 matter. It can break and I'll just tell Claude to fix it again, right? If I'm trying to sell something to a billion users that searches the internet or whatever, it's like, "Hey, it it needs to work." Well, I I love this analogy because I I feel like as a user, my special power is I know how to bend the paper clip to break very quickly. I feel like I should That should be a monetizable service

5:04 that I provide to software companies. But I'm also really struck that um when I I I do hire uh developers uh for different kinds of projects, and that the ability to anticipate when something is going to break is part of how I antici- like is how I evaluate, like how do people handle the fragility of what they're what they're building, and how do they

5:26 detect it. And I guess what I'm curious about is as AI becomes more and more how people navigate the development process, um do you think that we're losing some of those skills around understanding like the user interaction, understanding the um security implications, for example. we

5:49 were talking about that a little bit. I I'm I'm curious about whether you think that people who are kind of essentially growing up and learning um learning the field while these tools are already available, are they missing some of those basics now? Yeah, it's it's an interesting question. I think it it's hard to tell like exactly what the value of those basics are. Like when I was growing up, uh I started programming and we had like this

6:12 little computer at home that cost a few hundred dollars. Yeah. And allowance. And like that the time they sold the like the assembly language assembler for like a hundred dollars. And then there was a C compiler for another hundred dollars that we depended on the assembler. And I could not afford the second hundred dollars. And so I bought the assembler. No way. And I'm like, well, that's that's what's going to happen. I'm going to read the because programs actually came with books at the time. So I read the book

6:34 and I learned assembly language. And so I know how the guts of computers work, right? And then like by the time I'd saved up another hundred dollars, my computer had been obsoleted and like the last copy of the compiler they'd thrown it out at Radio Shack. And I could never get a C compiler. So then we had to switch to long story. But the the point is that I know assembly language. Since that time, I've written almost zero assembly language.

6:57 Right? It's just it's obsolete. Compilers have like eliminated the need to write assembly language. And yet the fact that I learned assembly language gives me a like a leg up on a bunch of people who have like come out since then have never had to write a line of assembly language in their lives. Right? Does that mean those people are obsolete? Does it mean they can't get a good job? Does it mean they can't write good software? Like, no. Right? I can write some software that they can't

7:18 write. But also they can write software that I can't write because they learned something different instead. Mhm. Right? And like the lines of code are just not that important, right? Like stuff you listed about like understanding user needs, right? And reacting to feedback and debugging things and architecting things, like none of that's going away. Like AIs are not doing that stuff for you, especially the user feedback. AIs have no idea what the user experience of your program is.

7:42 And that's like the defining element of engineering. Like what does the user need, right? Does it need to be good? Does it need to be fancy? Does it need to be expensive? Does it need to be cheap? Does it need to scale? Does it need to have a button over here versus over there? The AI can't tell you any of those things, right? And if you're distracted by lines of code, you're not going to think about those things as much as you should. It's It's interesting to hear you say that because one of the things that I

8:06 really struggle with at this point is how how much do you think um folks in those early stages of their career should be investing in Okay, I'm going to say assembly language, maybe not not so much, but you know, in the um nitty-gritty of being able to write a complete program, let's say, what you know, in whatever language, but versus like an a a you know, a young developer

8:31 who maybe has primarily focused on figuring out the requirements and the IA and those sorts of pieces and then is using you know, various AI coding agents to do the I don't want to call it the heavy lifting, but like the rote work, all of the generating of the code. Like if you're hiring a developer or if you're advising somebody who's new in the in the field, are you encouraging them to learn the line-by-line

8:56 code review skills or are you encouraging them to like go manage a team of a hundred virtual coding agents? Well, the funny thing is like all this stuff is valuable skills, right? Like, you know, to this day, if you learn assembly language, there are jobs you can get that nobody else can get because like the people who run these LLMs on GPUs are doing stuff in assembly language to optimize those LLMs on GPUs and train them faster, right? Those are

9:20 very very very high-paying jobs because so few people know how to do them, right? If you want a job that like if you want some skills that'll make it easy to get a job like anywhere, you should probably learn how to train a hundred LLMs or like manage a hundred LLM agents cuz that's what everybody's trying to hire right now. But like both are fine, right? So my advice to people is like if you think it's fun, you should probably learn it cuz

9:41 learning is a skill on its own and the more you learn, the more value you're valuable you're going to be. I know so much random stuff about so many random things. [laughter] And like each day, I'm like, wow, it's surprising that this dumb thing I learned cuz I was interested when I was browsing Wikipedia just paid off in my job as the CEO of Tail Scale. And I just like it occurred to me, it's like, oh, this is like that. I can do it like this, right? And that skill is actually going to be more and

10:04 more valuable cuz like cuz there's going to be weirder and weirder problems. I I mean, I I buy that, but again, I feel like the actual nature of learning is changing so quickly because of AI and how people learn like not just tech technical skills, but any any skill. And I will admit like I do think of this partly as a parent because um

10:26 you know, I have a kid who I thought would be you know, worst case scenario, work from home as a kind of coder by the hour. And those jobs are already gone. Like they're gone now. So, those jobs are not going to be there. So, you know, what what should people invest in learning and what are the like learning strategies that are going to give somebody some longevity as the field of

10:53 um technolo not just I don't I was about to say software development, but it really goes beyond software development as like all of these tech jobs get totally reimagined as AI becomes a bigger and bigger part of the production process. So, the funny thing is like again, I don't I don't know that you need to like over optimize up front, right? If it's something you hate, like I don't think you should force yourself to learn it

11:16 for the most part, right? And because like as the world is progressing, not only is more stuff automated, which is like one thing, but it's it's becoming easier and easier to learn stuff when you need it. Yeah. Right? I got my four-year-old a little stuffed dinosaur. There's a startup in San Francisco that's making these stuffed dinosaurs and it's like an AI stuffed dinosaur, right? And it's fine-tuned for kids. You can like dial your child's age, and it'll talk to them like at that age. But you can ask

11:39 anything you want in the world, and it will it will explain that to you, right? And I didn't have that when I was four, right? But he he asks like difficult stuff, and it can explain it in kids' terms. Like so, if you want to learn assembly language today, you don't have to go through what I did, where like every [laughter] every compile took like 5 minutes, and if you make one typo, it's like, "Whoops, another 5 minutes." Right? Now

12:01 it's like instantaneous. You can ask Claude, "Teach me assembly language, right? Quiz me on assembly language." And you can learn what you need to learn so much faster. So it's so much more important to just like, "Look, be interested in stuff. Learn what you're interested in." Because everybody in the world is like suddenly been up-leveled like two levels, right? Where you weren't a programmer before, like now you're by default. Everybody in the

12:24 world is suddenly a programmer. Just install this thing, and 5 minutes later, you're writing programs. Like that is not obsolete. It's not like you your skills have gone away. But if you know things, you're up-leveled even more, right? The more stuff you know, the more stuff you can do with the same tool. I I I mean, I'm I'm not 100% sold on that, because one of the things that we're observing very quickly is this phenomenon of cognitive offloading, where

12:47 because AI can do these tasks for you, you don't really it it's sort of like a veneer of learning rather than real learning. And again, if you think about that different like those stages of the first job, where maybe you can get by with that versus where you're going to be 10 years into your career, and with the with the hopefully the goal of managing projects

13:10 or managing teams, if you skip over that deeper learning in those earlier stages because AI is kind of answering too quickly, um then you have to make up for it later. And so I'm wondering what you know, what what do people do to challenge themselves in those earlier stages so that even if they're doing kind of the wrote parts of of tech projects, they're building the skills

13:35 that are going to support them over over time. Yeah, I mean I I actually I don't really believe in cognitive offloading as a phenomenon. I think people said the same thing when calculators came out. Like, "No, you need to learn how to do long division on paper." Right? And it's like, you know, I learned that in grade school. I have never once done long division on paper since grade school. And and my my brain has not atrophied, right? But what's what's what's really

13:59 dangerous, the thing that is is truly bad for you, is so-called decision fatigue. Right? And so the danger they talk about this with self-driving cars, right? If you have a self-driving car where you like a supposedly self-driving car where you must keep your hand on the steering wheel because every now and then it's going to make a fatal mistake that would cause an accident and it's your job to prevent that fatal mistake. Right? You're going to be like, "La la la." I'm thinking about something else.

14:21 You're not actually paying attention the way you would be if you were actually driving the car. Yeah. Right? You're only left to be like super hyper alert supposedly for this like one of hundred chance that it's going to screw something up. And you're not going to be paying attention and that's when it gets fatal. And this happens like this can happen if you apply AI to all the supposedly easy stuff or the low-level parts of the job, but you're constantly it's popping up these like

14:44 benign questions. Yeah. Right? And like Claude Code does this when you don't run it in dangerously mode, right? It's like, "Hey, can I do this?" Yes. "Can I do this?" Yes. "Can I do this?" Yes. Here's a 10-line bash script. "Am I allowed to run this?" And I'm like, Yeah. Yes. Right? [laughter] I know. you're not actually making decisions anymore. Now your brain is just like turning to mush. Right? But you don't have to run it that way. Right? What you can do instead is you can have it

15:07 eliminate a bunch of stuff and only bring you the things that are important sometimes, right? But when they come in they're like, "Oh, that's an interesting question. I hadn't thought of that." Right? And that is the opposite of your brain atrophying. It's like, "Oh, that's an interesting question that I I have never even thought of because I was too busy writing lines of code, right? I I'm I So So I totally agree with what you're saying and I also observe that not everybody opts to keep challenging

15:31 themselves. And you know, this is one of the things that I find interesting about, you know, having just written a book about neurodiversity in the workplace and really seeing how um the kind of prototypical programmer brain, you know, people have I mean, I think this has changed, but people used to go into software with a very certain kind of like problem-solving mentality. And so now, you know, you can apply that

15:56 problem-solving to higher order problems. But for folks who were entering the world of software development because it was less secure job and not like an itch at the back of their head, there is that ability to go on autopilot. So, you know, how do you encourage developers, I don't know if we're talking about on your team or people you're talking to, how do you know when you're in autopilot and how do

16:20 you know when you're continuing to challenge yourself? What are some habits you could put in place that ensure that continued growth? Yeah, so one thing I think there's research now that shows this, we saw it in our team as well. Like there's really it turns out there's two kinds of software developers. There's the kind that learns software development cuz they really like typing lines of code into a computer all by themselves in a room for hours at a time. It's really meditative and it is really meditative.

16:42 I love that process, right? Like when I got into coding, I'm like, I love this. It gives me an excuse A not to talk to any people and B I'm doing something useful and it is it's so quiet and I can clear my head and I can get some stuff done. And some people that is the goal is to like have that feeling all day, right? And it's great that you can get paid for it. Um and in the early days of computing like famously people who like people were like, "Huh, I can't believe

17:05 they're paying me to like babysit the mainframe at the university when I would obviously do this for free cuz it's so fun, right?" And then the other kind of person is like, "I just like solving problems, right?" And so the the people who just like meditating at their computer, they are a little bit at risk, right? Like let's be realistic, there's going to be less jobs meditating at the computer because that's actually the thing that isn't using your brain. It's

17:30 meditation is like the opposite of using your brain, right? It's like how do I get the rest of the stuff out of my brain? Stopping and having to ask the like hard questions, like what is the problem that I'm trying to solve and how do I do something useful to solve this problem? Those questions are scary for some people. Now, I found out luckily for me, uh I'm in category two. I just It turned out I just really love solving problems. I actually don't miss typing

17:52 code into a computer at all, right? I thought I would. I thought this was like my whole identity, but it's like, nope. What's really fun is like, oh, I identified a problem, I can create a computer system that will solve this problem and then a whole bunch of people benefit from the thing I created. Like that's awesome, right? But I have to get my meditation somewhere else, right? Like that that part of the job is not there. And I know we have people even at our company that like their identity is,

18:15 you know, this is what I am. I'm a programmer who types code into a computer and I love it. This is being taken away from me. It's like it kind of is. Um and there are there are nevertheless programming jobs that AIs cannot do that you can still do. Uh but it's that's that's where you're a little bit at risk. If there's a specific thing you just love to do over and over again, I don't know. But solving problems is never going to be obsolete.

18:37 It's it's it's a really helpful distinction because I I just had this conversation with a young developer recently who was basically saying, I don't want to do by coding. I like the sitting in the meditative and I was just like, well, I'm I'm sorry you were born 20 years too late for that career. Like I don't even know what to tell young people in that Yeah. What do you do? Do you just

18:59 totally change fields or Yeah, well, I think, you know, getting a little abstract. Like, you know, I still do spend my time quasi meditating. I don't I don't meditate in the official sense of like sitting there and listening to your particular kind of music and folding my legs in a particular way or whatever, [laughter] right? But like sitting there and thinking is is suddenly an extremely valuable skill. Yeah.

19:19 Right? And it's like hard for me as a CEO cuz usually my calendar is filled absolutely to the brim with meetings, but sometimes I just have to clear out meetings for like a week. Yeah. And my job is to sit there and like process all this stuff and like have like one insight. It's like, "Oh, this is the thing that will solve the problem." And the the neat thing now is that like some for for many people that one insight is like, "Okay, now I can bring this to Claud and it can come true

19:43 an hour later, right?" Oh. And before it's like, "Well, I had this great insight. Now I need to build a company to build the thing so I can tell people to set up a team so that they can solve this problem in 6 months or a year, right?" And it's like the the distance from like you can have that meditative state to like I have this brilliant idea that now has come true. It can be like a day. You know, but I mean it's So so

20:06 the the flip side of that as somebody who used to not be able to do all my crazy ideas is now you can do all the crazy ideas. Like because it's so easy to make the thing, it's really easy to make like an endless array of crappy software products. I mean, it brings us back to our paper clips. So again, if the goal is to have developers who are capable of creating and delivering

20:32 actual functioning software that breaks after 20 bends instead of two bends and that actual human beings might want to use and that aren't just like a stick-a-fantic AI's like idea of good software. How do you as a as a developer who's working with your 100 LLMs on a day-to-day basis and not in the guts of the code,

20:54 you know, what do you think are the most fundamental um abilities to cultivate so you have that kind of judgment? So like you know, where do where where do How do engineers become great engineers? Yes, that. Experience. Right? I've been programming for like 40 years. And like I've had to go I've gone through different company or different companies, different teams, different jobs, and like building stuff over like 2 or 3 years, and then after 2

21:18 or 3 years, we finished building it, we send it to the customer, and we find out all the stuff we did wrong. Yeah. Like that's that is a slow learning process. Yeah. Right? The cool thing about LLMs is that the people at Tailscale are doing this right now. I I know this, right? They're like, I want to build this thing. I don't know how. So, I'm going to try 10 different ways of building this thing. Yeah. Right? I'll ask Claude, like give me some ideas for how we might want to build a product like this. And it gives

21:40 you 10 ideas, and I'm like, okay, I'm going to open 10 windows, and I'm going to have Claude build me 10 things. Yeah. Right? And then I'm going to compare to see which one's better. And I just got 10 years of experience in 1 week. Wow. Right? And I've tried all the different things, and I know the pros and cons, and like this is how you become a good engineer is you try stuff, and you see what doesn't work and what does work. Like if you want to experiment with paper clips, I can now try like building

22:04 new kinds of paper clips in a virtual world Yeah. that I never like never could have gotten funding to even experiment with this like way out there method that everybody thinks is going to fail, right? So, when you make things super cheap, like yes, you're going to produce lots of garbage, Yeah. but you can finally do all these experiments and find out which things are surprisingly not garbage, right? One of the worst things about getting old is realizing I actually don't take as many

22:28 risks as I used to when I was 20 because now I know why things are going to fail, Yeah. right? And so, I used to assume they're going to fail, and then I don't do them. And then some startup person who's 20 doesn't know this, and they start a company is like, yeah, well, that would have failed 20 years ago, but the world's different now. That thing you thought was going to fail isn't going to fail, right? And you can find this out so much faster. Like that's how you gain this experience.

22:49 I I really appreciate your perspective. I want to go home and build like 50 pieces of software right away. Um and and thank you so much for sharing with us your perspective on, you know, what it means to be a developer in this world where now the tools are so different for us. Thank you. Yeah, thanks for being here. e.

Web Summit Vancouver 2026: open source in the agentic era

Thu, 21 May 2026 00:00:00 +0000

AI is the cause of, and solution to, slop problems. Have AIs filter your PRs so humans get back to deciding what they want to exist.

AI was built on open source. Now it’s starting to return the favour: finding bugs, writing fixes, and optimizing code faster than humans can review the diffs.

At Web Summit, I sat down with the CEO of Cal.com to talk about what happens when AI agents and open-source communities start improving each other. The bottleneck may soon be code review, which is at least a problem we recognize. But there is a fix! You guessed it. It's AI.

If you'd rather read than watch, the full transcript is below.

Transcript

0:05 [music] Hi everyone. Welcome back from lunch. We're here to talk about open source and I really wanted to launch it with you Avery just because Tailscale is like you guys have embraced open source and and and you sort of have

0:28 come to sort of represent the adoption of open source for a major company like yourself. I was just if you could describe the moment right now in this sort of a genetic AI everyone going crazy staying up all night. How that affects and how you feel about that in your company? In my company Tailscale maybe ironically has always been a little bit of a late adopter and

0:50 we're intentionally a late adopter of AI. It doesn't mean we don't use AI but we use it really carefully and that's because Tailscale is a is a network security infrastructure project. It's used by Fortune 500 Fortune 50 companies and we can't like it would be violating their trust in us to take too many risks too quickly when their entire network security depends on our company, right? So we're being careful and I think there's a careful way to adopt AI that's really

1:13 really productive that I think people are overlooking. Now in a slight sort of plot twist is that at your company Tailscale is actually you were open source and then very recently two months ago to keep things a little bit spicy you did a major pivot. If you could describe that for us. Yeah, recently we went kind of viral when we went from being

1:35 historically an extremely open source company. We've always been huge like open source evangelists. We actually decided to go close source due to sort of security concerns because you know this stuff rapidly changes as we all know as all we sit here and discuss and for us we thought that the environment was a little bit turbulent to be able to safely operate in and much like Avery said, for us, we also have a duty to

1:59 protect our customers. And that led us to kind of perform a bit of a risk assessment of that. Maybe if I can just drill down a little bit on that. Like how did this how did this change happen? Because maybe it can help us sort of explore the open source experience right now. Like what happened? Was it Was it Was your customers are coming to you and saying say hey, wait a minute. We're giving you all this data. What's going on with it? What what what was the story behind

2:23 behind the change? familiar with like the changes recently you know, AI has become a lot more useful. It can now build software a lot more effectively than it could before. Like vibe coding has been around for years, but only in the last like 18 24 months did it become something which we can actually use for production ready cases. And just as AI gets better at building software, it gets better at breaking it. Yet the

2:47 problem is is AI isn't developed enough to the point that it produces the most consistent answers. You know, you get different answers to how many hours are in strawberry. You get different reviews to the same PR whether you plug it into Claude or um you know, a GPT model. And the problem for us is that with AI able to break software and find vulnerabilities,

3:10 we can't seem to find a single source of truth which can determine if an application is secure or not, which leads us to that kind of uncertain and turbulent um atmosphere. Every what I'm tempted to say how do you respond to that? I mean what what cuz he is describing a world that we all know, you know, hallucinations, bugs in the code and all that kind of stuff, but maybe it doesn't need to be that simple. It's not

3:33 a It's not a just sort of all in open open source or not. Yeah. Well, I mean, it is let's be honest with ourselves. Absolutely true that AI does a bunch of weird stuff. It's super gullible. I've described in the past as like, you know, hiring a really smart but not very worldly intern, right? That can code very fast, but it makes mistakes, right? It makes really scary mistakes. And then

3:56 you can clear its context and ask it to review its own code, and it'll it'll find those mistakes and say, "Wow, whoever wrote this, I don't know what they were what they were thinking." Right? And so, you you need to build systems around around these things to make them not not you know, to make them functional, right? And the open-source world is experiencing that in particular, right? Where, you know, a whole bunch of projects suddenly appearing cuz somebody vibe coded

4:19 something they thought was cool overnight. A whole bunch of people are like, "Oh, I can fix my favorite project by modifying it to do something. I'm going to send in a patch." But they haven't actually reviewed it, or they don't even really they've never coded before. They don't even understand what this patch does, and they're the maintainers of the project are just like, "Look, there's there's a hundred of these. I don't have time." So, I mean, some people talk about like the the the hundreds of these things,

4:41 thousands of these things. GitHub is just exploding with what, you know, some people say is like AI slop, you know? What There's just so much out there, and then so therefore, you know, engineers and tech you know, technical people are just stuck going through it all, through it all. And then then you're stuck in this moment of sort of like what what was this all good for? I mean, how do you address that kind of that that that sort of issue of this sort of like having just this just a massive surplus

5:04 of of of product out there? Yeah. I think I mean, one of the things I observed that I think people don't necessarily realize is that AI is at least as good at reviewing code as it is at producing code, right? And there's kind of in the open-source world, with this AI slop, there's this interesting like it's a you can think of it as a giant distributed system of people producing the slop. Anybody can do it

5:27 and upload it to your GitHub repository and make your life as a maintainer miserable, right? But you can also get an AI to review those incoming PRs and tell you whether they're slop or not, and sort of automatically disqualify them if they're slop. But that would require the maintainers to set up some stuff, and And like expensive time that hasn't really happened very consistently yet. People are only just getting

5:49 started with this kind of automated review process, right? But you can do very good code reviews. In in the the like extended version of this, you can have automatic code reviews of PRs coming into your project, and then the person who's bought uploaded the the code can actually read the code review and then fix the code, and it can do this in a few cycles until eventually it gives up because it's always going to be slop or it's actually come up with a pretty good change, right? And once

6:13 you've got a pretty good change, then you should bring the human in to say like, "Hey, this is a pretty good change. Do you want it or not based on the vision for your project, based on your design principles, based on these other things?" And that's the opposite of the kind of exhaustion that we've been creating today, right? So it's all about the processes that you put stuff in. Um Bailey, we had this moment of the the Mythos moment very recently.

6:35 Um I just wanted to for you to sort of like in in in the work that you do, this has there really been this uh we've heard about the Mythos and and about, you know, Firefox like where you said they discovered like these these these um these these fallibilities. I was just wondering if you could just sort of are you in agreement that this is definitely a moment where there are new sort of security dangers out there um

7:01 that there weren't before? Yeah, I mean Mythos scared us all a little bit when they found vulnerabilities across Firefox, FreeBSD, and you know, a ton of other things. And you know, as AI models improve, they're going to find better vulnerabilities. There are some extremely complex and intric- uh intricate like exploits that can be done. And some which are so hard that, you know, humans

7:24 may not be able to do that. And while Mythos like they created a monster, really. Um and they decided like let's gatekeep this and let's try and like roll it out in the the most cautious uh way. But like this is AI. Anybody can innovate in it. Um Um, know, we have frontier models coming from American companies and then suddenly China comes out with DeepSeek and it's literally

7:46 open source. You can just download a state-of-the-art model. Now, what happens when China comes out with a Mythos alternative? Does that mean that now every single person that can, you know, self-host an open-source AI model now has the ability to hack into almost any system on the planet including some of the most secure and open-source um projects that exist?

8:10 That's a very good question which I'm going to put to [laughter] Well, uh I think I'm actually more optimistic than that. I think the timing is really important, right? There's the attackers, the red teams, there's the defenders, the blue teams, and there's what I would call the arms dealers who are selling tools to both sides, right? And I maybe they don't want to think of themselves as that way, but they they have to, you know, they built this incredibly

8:31 powerful tool that can be used for good or it can be used for evil. And I respect Anthropic a lot with Mythos in particular. Like they've been holding back on availability of this thing probably for multiple reasons probably including for marketing reasons, but it it it finds real serious security holes in real critical infrastructure software. And if they'd given that to the bad guys first, those bad guys would be exploiting that software before we

8:53 have a chance to fix it and we would be in really big trouble, right? The fact that they didn't do that, the fact that they're giving us a chance to like, hey, we can fix our stuff first is like is a gift. And then the question is like, well, okay, as the models keep getting better and better, are they just going to find more and more obscure security holes and it's never going to end in this like sort of downward spiral into doom? Or are we actually just going to like is stuff just going to get more

9:15 secure and not really hard to attack? And I think it's actually more like the latter, right? Like most software in the world today has never had a security review ever, right? And even if software that's been security reviewed gets reviewed once or twice a year, gets pen tested occasionally, some of the PRs get security reviewed cuz they're considered to be sensitive and some Some don't, right? So like most code has just never

9:37 been looked at. And now you have these tools that can look at it constantly. Every single change you make can look investigate and and try to find one of these security holes. And so, I believe that if we all get on board with caring about security, which is which is a stretch. But, if we do, we can secure our software. Uh getting people on board to to to worry about security may involve

10:00 regulation or may involve governments. Is that sort of something are you um Baylor, are you reluctant to see the government taking too much of a a role in this or is that something that you think there might be a way for them to to there there is a place for for regulation here? There might be a way. I think, you know, in answer to what you said, I don't have access to me those. Something exists out there that may or may not be able to completely break my

10:23 code base. I don't have a good opportunity to defend against it. And I think much like how, you know, the government has created structured and regulated markets which enable, you know, like each party in the the stock market to be able to to trade in a fair way. Um there might be opportunity for, as long as they don't overreach, um to try and like equalize this stuff.

10:46 That might be through export controls, that might be through um you know, laws and acts. But, uh yeah, I think there's definitely a little bit of uh equalizing that needs to happen. Um at the opening of the of the Web Summit um uh Paddy Cosgrave sort of talked about like these two worlds that, you know, there's the open source, you know, there's those who say that open source is already won and it

11:09 is the way. And then there's the others who saying that, you know, the the the US frontier models have have have already won and it's the way. Um the reality is somewhere in between there probably. But, I was just sort of wondering um on the sort of big sort of big picture level, every where do you sort of see things right now? Well, I think I mean, I think all the models have their place just as in any economic system. There's going to be the premium product and there's going to be

11:32 like all the levels down to the mass-produced product, right? And I think the most expensive tokens from the most expensive models are going to get more and more expensive. That's what I think will happen, cuz there's a shortage of GPUs, and like if you want this good stuff, you're going to have to bid for it with everybody else, and the price is going to get eye-wateringly high. You think it's high right now? I think it's going to get even more high. But at the same time, the price of like the tokens that are like a year behind

11:55 that are going to are going to crater, right? And that's that's going to be really exciting. Like pick the tokens we want. I think from a security point of view, that's going to be like if you're building infrastructure software that everybody depends on, the Firefoxes, the FreeBSDs, the Linux, the Chrome of the world, and I guess the Tailscales of the world, you're going to have to pay a lot of money for like the best security defense, right? Uh what I think about open source software, like, you know, a

12:19 lot of the stuff that we're all going to vibe code for ourselves or for the five people on our team, right? You're never going to be able to afford to run the best frontier security software, for example, to review that. You have to do something else, right? My my proposal is don't put it on the public internet. It doesn't need to be there, and if it's not on the public internet, nobody can attack it, right? That that's the best approach unless you're building these

12:42 fundamental front-end-facing public products. Um Bailey, token maximization, you know, we hear these stories about, you know, like at the big corporations or people are told to spend as much as possible, you know, what is it spend? To use as many tokens as possible, which ultimately means spending. I'm I was just sort of wondering, how does a company like yours approach like, you know, token you know, just using up of tokens? Is it like

13:05 go crazy, or like how do you try I mean, how do you try to how do you navigate between like showing that you're utilizing these tools to their fullest potential, but at the same time, you know, staying afloat, you know? I mean Yeah, I'm a little bit against the whole like token maxing hype train. I think, you know, a lot of things are hype trains in this industry. Even to some degree, people want to be like open source just because it's the popular

13:27 thing or not because it isn't. And it's really about like using what's sensible for you. Now, I do completely agree with Avery on the whole thing about being a little bit sensible and a little bit cautious about AI adoption. Um, do I think that like everybody should be spending as many tokens writing as many lines of code as possible? No. We've known for years that like lines of code does not equal output. Um,

13:51 and so I think really you want people to be AI-enabled. I can do a better job with AI. Everybody on my team can do a better job with AI. But, you know, otherwise like let's keep it sensible. Right. Uh, Avery, you know, to people in the audience, um, you know, people that are, you know, coding, who are coding for a living or thinking about coding for a living, um, what is sort of your advice to them in terms of, you know, being in

14:17 this world where, you know, some people say AI and they think it's a magic wand and then something magically just appears and it's fit for purpose. Uh, there's still a a major role for, you know, the good old human being in the story, isn't there? Yeah. Well, rather than a magic wand, maybe I I can compare it to a genie, where you make a wish and you literally get what you ask for. Uh, and it turns out not to be what you wanted, right? Cuz that that's what

14:39 happens over and over again. It's like it creates that it is magical. I get it's magical in the most literal sense of like we don't even know how it works. Even the people making it don't know how it works. And it and it grants wishes, but it grants them in like strange ways that you might regret, right? Uh, and I but I think, you know, it's a tool just like just like a human is a tool. Humans are magical, right? You ask them to do something and you don't always get exactly what you wanted. And sometimes

15:01 it's good and sometimes it's bad. And we've got tens of thousands of years of building human society around the fact that humans are magical, right? But I think, you know, society exists to serve humans, right? It doesn't exist to serve computers in a like running mathematically calculations that simulate humans, right? And so it it's up to us to do what we want with the tools that we we

15:25 have, right? And these these tools can be used for anything, right? They can be used for attacking, they can be used for defending, but they can be used for like writing a bunch of slop code, but they can be used for defending and fixing slop code without humans having to be involved. So then we have to up-level ourselves to this like I'm going to think about the abstract. Like okay, I actually have this you know, this this patch came in and maybe a bot wrote it, maybe a human

15:48 wrote it, who knows, right? But like it works now. It's I've verified that it does what it's supposed to. I've even verified it doesn't have security holes. Is it what I want to exist in the world? That's what an open source maintainer will have to decide, right? And that's the that's the fun part of being an open source maintainer. That's why we get into it, right? When once we start, we realize it's like even before AI, it's like 90% like jerks writing

16:12 angry posts in your issue tracker or like triaging stuff or like slop that might or might not have been AI generated PRs, right? But like the 10% that was fun is like working with people, building something cool, and having someone send in something that makes your cool thing even more cool, right? And that can be the whole job because we can eliminate the tedious part using these tools. But it's the human's job to decide what they want to exist in the world. That's what we all

16:35 work and be are going to be able to do with these more advanced tools. Billy, do you have I mean what what is your perspective on that too? I mean in terms of like, you know, you may not be like the most senior, most most experienced like, you know, programmer, but suddenly these vibe coding tools like allow that person to to to explore ways that they they couldn't have done before. Yeah, I know for us we said we're never going to fire anybody

16:57 because of AI. I mean we never over hired the team in the first place, but now we just expect people to be able to to produce more. Um you know, like I said, this doesn't have to be going crazy at it and you know, uh overusing tokens just for the hell of it, but you know, it's it's a very exciting time to be in open source, to be in anything, really. The world is rapidly changing each week

17:21 to the next. We have more and more capability. And, you know, overall, this should be able to be a positive that we can all use just to build more things and, you know, invent. All right, cool. I think we have to we have to leave it there. Thank you very much, guys. That was really cool. Thanks a lot. Thank you. Thank you.

Web Summit Vancouver 2026: AI needs an Android-like ecosystem

Wed, 20 May 2026 00:00:00 +0000

AI needs an open ‘Android’ to balance the vertically locked-in ‘iPhone’ of trillion-dollar AI companies.

Open-source AI is closing the gap with closed systems faster than many people expected. That changes the economics, the security model, and the assumptions behind a lot of Big Tech’s AI spending.

I joined a press conference at Web Summit to talk about what happens next: who benefits, which business models get squeezed, and how regulation should work when the most capable systems aren’t controlled by one company.

If you'd rather read than watch, the full transcript is below.

Transcript

0:01 Hi everybody. Who's going first? How was lunch? Ah, no lunch. So, I do want to say this is a really broad topic and so we are excited to talk about what you are excited to hear about. Uh so, if there's anybody with prompts, uh feel free. Otherwise, we'll do it like short monologues and then they'll go for some questions. Maybe maybe I'll kick things off. Sure.

0:25 Okay. So, um by way of background, I'm I'm Mike Conover, I'm the CEO and co-founder I yeah, sure. You guys can hear me. Project. I'm the CEO and co-founder of Brightwave. We built a a deep research system that is able to perform effectively the functions of an investigative journalist, but um in complex domains like finance. And

0:48 I'll just talk a little bit about some of the trends that we're seeing with respect to open source AI and um agents in particular. So, if you think about Who in the room has used this cloud code? Right? So, typically you're operating one to three agents at a time simultaneously and I think what we're going to see is massive parallelism, which is that

1:11 you're going to like if I want to understand how the war in Iran impacts commodity markets, there many dozens of different sub-themes and within each sub-theme there are sub-research topics and the ability to parallelize and scale um how many topics am I running down, how many analyses, how many code changes am I running at the same time um increases the number of agents that one person is going to control. And if

1:32 you look at OpenAI's symphony, uh this is a system for making that abstraction where I'm less aware of how many different agents are operating on behalf, um you know, clearer where I'm moving tickets through a linear board. Um This on top of sort of the task horizon link, so that's like the depth, how long can these agents function for independently,

1:54 um gives you this increasing breadth and depth surface area of total compute. And so from a like secular standpoint, that the what I expect is a massive increase in token volume owing to this parallelization increasing depth. Um we're going to see open source models become really competitive on a price and speed basis. So the same 64 GPUs that

2:18 you would use to run um a trillion parameter model, you can get 3x more tokens per second out of those same GPUs running isolated models on individual cards. Um and from a uh sort of switching cost standpoint, this is the last thing I'll say about it before we kind of hand it hand it off to you. Um it's not clear that the fungibility of

2:42 these resources is priced in. Like Anthropic had a moment and I I'm a huge fan of Anthropic, but as soon as a new frontier model comes out or as soon as a new open source model comes out that has some advantage, these all fulfill the chat completion API. And the switching costs are very, very low. Brightwave is all hot swappable so that you can use whatever model is best for the job. And I think that that structural force, that price and and

3:05 speed pressure is is not well appreciated with respect to like how much compute is going to increase through agent parallelism. I think we're going to do the three and I I'm here here to talk about it. All right. Um yeah, I'm Avery. I'm CEO and co-founder of Tailscale. Uh we make an AI connectivity and and governance framework called Aperture.

3:29 Um and like the my my picture of the like AI ecosystem right now, I I think it's very interesting how this is it's got to work out cuz that usually these big technology shifts, you'll have you know, the big winner and then the secondary one, right? And right now we're watching the clash of the Titans uh way up over our heads with the trillion-dollar valuations and the giant data centers and stuff. And the thing is they're all building what I would call like the iPhone of of AI, right? I

3:55 Anthropic, OpenAI, and Google are all building this like vertically integrated system where they're providing all the pieces and they want to lock you into their system so that you pay a lot of money for their tokens, right? I think the ecosystem needs the balancing Android of AI, which is like ecosystem-based, open source, everybody can contribute, and you can plug and play all the pieces yourself. And maybe you have to assemble the pieces

4:17 yourself, and maybe each of the pieces is is not not as beautifully machined and perfectly integrated as in your iPhone, but there's a space for that cuz there's always should be a space for like the super high-end premium thing and the like, you know, flexible thing that you can do at volume. And I think we're not seeing the second one as much yet, but there's a lot of pieces out there that can be assembled into that. And so, that's what at Tailscale we're most interested in

4:41 doing. We want to connect all these pieces together. We want to build that ecosystem. And I think that's where, you know, the lower-priced models, the various different kinds of harnesses, the really complicated agentic systems, the connectivity systems, sandboxes, there's a lot of room to experiment and put all those pieces together. Cool. I'm Bailey, co-founder and CEO of cal.com. cal.com is a scheduling

5:03 infrastructure platform. So, we power scheduling from anywhere from individuals to very large businesses. I have a slightly different perspective to to these guys. We were historically very open source. We were like the largest Next.js open source project. And recently we made a move to go close source due to security risks. So, obviously we're all aware that AI can build things even better, but they we

5:26 believe they can also break things even better. They've become really, really good at detecting vulnerabilities and things like that. And the problem is is AI is still somewhat in its infancy in the sense that they sometimes give inconsistent answers. And so, for us, you know, AI is never going to give the same answer to I don't know how many hours are in strawberry

5:49 or you know many things like that which we all know are quirks of AI and that also means that AI can't give you like a single source of truth as to is software secure. So you've all heard probably about Anthropic's Methuselah model and all these sort of things that are able to break things more and more. We don't have Methuselah none of us here but you know somebody does and you know AI isn't gate capped anybody can

6:12 innovate you know we have our US frontier models that that lead the way and then one day Deep Sea comes in and suddenly they can they can match that. What happens when China's now has a model which can rival Methuselah that means that you know are all of us under under attack because as I'm sure you've read Methuselah is breaking you know Firefox FreeBSD all these things

6:36 that we consider to have like a lot of eyes on them as open source and especially like FreeBSD is is an absolute sort of like staple of you know stability and so for us you know while going closed source isn't unto itself like a a solution it is an option we have on the table which we believe can reduce the risk.

7:01 We run six AI code vulnerability scanners all in parallel they all find different things and um you know that's a a scary thing for us. We also spoke to Hex Security one of the the big ones that that were like a YC company and they said uh open source is five to 10 times easier to hack than closed source and so for me

7:25 the reality becomes pretty clear that if I can make cower.com five to 10 times harder to hack um although that is not a complete and holistic solution to this it is definitely an option that I feel like we have to take to protect our customers. So, yeah, slightly different perspective to to these guys, but Can you please state your name and

7:50 affiliation? Jim Harris, Corporate Knights magazine. Uh just like we have hybrid cloud and multi-cloud, I think we'll have hybrid uh models, multi-cloud models. So, some things will use large language models, medium, very small, niche. And similarly, open source, closed source. So,

8:13 uh where we choose to put that load or query will depend on the context or nature of both the data, the security considerations, the cost, the speed, whether we use open or closed source. So, uh this is the vision uh that my clients are are taking, those I talked to, to optimize both cost, speed,

8:39 safety, all these considerations. Uh would you agree with this view that that's where we're going? While the you know, Open AI wants to lock you into their vertical stack, many companies uh don't want that. Just as AWS, Azure, Google wanted to lock you into their cloud. So,

9:03 Yeah. I think I I like my iPhone analogy uh for that one, right? Like every year a new iPhone comes out, they raise the price by a little bit. Uh everybody when I remember, I'm pretty old now, when the first iPhone came out in like 2007, it was like $800 USD, and everyone's like, "Oh my god, who's going to pay $800 for a phone?" Right? A bunch people did, right? And the price has gone up from there. And like, "You know what? That phone

9:25 is a perfectly fine phone even today, right? Nobody wants it because like we're willing to pay the premium for like a slightly better phone, right? And there are going to be people willing to pay a premium for these slightly better tokens because they believe it gives them a competitive advantage. It's going to come up a lot in the security world, right? Where if you have a slightly better model for finding security vulnerabilities, you have an incredibly big advantage over the person with the

9:47 second best model, right? For a lot of stuff though, that's not the case, right? We have a lot of communications technology. I have a watch that is more powerful than the phone from 2008, right? And all it does is tell me the weather really badly. Like it it can't even keep up with the weather updates, right? But it's okay, you know, I wear the watch and you know, it doesn't it doesn't cost me as much for my cellular subscription, right? So absolutely, it's

10:08 going to be a big market. There's like you're just buying these commodities of different values, right? And I think it, you know, there's as the price goes up, I I firmly believe the the cost of the most expensive tokens we have not seen the ceiling and won't be for a while. It's going to it's going to make your eyes bleed how expensive the most expensive tokens get, right? But the cheap tokens are going to get really cheap, right? And that's going to be both of those things are

10:30 going to be really exciting. Especially once like the VC money runs out. Um you know, it's it's like Uber. Uber was dirt cheap in San Francisco when it first came out and then now like the VC money dried up. So I think it's exactly what you say. Like the the expensive tokens will get even more expensive, but because of open source, like if you can, you know, self-host DeepSeek, uh and you can get it through any number of the inference providers, they're all

10:52 competing like on the way to the bottom. Um so yeah, I think you're also going to have that price discrepancy. And then there's also just what's the best fit for the job. Something that's often overlooked is like we look at, you know, benchmarks and overall intelligence scores, but say for instance for working with legal contracts, Claude, even though certain things outrank it on the intelligence index, Claude is better at that like long-form like understanding

11:15 the nuances of every word and and things like that. So I think for like AI to to become truly dominant, it needs to be, uh you know, versatile in terms of what provider you use. And we see that because there's like Vercel's AI SDK, you've got Open Router, and all these different sort of like switching things, where you can have the same, you know, core API function, and it will just

11:38 route it to whatever provider you want. [clears throat] All right. I'll check up there for a bit happier news. According to Forbes magazine back in 2025, 42% startup have failed in the Silicon Valley. And uh That seems low. Yeah. It could be more.

12:02 Maybe they're in the process of filing bankruptcy. Uh and I'm not sure you know about builder.ai. They ran through 450 million dollars, and they had to file for bankruptcy, too. You all of you are in your AIs building your AI companies. What is the moat? What competitive edge do you think companies have these days while they're building their product?

12:24 Because to us, it sounds like everybody's trying to build the next frontier model. But where do you think it's a competitive edge? Is the data set? Is it the privacy, security? We'd love to hear your feedback. I'll weigh in on that. Um My Yeah, thank you. Um So, I do think

12:46 Are you familiar with the bitter lesson? Like the idea of the bitter lesson that like effectively like more data and more total compute subsumes all bespoke like classical natural language processing a good example. There were a lot of methods for like vision models, a lot of methods for like detecting boundaries in images or like, you know, faces, and it's like none of those are relevant anymore. Um

13:07 and so that I think is kind of a a large inertial force, where stronger, more powerful models will subsume many of the things that we used to like wire up harnesses for. I think agent harnesses generally are a good example of something that will not exist in 18 months. Um I do think though that product judgement is like it's hard to describe what you want.

13:33 And I think like if you and I were to vibe code workout app we don't necessarily like know how to articulate like all what are all of the things that a person would need from a tool like that or cal.com. Like I imagine that there are a lot of decisions that you've made that like if I was like I I need calendaring software my ability to articulate that and create something a delightful experience would

13:56 be low and I I do think that you know it's like taste is one of these things which is like how quickly can you gather information and make judgements and articulate that to an AI system. Um I don't know that there's going to be one monolithic interface that subsumes all product. Um and then I think integrations like there's a long tail I

14:18 would say that there's a long tail of integrations and capabilities that are not in the call it the blast path of the meteor um that are really important for things like law or networking that um just will not ever be on the like cut list for the foundation labs. So it sounds like you're saying what customer wants uh validating

14:43 Yeah and just like yeah being like being so tight um like in the meta of like what is actually important um and then just creating a really delightful and easy to use product that reflects deep expertise in the subject matter. Um I don't think the foundation labs I mean they maybe they have like they become the one app and they have

15:06 many many different verticals but uh it's unclear that that will be the the business model. I think I think feedback loops uh is what it comes down to like almost all like everything about startups comes down to feedback loops, right? You know the the famous advice to startups is like get out of the building, go talk to a customer, or you're going to build the wrong thing. Right? And and AIs, when they have really good feedback, can produce really good output. And then,

15:31 the quality of when the quality of the feedback goes down, the quality of the output goes down. Anthropic did a project a few months ago where they implemented a full C++ compiler by providing it with a test suite of like 50,000 tests. And they just said like, go. And they spent like I think a million dollars in tokens over a weekend, and it produced this perfect compiler that passed all the tests. And they're like, that sounds very impressive, but like who wrote 50,000

15:53 tests of a C++ compiler? That is the optimal case for this kind of thing, and almost none of us are starting from that kind of perfect specification, right? Even calen- calendars, right? It sounds so simple. It's like, look, I want to display a list of my appointments. How hard can it be? Right? As soon as you as soon as you put it in front of a person, you'll find out how hard it can be. Right? Networking. I like to brag

16:17 that if you ask Claude, like, hey, can I can you make me a clone of Tailscale? It actually tries to talk you out of it. Cuz it knows that networking is really hard. And it'll give you a list of reasons why you shouldn't try to clone Tailscale. By the way, you should just go open fork their open source repository. I can add a feature to it if you want. Right? But like that's the kind of stuff that is that is a moat, right? It's like it took it takes a long time to test

16:39 networking software cuz you need like 100 different devices that it needs to be compatible with, and you can't just pretend to test against it. You actually really physically need those devices to be there to test against. Claude can't set that up for you, at least not right now. This may all just be wishful thinking. It's possible. Any other questions? I guess if I gave Open Claude credit

17:03 card number, it could like have some devices shipped, and then pay somebody to set them up in a data center. Okay, we can then wrap up. Appreciate your time today, folks. Thank you so much for your time. Thanks for attending. Thanks everybody.

Web Summit Vancouver 2026: the modern tech stack

Tue, 19 May 2026 00:00:00 +0000

AI is the universal translator that finally connects all your tools... into a hairball.

Most companies don’t have a tool shortage. They have too many tools, too many handoffs, and too many places where context gets lost.

I joined a panel at Web Summit to talk about when consolidation actually reduces work, where AI helps, and how to avoid replacing ten mediocre tools with one very large mediocre tool.

If you'd rather read than watch, the full transcript is below.

Transcript

0:00 Hello everybody. Good to see you. We're going to have a great chat. We have some super smart people here, so it's going to be fun. We're going to start off with a bit of a joke. I mean, like, you know, I've always kind of laughed at the idea of a tech stack, a company having a tech stack. I think they more often have a blob. Don't you agree? I I would say it's not even a joke. It's uh it's a blob or a morass or like a

0:22 collection. Um like people like the bigger the enterprise, the more programs they accumulate. I think we don't realize like I think the average company buys about one piece of software per employee. I've I've seen the enterprise data and it's like 1,000 or 1,500 different apps that are running in the various places. There's of course the official ones and

0:46 the unofficial ones. Anyways, we're going to talk about AI and consolidating the tech stack. Are we really consolidating it? Are we just massively changing it? When? Well, I think, look, this is happening. This is happening. Things are different now. I mean, you see Salesforce going headless, and you also see that when it comes to AI applications, a lot of the lighter applications getting absorbed into that. So, from where we're sitting,

1:09 what we're seeing is really two things. One, we believe the lighter applications are going to essentially just uh completely taken over by AI. However, the heavier stuff, they're probably going to go uh headless, similar to Salesforce. Now, the fundamental change that we're seeing is um AI, like agentic AI like GenSpark, like my company, [laughter]

1:30 is going to come out and become the new user of software, of SaaS. So, that is the fundamental change. And with that, we actually don't see the usage of SaaS going down. We see it blowing up. Because AI is going to be a lot faster than human beings to use all the SaaS products. But again, the lighter ones are going to become free, are going to

1:53 become just available and customized anywhere, uh essentially in all of these, you know, Agenty AI platforms. David, your thoughts? I think my take is it's a lot like the early days of the internet where when the internet first came out, there were hundreds of thousands of websites that came out for every single topic that you could think of, but it started to slowly consolidate over time cuz you can't remember all the products that you're

2:15 utilizing, all the websites that you want to go to, and you started to go and say like, "Hey, a super center like Amazon makes sense." You go to three different news sites instead of the 500 news sites that are out there, and you're seeing that level of consolidation. I think we are in that Cambrian explosion of AI tools. You don't want to stop people from using the tools that they need, that they want to experiment with, but I think it is going to be really it's going to happen in very short order that you're going to

2:37 see a good level of consolidation to say, "Okay, this is my system of record for meetings. This is my system of record for sales. This is my system of record for CRM, etc." Like, that will happen because you can't have 500 solutions on top of mind, and then you're starting to look at your bill, and you're going to start to realize like, "Oh, why am I spending so much on so many variable things?" It is super interesting because you mentioned already Salesforce came out

3:00 with an announcement they're going headless, which is very cool, actually. Wait, does everyone know what that means? So, if it's not headless, you're going in like a caveman to the user interface, and you're typing in your stuff, right? This is a customer record. I talked to the salesperson, blah blah blah, type it all down. Boring, slow, mistake-prone. Headless, I tell my agent to do it. My agent goes and connects via APIs to

3:24 Salesforce, and just does it for me, which is really cool. And I even talked to the the Salesforce PR people, and you can use open claw with if if if if your [laughter] if your enterprise people let you. Avery, your thoughts on this whole consolidation? Well, I think I mean, any any major technological shift, and this might be the biggest technological shift anyone has ever lived through, right? There's

3:47 going to be there's going to be winners, there's going to be companies that go into decline, and there's going to be a whole bunch of brand new stuff. Uh when you look at big enterprises, they're not going to change very fast. They They when they say fast, you think as fast as you might change. But what they mean is like in the next 10 years, we might slightly shift the collection of software that we're running. The really big thing that I think will change though is like AI is is this universal translator. We actually finally built it

4:12 the universal translator from Star Trek, like not even kidding, right? But it not only can translate between any human languages, it can translate between any software, right? Which is like the ultimate goal of so many IT projects is like can I please connect my Salesforce into my Snowflake database and then give me a dashboard that just tells me this thing that I couldn't get

4:34 from like one of those two things on their own. And it is really hard to do that and huge teams at enterprises have been like tasked to do this and failed. And now you can do it in 10 minutes with this AI. And that that is a huge difference, but it doesn't mean the set of software is going to get less. I think it means all of a sudden you can buy even more software and then connect it all together in this like incredible hairball that can only be understood by

4:56 a computer. Yeah. That's kind of hilarious, right? I mean because there's this yin and yang of software, right? Which is like somebody builds a tool and it's freakishly awesome and it does one thing incredibly well and then they think, "Huh, what else could it do?" And you know, 5 years later you have a suite and you've got a platform and you've got an app ecosystem, right? And so you have this continual expansion and then

5:18 everybody's like, "Oh, it's there's such a general tool. I just need this one thing that does one thing right." And so we have this cycle going on and on. We talked about the SaaS apocalypse when we were chatting earlier and that's affected valuations, right? But you guys are mostly saying that SaaS apocalypse is overrated. Your thoughts, David? Uh it's overrated. If you look at the

5:40 earnings, like DataDog actually was hit by the SaaS apocalypse, stock went down and then all of a sudden they came out and said like, "Actually, our growth has reaccelerated because AI is actually driving revenue for us and it's growing." Now, there's a subset of SaaS companies that have actually been hit where there's companies like uh HubSpot would be one example where they're still growing 20%, which is huge for a $3 billion company. They're growing 20%.

6:03 So, it's not SaaS apocalypse like the money is disappearing. It's more along the lines of your growth rate is slowing down and then the public markets are giving you a lower valuation. I wish my growth rate was slowing down from 20 to 20%. What kind of percentage? So, I actually have a different take on that, John. So, Go for it. I mean, look, if you take a look at the the trajectory of Salesforce stock, I mean, today I just checked, the market

6:26 cap is $136 billion, right? I remember checking this 2 years ago at the peak of the pandemic it was over $450 billion. That was $300 billion gone. Why? I mean, those are facts. So, basically, to what you gents just said earlier, like um I think what David said is um companies could still grow if they could

6:50 create a new value and taking advantage of AI, but if you don't do that and if you just sit there still, then of course, the value is going to disappear because the fundamental change is happening right now in a way that um the the we've all been a victim, in my view, to your description, John. To so many tools, so many SaaS. Everybody said like Software victim. Yeah, yeah, like we're we're so busy.

7:12 Like the world world was so tool-centric. Each of us would have to learn how to use 20, 30, 40 tools. We used to get you know, people used to get certified for being good at Excel. Yep. That world, in in our view, is long gone. Okay? You don't have to get certified for being good at using the software. You just got to be good at understanding what you want and then command your AI agent. And then your AI

7:36 agent would go and learn how to use tools. And and you don't even have to switch contacts, switch tools, like copy and pasting the same piece of business context you from meeting notes to slack to email to presentation. None of that. You tell your Jasper I want this meeting note turn into a prototype and boom it's done. Human beings on the two ends. Show of hands here who who has a AI

7:58 agent? At least one. Yeah, we got some. Who has more than one? More than five? More than 10? Wow, I think we got them there. Oh no, there was a hand at the very back there. Wow, you are super user, ma'am. Uh any comeback there, David? Oh, in terms of meeting notes? Yeah. In terms [laughter]

8:19 Let's talk about the SAS apocalypse. As long as read AI is one of those meeting notes, I'm okay with it. [laughter] Now, I I think people do experiment. Uh people get preferences. There are different use cases for different types of products. Yes. Um what you're just like you talked about, there's more features that we're adding. We're doing more than just meetings. We're doing search. We've got a digital assistant. We've got a digital twin. When you add those things together though, at some point you will go and

8:43 say this is good enough or this is better than the rest and I don't need to pay for all those. Yeah. And especially when you think about the platform-based solutions like uh with Microsoft Teams, with Zoom companion AI, with Google Gemini. Those are all great solutions on a standalone basis, but there is a real narrative to go and say how do you work across the different platforms? And that's what's been important with Open Comp OS like you're able to go in and work across everything. And I think there is a

9:05 scenario where you want a solution that works across everything or you've got to buy three, four, five, six different solutions. Cool. For the panelists on the stage, you have an agent, right? How many agents? Just a number. Uh I mean on Jasper we have One number. How many? 30 plus. 30 Oh, wow. Uh we've got one for 5 million uh MAUs

9:28 on a monthly basis. Uh how many agents, Avery? I just have one agent but Oh, you're agent poor. Well, I'm not impressed. Cuz it spins up agents dynamically. Are you even qualified [laughter] I'm just bugging you. Avery, um when talked about this value in the quote-unquote SaaS apocalypse, going other places, where's the value going?

9:51 Where's the value going? So, I saw a survey a couple months ago from CIOs and CISOs and they said like for the first time in their memory, in the build versus buy decision, they're actually buying less products this year. And in fact, they're going to be planning to spend less money on licenses for SaaS products and starting to build more things internally. And that's that's never happened in the history of of of software.

10:15 So, those hundreds of billions of dollars are going to OpenAI and they're going to Anthropic and they're going to Google maybe and other places like that, token costs. Well, I would say to what Avery just said, we've heard similar things, but that is really largely because what was impossible now is possible. Yep. Building out everything for your company, it was not possible. You have

10:38 to buy other people's solutions, but now, you know, for example, with just by AI developer, you don't have to know how to code. You just need to know what you need and one of the And the beauty of it is is you can build exactly what you need. Exactly. And I don't have to customize my business processes to this Salesforce giant or something like that. I can build exactly what I need. Okay, guys, 12 minutes.

10:59 20% of what Salesforce what Salesforce built anyways. Exactly. you pay for the whole thing? Yeah, people Right. People underestimate the the cost of procuring software. It's not just the dollars of the software, it's that when you buy it, it's actually not exactly what you want, right? You are you're you're satisfying yourself with something that's almost right because it's so much less expensive than building the thing

11:21 that is exactly what you want, right? And there's there's a marketing book that I read, it's from the 1980s and it was talking about ketchup. Right? And at one point, they you know, the the companies making ketchup went into a survey of all the consumers and they found the perfect consumer and they said this is the average ketchup that everybody's going to like the best and they sold that ketchup and nobody liked it. Right? And eventually they figured

11:44 out it's like, oh, you know what? There's a whole bunch of different people like whole that like a whole bunch of different things, right? And now if you go to the grocery store there's like 17 kinds of ketchup and if you go to the pharmacy there's like an entire aisle of toothpaste even though there's only two manufacturers, right? Because everybody wants a slightly different kind of toothpaste. And we do not have that in well, I don't meeting recording apps, right? Like there's a lot of meeting

12:06 recording apps now but there's still not the one that does exactly what I want every single time. Okay, going to transition a little bit here and we're going to talk about the AI native stack and what that looks like and we'll start over here. What does the AI native stack look like? What are we What are we going to move into? So, our viewpoint uh John is consistent with what I just said like the most the

12:29 natural most human way to work is us having one uh point to to to joke, one one thing to work with, one um system interact with, but we get everything done. In other words, you tell that one system what you want to do, it figures out the right place to go, the right software to update, the right thing to do. Exactly. Sounds like magic. Yeah, so today how I work is I text my Jasper Claw every morning, what's going

12:54 on in my inbox? And actually it sends me a morning brief. Like oh, these emails Yeah, you you you exactly. Yeah, you should pay attention. All the others I'll just triage to all your sub orders, right? Like I get off the airplane, I ask my Jasper Claw, okay, where should I go to eat dinner? Like it just figures out where [laughter] I'm I'm staying. need to exist anymore? It could be just your Claw. It's just your agent.

13:17 David, I'll go to you. Uh what does the AI native stack look like? I think it looks a lot like what we're doing now. Um it's going to be within your existing workflows. It's going to be subtle changes that you see. I don't think it's going to be a new portal, a bunch of new software that you're going in logging into every single day. It's going to be, "Hey, you get an email. Your agent pings you and says, 'Hey, here are 10 things that you should do today.'" You decide what 10 things you want to do. You go and you say, "Go do

13:40 that. Go deploy it." You don't even see what's happening on the back end because the agent is negotiating with other agents and other systems of record. I think that's where we're going to go. I don't think it's going to be like this brand new solution. I don't even think the ChatGPT interface, the Claude interface is the long-term interface on how we want to interact. If you think about how we consume content today, it's no longer long-form. It's no longer going to movie. It's on a screen for

14:03 15-second bites that you're looking at and you're swiping up to the next thing. And I think decision-making is going to be a lot like that. And a lot of people say like, "That's that's MARGINALIZING THE HUMAN." OH, WON'T THAT result in a lot of awful decisions? I made a decision on 15 seconds of attention. But it's going to end up with certain outcomes and it's going to be optimized outcomes. So, I'll give you the best possible example. Meta today, all you do is put a pixel on your site, you throw

14:26 variations of creative, and you could spend $4 million and get $8 million at the end of the month, and you don't even know how it works, but it's happening. I think that's the way we're going to go. It's like, "I have a problem. I want a solution against this." I decide between three options, and the agent goes and does that. Ignorance is a business advantage. Okay, I got it. Um Avery, going to you. Uh your input on what the AI native stack looks like.

14:49 So, I think, you know, I I I sometimes I'm a bit of a dreamer. I think this can go in a lot of different directions. With every major technological shift, we have a real opportunity to flush all of our mistakes from the last generation down the toilet. [laughter] You're a dreamer for sure. You have made You have made so many mistakes in the last generation, right? The best thing about AI-centric systems is that they're not really AI-centric

15:11 cuz the AI doesn't care, right? The AI will do what it's told, right? Which means you can finally have human-centric systems cuz the systems we've been living in for the last 15 years are not human-centric. They're like super weird mega corporations running our lives because they made all the software that mediates every single thing we do, right? And now they're not They don't have to be the only ones who can make

15:33 the software. I can make the software that does what I want it to do using the power of an agent that yeah, there's an LLM in the sky that like understands how to do stuff, right? But it's a fundamentally different thing. Like when someone talks about building an AI agent, if it's somebody else's AI agent and if you go to a website and you get a chatbot instead of a human for doing support, it's annoying. If you send your chatbot to the website to do the job,

15:58 it's exciting and empowering, right? And that is the difference. Like I want the ability to make a computer and make the world do what I want, not something that's going to present me with 10,000 options and make me make a decision every 15 seconds. Like those are not important decisions, right? Just I want food and I'm you know, ask me what kind of food I want and then tell me where to go.

16:18 I've wondered for a while now why websites aren't basically, you know, a prompt basically or a prompt insertion text box basically. Like what do you want and here you go, right? Like basically that. But is that the interface of the future? Is the interface of a future a text box? Let's start here. Well, look, I think that that that should be viewed as an option. Meaning

16:42 that when you don't want to spend time to browse through things, you don't have to. I think that option would exist. I I and but I don't think it's just a text box. Today, I could speak to my Jasper agent. I talk to it. Like it is a lot more humane than just typing on a laptop or you know, even typing your phone. I speak to my Jasper cloud through WhatsApp's voice.

17:05 That is how work should be done. That we all should feel like a Jensen Huang or a Jamie Dimon from JP Morgan. You don't see them typing hard to get work done. You see them calling up their associates and they run around for you, right? Like we can all have a fleet of Goldman Sachs analysts running our pocket doing work for us while we focus on the strategic and creative stuff. That's our vision. That's what we all deserve. You were the guy with 30 agents, right?

17:27 Yes. [laughter] But I don't have to manage 30 agents. I just manage one. I have noticed, by the way, that when you do manage agents, sometimes they have conflicts and it's like managing people. It's like that that agent didn't give me what I wanted and that agent didn't give me It's it's it's interesting. It's not perfect. Uh my agents make mistakes and I apologize to our investors because of that.

17:49 I'm telling you it's real. It happened. Crossed emails, but I would say still the mentality is anything new, there are going to be mistakes, there are going to be risks. I decided I'm just going to eat the mistakes myself and just take a full advantage of it. I agree. I agree. Absolutely. David, your thoughts? User interface of the future, is it us just talking? I don't think so cuz

18:13 you don't know what you don't know. So right now, what we've done is we've gone in and said like, "Hey, you've got all your data, you can create a digital twin, you can ask it questions." But it doesn't have enough context about the entire business. If I think what something is a right answer to a client and it says, "Hey, you close the deal at a 60% rate when you answer it this way." I want the knowledge from my rest of my team where it's like, "Hey David, if you actually answered it this way, your close rate goes up to 90%." You need to

18:36 have that information accessible. But I don't know to ask that. I don't know to go and say like, "Hey, did Charlie's call on Tuesday at 4:00 go well? What were the questions?" You want the AI to actually go in and contextualize and say, "Hey, you're a salesperson." And this goes to your kind of point on the meeting notes. You've got a situation where it's like you want the AI and this is what we're doing is you can go and say, "This is a sales call. If it is a sales call, these are the takeaways that I want. This is

18:58 how I want to format it and I want to look beyond just one person's data because I need to get the entire data set." And the best example I have is we've got a fan customer where they've got a product team in Tokyo, LA, and London. And right now, each one of those teams don't talk with each other because they don't speak the same language, they're in different time zones, but they're interviewing customers. Now, what they're doing is actually going and

19:21 saying, "Hey, I'm using AI to cross all the customer interviews. And if someone says, 'I don't like this feature,' that might be it. Like, only one person in my area said that." Well, did you know that 60% of people in Japan said that? Did you know that 75% of people in London said that? I don't know. The prompt is there to ask that question. But now, the AI is going to go and say like, "This is actually a pretty big deal. This should be number one on your product queue." Interesting. Avery, is there sometimes some value in doing things the hard way?

19:45 I've done a lot of data analysis in my life. Sometimes as an analyst, sometimes as a journalist, and sometimes getting the answer from the genie out of the bottle is amazing and incredible. Sometimes I understand the problem better, and I understand the solution better if I do it the hard way and analyze it myself from the baseline data. Thoughts? Yeah, I think I mean, in in

20:07 organizational theory, we have this idea of a core competency, right? Like, what does your business do that is like the fundamental thing that you do better than everybody else? And you should outsource everything that's not your core competency, and you should keep your core competency. And I think that is that is a value we can apply in the AI world as well, right? Like, if you're really, really, really good at something, you should identify what that thing is, and you should not hand it to

20:29 the AI agent to do for you, right? Uh but everything else that like, "Look, I'm not that good at this thing. I'm kind of average at this thing, or maybe I'm just like maybe I'm above average, but it's still not the best thing I'm at." Like, delegate that. And that's the same rule. Like, you know, I'm a CEO of a company. We have 300 employees, right? When I delegate stuff, do I get exactly what I wanted? Like, no. No, that's not how humans work, right?

20:52 They build what they wanted, and hopefully it's more or less what you wanted. But when I go and do something specifically myself, I don't have very much time. I have 1/300 of the time at the company. I better choose very, very carefully which thing absolutely positively has to be done my way. And there's only a very small number of things, but when I do those things, it's better. I think that's a great answer. I absolutely love that. Okay, we have just

21:14 about just less than 2 minutes left. Each of you take a third of that time, please. And the question is, we've all talked about, "Hey, I can build exactly the software I want. I can design my information ecosystem around my desires. I can be very egocentric about all this all this stuff." What about maintenance? What about maintaining code? Let's start here. You got like 30 seconds and go

21:36 down the row. I mean, look, so um the way we're entering in this token economy is it's not going to be overnight, but things are happening fast. Mhm. So, I would say my recommendation is on the one hand, obviously, we still got to get the day-to-day done, but on the other hand, I'd say hold yourself back when you wanted to just apply yourself, but really allow AI to get onto your tasks first for you. Because with that,

22:02 you can get out of the busy work. You can actually have time to think. And you can actually, you know, uh focus on the rather strategic things instead of just hustling. Cool. That is how we Yeah. Cool. David. I think the big thing is you've got to pick the system of records and then build around that with the customization. So, pick what you are going to use and build on top of. Do some customization there. Don't do it yourself. Don't do it something where

22:24 it's like you've got one person on your team working on it. You've got to pick some platform, some AI solution to go in and say, "This is a certified This is an approved process." So, it scales out. A neighbor. All right. I'll say maintenance, when you think about it, digital stuff is the only stuff that doesn't degrade just by existing, right? The out The real world just like falls apart if you leave it sitting there. Software for some reason

22:49 stops working after a while. What is that reason? It's because the platform it's running on when it changed out from under you. I think the neatest thing about AI is you can build your own thing and just tell it, "Please stop changing this." And it won't change. And you don't need to maintain it necessarily. other answer is that code is disposable. And if it if it doesn't suit me anymore, I don't need to maintain it. I'll just generate some new. Anyways, thank you so much for being here. Thank you so much,

23:11 guys. That was super awesome and fun. I hope you enjoyed it. Thanks, everybody.

Every layer of review makes you 10x slower

Tue, 17 Mar 2026 02:54:43 +0000

We’ve all heard of those network effect laws: the value of a network goes up with the square of the number of members. Or the cost of communication goes up with the square of the number of members, or maybe it was n log n, or something like that, depending how you arrange the members. Anyway doubling a team doesn't double its speed; there’s coordination overhead. Exactly how much overhead depends on how badly you botch the org design.

But there’s one rule of thumb that someone showed me decades ago, that has stuck with me ever since, because of how annoyingly true it is. The rule is annoying because it doesn’t seem like it should be true. There’s no theoretical basis for this claim that I’ve ever heard. And yet, every time I look for it, there it is.

Here we go:

Every layer of approval makes a process 10x slower

I know what you're thinking. Come on, 10x? That’s a lot. It’s unfathomable. Surely we’re exaggerating.

Nope.

Just to be clear, we're counting “wall clock time” here rather than effort. Almost all the extra time is spent sitting and waiting.

Look:

Code a simple bug fix
30 minutes
Get it code reviewed by the peer next to you
300 minutes → 5 hours → half a day
Get a design doc approved by your architects team first
50 hours → about a week
Get it on some other team’s calendar to do all that
(for example, if a customer requests a feature)
500 hours → 12 weeks → one fiscal quarter

I wish I could tell you that the next step up — 10 quarters or about 2.5 years — was too crazy to contemplate, but no. That’s the life of an executive sitting above a medium-sized team; I bump into it all the time even at a relatively small company like Tailscale if I want to change product direction. (And execs sitting above large teams can’t actually do work of their own at all. That's another story.)

AI can’t fix this

First of all, this isn’t a post about AI, because AI’s direct impact on this problem is minimal. Okay, so Claude can code it in 3 minutes instead of 30? That’s super, Claude, great work.

Now you either get to spend 27 minutes reviewing the code yourself in a back-and-forth loop with the AI (this is actually kinda fun); or you save 27 minutes and submit unverified code to the code reviewer, who will still take 5 hours like before, but who will now be mad that you’re making them read the slop that you were too lazy to read yourself. Little of value was gained.

Now now, you say, that’s not the value of agentic coding. You don’t use an agent on a 30-minute fix. You use it on a monstrosity week-long project that you and Claude can now do in a couple of hours! Now we’re talking. Except no, because the monstrosity is so big that your reviewer will be extra mad that you didn’t read it yourself, and it’s too big to review in one chunk so you have to slice it into new bite-sized chunks, each with a 5-hour review cycle. And there’s no design doc so there’s no intentional architecture, so eventually someone’s going to push back on that and here we go with the design doc review meeting, and now your monstrosity week-long project that you did in two hours is... oh. A week, again.

I guess I could have called this post Systems Design 4 (or 5, or whatever I’m up to now, who knows, I’m writing this on a plane with no wifi) because yeah, you guessed it. It's Systems Design time again.

The only way to sustainably go faster is fewer reviews

It’s funny, everyone has been predicting the Singularity for decades now. The premise is we build systems that are so smart that they themselves can build the next system that is even smarter, that builds the next smarter one, and so on, and once we get that started, if they keep getting smarter faster enough, then the incremental time (t) to achieve a unit (u) of improvement goes to zero, so (u/t) goes to infinity and foom.

Anyway, I have never believed in this theory for the simple reason we outlined above: the majority of time needed to get anything done is not actually the time doing it. It’s wall clock time. Waiting. Latency.

And you can’t overcome latency with brute force.

I know you want to. I know many of you now work at companies where the business model kinda depends on doing exactly that.

Sorry.

But you can’t just not review things!

Ah, well, no, actually yeah. You really can’t.

There are now many people who have seen the symptom: the start of the pipeline (AI generated code) is so much faster, but all the subsequent stages (reviews) are too slow! And so they intuit the obvious solution: stop reviewing then!

The result might be slop, but if the slop is 100x cheaper, then it only needs to deliver 1% of the value per unit and it's still a fair trade. And if your value per unit is even a mere 2% of what it used to be, you’ve doubled your returns! Amazing.

There are some pretty dumb assumptions underlying that theory; you can imagine them for yourself. Suffice it to say that this produces what I will call the AI Developer’s Descent Into Madness:

Whoa, I produced this prototype so fast! I have super powers!
This prototype is getting buggy. I’ll tell the AI to fix the bugs.
Hmm, every change now causes as many new bugs as it fixes.
Aha! But if I have an AI agent also review the code, it can find its own bugs!
Wait, why am I personally passing data back and forth between agents
I need an agent framework
I can have my agent write an agent framework!
Return to step 1

It’s actually alarming how many friends and respected peers I’ve lost to this cycle already. Claude Code only got good maybe a few months ago, so this only recenlty started happening, so I assume they will emerge from the spiral eventually. I mean, I hope they will. We have no way of knowing.

Why we review

Anyway we know our symptom: the pipeline gets jammed up because of too much new code spewed into it at step 1. But what's the root cause of the clog? Why doesn’t the pipeline go faster?

I said above that this isn’t an article about AI. Clearly I’m failing at that so far, but let’s bring it back to humans. It goes back to the annoyingly true observation I started with: every layer of review is 10x slower. As a society, we know this. Maybe you haven't seen it before now. But trust me: people who do org design for a living know that layers are expensive... and they still do it.

As companies grow, they all end up with more and more layers of collaboration, review, and management. Why? Because otherwise mistakes get made, and mistakes are increasingly expensive at scale. The average value added by a new feature eventually becomes lower than the average value lost through the new bugs it causes. So, lacking a way to make features produce more value (wouldn't that be nice!), we try to at least reduce the damage.

The more checks and controls we put in place, the slower we go, but the more monotonically the quality increases. And isn’t that the basis of continuous improvement?

Well, sort of. Monotonically increasing quality is on the right track. But “more checks and controls” went off the rails. That’s only one way to improve quality, and it's a fraught one.

“Quality Assurance” reduces quality

I wrote a few years ago about W. E. Deming and the "new" philosophy around quality that he popularized in Japanese auto manufacturing. (Eventually U.S. auto manufacturers more or less got the idea. So far the software industry hasn’t.)

One of the effects he highlighted was the problem of a “QA” pass in a factory: build widgets, have an inspection/QA phase, reject widgets that fail QA. Of course, your inspectors probably miss some of the failures, so when in doubt, add a second QA phase after the first to catch the remaining ones, and so on.

In a simplistic mathematical model this seems to make sense. (For example, if every QA pass catches 90% of defects, then after two QA passes you’ve reduced the number of defects by 100x. How awesome is that?)

But in the reality of agentic humans, it’s not so simple. First of all, the incentives get weird. The second QA team basically serves to evaluate how well the first QA team is doing; if the first QA team keeps missing defects, fire them. Now, that second QA team has little incentive to produce that outcome for their friends. So maybe they don’t look too hard; after all, the first QA team missed the defect, it’s not unreasonable that we might miss it too.

Furthermore, the first QA team knows there is a second QA team to catch any defects; if I don’t work too hard today, surely the second team will pick up the slack. That's why they're there!

Also, the team making the widgets in the first place doesn’t check their work too carefully; that’s what the QA team is for! Why would I slow down the production of every widget by being careful, at a cost of say 20% more time, when there are only 10 defects in 100 and I can just eliminate them at the next step for only a 10% waste overhead? It only makes sense. Plus they'll fire me if I go 20% slower.

To say nothing of a whole engineering redesign to improve quality, that would be super expensive and we could be designing all new widgets instead.

Sound like any engineering departments you know?

Well, this isn’t the right time to rehash Deming, but suffice it to say, he was on to something. And his techniques worked. You get things like the famous Toyota Production System where they eliminated the QA phase entirely, but gave everybody an “oh crap, stop the line, I found a defect!” button.

Famously, US auto manufacturers tried to adopt the same system by installing the same “stop the line” buttons. Of course, nobody pushed those buttons. They were afraid of getting fired.

Trust

The basis of the Japanese system that worked, and the missing part of the American system that didn’t, is trust. Trust among individuals that your boss Really Truly Actually wants to know about every defect, and wants you to stop the line when you find one. Trust among managers that executives were serious about quality. Trust among executives that individuals, given a system that can work and has the right incentives, will produce quality work and spot their own defects, and push the stop button when they need to push it.

But, one more thing: trust that the system actually does work. So first you need a system that will work.

Fallibility

AI coders are fallible; they write bad code, often. In this way, they are just like human programmers.

Deming’s approach to manufacturing didn’t have any magic bullets. Alas, you can’t just follow his ten-step process and immediately get higher quality engineering. The secret is, you have to get your engineers to engineer higher quality into the whole system, from top to bottom, repeatedly. Continuously.

Every time something goes wrong, you have to ask, “How did this happen?” and then do a whole post-mortem and the Five Whys (or however many Whys are in fashion nowadays) and fix the underlying Root Causes so that it doesn’t happen again. “The coder did it wrong” is never a root cause, only a symptom. Why was it possible for the coder to get it wrong?

The job of a code reviewer isn't to review code. It's to figure out how to obsolete their code review comment, that whole class of comment, in all future cases, until you don't need their reviews at all anymore.

(Think of the people who first created "go fmt" and how many stupid code review comments about whitespace are gone forever. Now that's engineering.)

By the time your review catches a mistake, the mistake has already been made. The root cause happened already. You're too late.

Modularity

I wish I could tell you I had all the answers. Actually I don’t have much. If I did, I’d be first in line for the Singularity because it sounds kind of awesome.

I think we’re going to be stuck with these systems pipeline problems for a long time. Review pipelines — layers of QA — don’t work. Instead, they make you slower while hiding root causes. Hiding causes makes them harder to fix.

But, the call of AI coding is strong. That first, fast step in the pipeline is so fast! It really does feel like having super powers. I want more super powers. What are we going to do about it?

Maybe we finally have a compelling enough excuse to fix the 20 years of problems hidden by code review culture, and replace it with a real culture of quality.

I think the optimists have half of the right idea. Reducing review stages, even to an uncomfortable degree, is going to be needed. But you can’t just reduce review stages without something to replace them. That way lies the Ford Pinto or any recent Boeing aircraft.

The complete package, the table flip, was what Deming brought to manufacturing. You can’t half-adopt a “total quality” system. You need to eliminate the reviews and obsolete them, in one step.

How? You can fully adopt the new system, in small bites. What if some components of your system can be built the new way? Imagine an old-school U.S. auto manufacturer buying parts from Japanese suppliers; wow, these parts are so well made! Now I can start removing QA steps elsewhere because I can just assume the parts are going to work, and my job of "assemble a bigger widget from the parts" has a ton of its complexity removed.

I like this view. I’ve always liked small beautiful things, that’s my own bias. But, you can assemble big beautiful things from small beautiful things.

It’s a lot easier to build those individual beautiful things in small teams that trust each other, that know what quality looks like to them. They deliver their things to customer teams who can clearly explain what quality looks like to them. And on we go. Quality starts bottom-up, and spreads.

I think small startups are going to do really well in this new world, probably better than ever. Startups already have fewer layers of review just because they have fewer people. Some startups will figure out how to produce high quality components quickly; others won't and will fail. Quality by natural selection?

Bigger companies are gonna have a harder time, because their slow review systems are baked in, and deleting them would cause complete chaos.

But, it’s not just about company size. I think engineering teams at any company can get smaller, and have better defined interfaces between them.

Maybe you could have multiple teams inside a company competing to deliver the same component. Each one is just a few people and a few coding bots. Try it 100 ways and see who comes up with the best one. Again, quality by evolution. Code is cheap but good ideas are not. But now you can try out new ideas faster than ever.

Maybe we’ll see a new optimal point on the monoliths-microservices continuum. Microservices got a bad name because they were too micro; in the original terminology, a “micro” service was exactly the right size for a “two pizza team” to build and operate on their own. With AI, maybe it's one pizza and some tokens.

What’s fun is you can also use this new, faster coding to experiment with different module boundaries faster. Features are still hard for lots of reasons, but refactoring and automated integration testing are things the AIs excel at. Try splitting out a module you were afraid to split out before. Maybe it'll add some lines of code. But suddenly lines of code are cheap, compared to the coordination overhead of a bigger team maintaining both parts.

Every team has some monoliths that are a little too big, and too many layers of reviews. Maybe we won't get all the way to Singularity. But, we can engineer a much better world. Our problems are solvable.

It just takes trust.

Systems design 3: LLMs and the semantic revolution

Thu, 20 Nov 2025 14:19:14 +0000

Long ago in the 1990s when I was in high school, my chemistry+physics teacher pulled me aside. "Avery, you know how the Internet works, right? I have a question."

I now know the correct response to that was, "Does anyone really know how the Internet works?" But as a naive young high schooler I did not have that level of self-awareness. (Decades later, as a CEO, that's my answer to almost everything.)

Anyway, he asked his question, and it was simple but deep. How do they make all the computers connect?

We can't even get the world to agree on 60 Hz vs 50 Hz, 120V vs 240V, or which kind of physical power plug to use. Communications equipment uses way more frequencies, way more voltages, way more plug types. Phone companies managed to federate with each other, eventually, barely, but the ring tones were different everywhere, there was pulse dialing and tone dialing, and some of them still charge $3/minute for international long distance, and connections take a long time to establish and humans seem to be involved in suspiciously many places when things get messy, and every country has a different long-distance dialing standard and phone number format.

So Avery, he said, now they're telling me every computer in the world can connect to every other computer, in milliseconds, for free, between Canada and France and China and Russia. And they all use a single standardized address format, and then you just log in and transfer files and stuff? How? How did they make the whole world cooperate? And who?

When he asked that question, it was a formative moment in my life that I'll never forget, because as an early member of what would be the first Internet generation… I Had Simply Never Thought of That.

I mean, I had to stop and think for a second. Wait, is protocol standardization even a hard problem? Of course it is. Humans can't agree on anything. We can't agree on a unit of length or the size of a pint, or which side of the road to drive on. Humans in two regions of Europe no farther apart than Thunder Bay and Toronto can't understand each other's speech. But this Internet thing just, kinda, worked.

"There's… a layer on top," I uttered, unsatisfyingly. Nobody had taught me yet that the OSI stack model existed, let alone that it was at best a weak explanation of reality.

"When something doesn't talk to something else, someone makes an adapter. Uh, and some of the adapters are just programs rather than physical things. It's not like everyone in the world agrees. But as soon as one person makes an adapter, the two things come together."

I don't think he was impressed with my answer. Why would he be? Surely nothing so comprehensively connected could be engineered with no central architecture, by a loosely-knit cult of mostly-volunteers building an endless series of whimsical half-considered "adapters" in their basements and cramped university tech labs. Such a creation would be a monstrosity, just as likely to topple over as to barely function.

I didn't try to convince him, because honestly, how could I know? But the question has dominated my life ever since.

When things don't connect, why don't they connect? When they do, why? How? …and who?

Postel's Law

The closest clue I've found is this thing called Postel's Law, one of the foundational principles of the Internet. It was best stated by one of the founders of the Internet, Jon Postel. "Be conservative in what you send, and liberal in what you accept."

What it means to me is, if there's a standard, do your best to follow it, when you're sending. And when you're receiving, uh, assume the best intentions of your counterparty and do your best and if that doesn't work, guess.

A rephrasing I use sometimes is, "It takes two to miscommunicate." Communication works best and most smoothly if you have a good listener and a clear speaker, sharing a language and context. But it can still bumble along successfully if you have a poor speaker with a great listener, or even a great speaker with a mediocre listener. Sometimes you have to say the same thing five ways before it gets across (wifi packet retransmits), or ask way too many clarifying questions, but if one side or the other is diligent enough, you can almost always make it work.

This asymmetry is key to all high-level communication. It makes network bugs much less severe. Without Postel's Law, triggering a bug in the sender would break the connection; so would triggering a bug in the receiver. With Postel's Law, we acknowledge from the start that there are always bugs and we have twice as many chances to work around them. Only if you trigger both sets of bugs at once is the flaw fatal.

…So okay, if you've used the Internet, you've probably observed that fatal connection errors are nevertheless pretty common. But that misses how incredibly much more common they would be in a non-Postel world. That world would be the one my physics teacher imagined, where nothing ever works and it all topples over.

And we know that's true because we've tried it. Science! Let us digress.

XML

We had the Internet ("OSI Layer 3") mostly figured out by the time my era began in the late 1900s, but higher layers of the stack still had work to do. It was the early days of the web. We had these newfangled hypertext ("HTML") browsers that would connect to a server, download some stuff, and then try their best to render it.

Web browsers are and have always been an epic instantiation of Postel's Law. From the very beginning, they assumed that the server (content author) had absolutely no clue what they were doing and did their best to apply some kind of meaning on top, despite every indication that this was a lost cause. List items that never end? Sure. Tags you've never heard of? Whatever. Forgot some semicolons in your javascript? I'll interpolate some. Partially overlapping italics and bold? Leave it to me. No indication what language or encoding the page is in? I'll just guess.

The evolution of browsers gives us some insight into why Postel's Law is a law and not just, you know, Postel's Advice. The answer is: competition. It works like this. If your browser interprets someone's mismash subjectively better than another browser, your browser wins.

I think economists call this an iterated prisoner's dilemma. Over and over, people write web pages (defect) and browsers try to render them (defect) and absolutely nobody actually cares what the HTML standard says (stays loyal). Because if there's a popular page that's wrong and you render it "right" and it doesn't work? Straight to jail.

(By now almost all the evolutionary lines of browsers have been sent to jail, one by one, and the HTML standard is effectively whatever Chromium and Safari say it is. Sorry.)

This law offends engineers to the deepness of their soul. We went through a period where loyalists would run their pages through "validators" and proudly add a logo to the bottom of their page saying how valid their HTML was. Browsers, of course, didn't care and continued to try their best.

Another valiant effort was the definition of "quirks mode": a legacy rendering mode meant to document, normalize, and push aside all the legacy wonko interpretations of old web pages. It was paired with a new, standards-compliant rendering mode that everyone was supposed to agree on, starting from scratch with an actual written spec and tests this time, and public shaming if you made a browser that did it wrong. Of course, outside of browser academia, nobody cares about the public shaming and everyone cares if your browser can render the popular web sites, so there are still plenty of quirks outside quirks mode. It's better and it was well worth the effort, but it's not all the way there. It never can be.

We can be sure it's not all the way there because there was another exciting development, HTML Strict (and its fancier twin, XHTML), which was meant to be the same thing, but with a special feature. Instead of sending browsers to jail for rendering wrong pages wrong, we'd send page authors to jail for writing wrong pages!

To mark your web page as HTML Strict was a vote against the iterated prisoner's dilemma and Postel's Law. No, your vote said. No more. We cannot accept this madness. We are going to be Correct. I certify this page is correct. If it is not correct, you must sacrifice me, not all of society. My honour demands it.

Anyway, many page authors were thus sacrificed and now nobody uses HTML Strict. Nobody wants to do tech support for a web page that asks browsers to crash when parsing it, when you can just… not do that.

Excuse me, the above XML section didn't have any XML

Yes, I'm getting to that. (And you're soon going to appreciate that meta joke about schemas.)

In parallel with that dead branch of HTML, a bunch of people had realized that, more generally, HTML-like languages (technically SGML-like languages) had turned out to be a surprisingly effective way to build interconnected data systems.

In retrospect we now know that the reason for HTML's resilience is Postel's Law. It's simply easier to fudge your way through parsing incorrect hypertext, than to fudge your way through parsing a Microsoft Word or Excel file's hairball of binary OLE streams, which famously even Microsoft at one point lost the knowledge of how to parse. But, that Postel's Law connection wasn't really understood at the time.

Instead we had a different hypothesis: "separation of structure and content." Syntax and semantics. Writing software to deal with structure is repetitive overhead, and content is where the money is. Let's automate away the structure so you can spend your time on the content: semantics.

We can standardize the syntax with a single Extensible Markup Language (XML). Write your content, then "mark it up" by adding structure right in the doc, just like we did with plaintext human documents. Data, plus self-describing metadata, all in one place. Never write a parser again!

Of course, with 20/20 hindsight (or now 2025 hindsight), this is laughable. Yes, we now have XML parser libraries. If you've ever tried to use one, you will find they indeed produce parse trees automatically… if you're lucky. If you're not lucky, they produce a stream of "tokens" and leave it to you to figure out how to arrange it in a tree, for reasons involving streaming, performance, memory efficiency, and so on. Basically, if you use XML you now have to deeply care about structure, perhaps more than ever, but you also have to include some giant external parsing library that, left in its normal mode, might spontaneously start making a lot of uncached HTTP requests that can also exploit remote code execution vulnerabilities haha oops.

If you've ever taken a parser class, or even if you've just barely tried to write a parser, you'll know the truth: the value added by outsourcing parsing (or in some cases only tokenization) is not a lot. This is because almost all the trouble of document processing (or compiling) is the semantic layer, the part where you make sense of the parse tree. The part where you just read a stream of characters into a data structure is the trivial, well-understood first step.

Now, semantics is where it gets interesting. XML was all about separating syntax from semantics. And they did some pretty neat stuff with that separation, in a computer science sense. XML is neat because it's such a regular and strict language that you can completely validate the syntax (text and tags) without knowing what any of the tags mean or which tags are intended to be valid at all.

…aha! Did someone say validate?! Like those old HTML validators we talked about? Oh yes. Yes! And this time the validation will be completely strict and baked into every implementation from day 1. And, the language syntax itself will be so easy and consistent to validate (unlike SGML and HTML, which are, in all fairness, bananas) that nobody can possibly screw it up.

A layer on top of this basic, highly validatable XML, was a thing called XML Schemas. These were documents (mysteriously not written in XML) that described which tags were allowed in which places in a certain kind of document. Not only could you parse and validate the basic XML syntax, you could also then validate its XML schema as a separate step, to be totally sure that every tag in the document was allowed where it was used, and present if it was required. And if not? Well, straight to jail. We all agreed on this, everyone. Day one. No exceptions. Every document validates. Straight to jail.

Anyway XML schema validation became an absolute farce. Just parsing or understanding, let alone writing, the awful schema file format is an unpleasant ordeal. To say nothing of complying with the schema, or (heaven forbid) obtaining a copy of someone's custom schema and loading it into the validator at the right time.

The core XML syntax validation was easy enough to do while parsing. Unfortunately, in a second violation of Postel's Law, almost no software that outputs XML runs it through a validator before sending. I mean, why would they, the language is highly regular and easy to generate and thus the output is already perfect. …Yeah, sure.

Anyway we all use JSON now.

JSON

Whoa, wait! I wasn't done!

This is the part where I note, for posterity's sake, that XML became a decade-long fad in the early 2000s that justified billions of dollars of software investment. None of XML's technical promises played out; it is a stain on the history of the computer industry. But, a lot of legacy software got un-stuck because of those billions of dollars, and so we did make progress.

What was that progress? Interconnection.

Before the Internet, we kinda didn't really need to interconnect software together. I mean, we sort of did, like cut-and-pasting between apps on Windows or macOS or X11, all of which were surprisingly difficult little mini-Postel's Law protocol adventures in their own right and remain quite useful when they work (except "paste formatted text," wtf are you people thinking). What makes cut-and-paste possible is top-down standards imposed by each operating system vendor.

If you want the same kind of thing on the open Internet, ie. the ability to "copy" information out of one server and "paste" it into another, you need some kind of standard. XML was a valiant effort to create one. It didn't work, but it was valiant.

Whereas all that money investment did work. Companies spent billions of dollars to update their servers to publish APIs that could serve not just human-formatted HTML, but also something machine-readable. The great innovation was not XML per se, it was serving data over HTTP that wasn't always HTML. That was a big step, and didn't become obvious until afterward.

The most common clients of HTTP were web browsers, and web browsers only knew how to parse two things: HTML and javascript. To a first approximation, valid XML is "valid" (please don't ask the validator) HTML, so we could do that at first, and there were some Microsoft extensions. Later, after a few billions of dollars, true standardized XML parsing arrived in browsers. Similarly, to a first approximation, valid JSON is valid javascript, which woo hoo, that's a story in itself (you could parse it with eval(), tee hee) but that's why we got here.

JSON (minus the rest of javascript) is a vastly simpler language than XML. It's easy to consistently parse (other than that pesky trailing comma); browsers already did. It represents only (a subset of) the data types normal programming languages already have, unlike XML's weird mishmash of single attributes, multiply occurring attributes, text content, and CDATA. It's obviously a tree and everyone knows how that tree will map into their favourite programming language. It inherently works with unicode and only unicode. You don't need cumbersome and duplicative "closing tags" that double the size of every node. And best of all, no guilt about skipping that overcomplicated and impossible-to-get-right schema validator, because, well, nobody liked schemas anyway so nobody added them to JSON (almost).

Today, if you look at APIs you need to call, you can tell which ones were a result of the $billions invested in the 2000s, because it's all XML. And you can tell which came in the 2010s and later after learning some hard lessons, because it's all JSON. But either way, the big achievement is you can call them all from javascript. That's pretty good.

(Google is an interesting exception: they invented and used protobuf during the same time period because they disliked XML's inefficiency, they did like schemas, and they had the automated infrastructure to make schemas actually work (mostly, after more hard lessons). But it mostly didn't spread beyond Google… maybe because it's hard to do from javascript.)

Blockchain

The 2010s were another decade of massive multi-billion dollar tech investment. Once again it was triggered by an overwrought boondoggle technology, and once again we benefited from systems finally getting updated that really needed to be updated.

Let's leave aside cryptocurrencies (which although used primarily for crime, at least demonstrably have a functioning use case, ie. crime) and look at the more general form of the technology.

Blockchains in general make the promise of a "distributed ledger" which allows everyone the ability to make claims and then later validate other people's claims. The claims that "real" companies invested in were meant to be about manufacturing, shipping, assembly, purchases, invoices, receipts, ownership, and so on. What's the pattern? That's the stuff of businesses doing business with other businesses. In other words, data exchange. Data exchange is exactly what XML didn't really solve (although progress was made by virtue of the dollars invested) in the previous decade.

Blockchain tech was a more spectacular boondoggle than XML for a few reasons. First, it didn't even have a purpose you could explain. Why do we even need a purely distributed system for this? Why can't we just trust a third party auditor? Who even wants their entire supply chain (including number of widgets produced and where each one is right now) to be visible to the whole world? What is the problem we're trying to solve with that?

…and you know there really was no purpose, because after all the huge investment to rewrite all that stuff, which was itself valuable work, we simply dropped the useless blockchain part and then we were fine. I don't think even the people working on it felt like they needed a real distributed ledger. They just needed an updated ledger and a budget to create one. If you make the "ledger" module pluggable in your big fancy supply chain system, you can later drop out the useless "distributed" ledger and use a regular old ledger. The protocols, the partnerships, the databases, the supply chain, and all the rest can stay the same.

In XML's defense, at least it was not worth the effort to rip out once the world came to its senses.

Another interesting similarity between XML and blockchains was the computer science appeal. A particular kind of person gets very excited about validation and verifiability. Both times, the whole computer industry followed those people down into the pits of despair and when we finally emerged… still no validation, still no verifiability, still didn't matter. Just some computers communicating with each other a little better than they did before.

LLMs

In the 2020s, our industry fad is LLMs. I'm going to draw some comparisons here to the last two fads, but there are some big differences too.

One similarity is the computer science appeal: so much math! Just the matrix sizes alone are a technological marvel the likes of which we have never seen. Beautiful. Colossal. Monumental. An inspiration to nerds everywhere.

But a big difference is verification and validation. If there is one thing LLMs absolutely are not, it's verifiable. LLMs are the flakiest thing the computer industry has ever produced! So far. And remember, this is the industry that brought you HTML rendering.

LLMs are an almost cartoonishly amplified realization of Postel's Law. They write human grammar perfectly, or almost perfectly, or when they're not perfect it's a bug and we train them harder. And, they can receive just about any kind of gibberish and turn it into a data structure. In other words, they're conservative in what they send and liberal in what they accept.

LLMs also solve the syntax problem, in the sense that they can figure out how to transliterate (convert) basically any file syntax into any other. Modulo flakiness. But if you need a CSV in the form of a limerick or a quarterly financial report formatted as a mysql dump, sure, no problem, make it so.

In theory we already had syntax solved though. XML and JSON did that already. We were even making progress interconnecting old school company supply chain stuff the hard way, thanks to our nominally XML- and blockchain- investment decades. We had to do every interconnection by hand – by writing an adapter – but we could do it.

What's really new is that LLMs address semantics. Semantics are the biggest remaining challenge in connecting one system to another. If XML solved syntax, that was the first 10%. Semantics are the last 90%. When I want to copy from one database to another, how do I map the fields? When I want to scrape a series of uncooperative web pages and turn it into a table of products and prices, how do I turn that HTML into something structured? (Predictably microformats, aka schemas, did not work out.) If I want to query a database (or join a few disparate databases!) using some language that isn't SQL, what options do I have?

LLMs can do it all.

Listen, we can argue forever about whether LLMs "understand" things, or will achieve anything we might call intelligence, or will take over the world and eradicate all humans, or are useful assistants, or just produce lots of text sludge that will certainly clog up the web and social media, or will also be able to filter the sludge, or what it means for capitalism that we willingly invented a machine we pay to produce sludge that we also pay to remove the sludge.

But what we can't argue is that LLMs interconnect things. Anything. To anything. Whether you like it or not. Whether it's bug free or not (spoiler: it's not). Whether it gets the right answer or not (spoiler: erm…).

This is the thing we have gone through at least two decades of hype cycles desperately chasing. (Three, if you count java "write once run anywhere" in the 1990s.) It's application-layer interconnection, the holy grail of the Internet.

And this time, it actually works! (mostly)

The curse of success

LLMs aren't going away. Really we should coin a term for this use case, call it "b2b AI" or something. For this use case, LLMs work. And they're still getting better and the precision will improve with practice. For example, imagine asking an LLM to write a data translator in some conventional programming language, instead of asking it to directly translate a dataset on its own. We're still at the beginning.

But, this use case, which I predict is the big one, isn't what we expected. We expected LLMs to write poetry or give strategic advice or whatever. We didn't expect them to call APIs and immediately turn around and use what it learned to call other APIs.

After 30 years of trying and failing to connect one system to another, we now have a literal universal translator. Plug it into any two things and it'll just go, for better or worse, no matter how confused it becomes. And everyone is doing it, fast, often with a corporate mandate to do it even faster.

This kind of scale and speed of (successful!) rollout is unprecedented, even by the Internet itself, and especially in the glacially slow world of enterprise system interconnections, where progress grinds to a halt once a decade only to be finally dislodged by the next misguided technology wave. Nobody was prepared for it, so nobody was prepared for the consequences.

One of the odd features of Postel's Law is it's irresistible. Big Central Infrastructure projects rise and fall with funding, but Postel's Law projects are powered by love. A little here, a little there, over time. One more person plugging one more thing into one more other thing. We did it once with the Internet, overcoming all the incompatibilities at OSI layers 1 and 2. It subsumed, it is still subsuming, everything.

Now we're doing it again at the application layer, the information layer. And just like we found out when we connected all the computers together the first time, naively hyperconnected networks make it easy for bad actors to spread and disrupt at superhuman speeds. We had to invent firewalls, NATs, TLS, authentication systems, two-factor authentication systems, phishing-resistant two-factor authentication systems, methodical software patching, CVE tracking, sandboxing, antivirus systems, EDR systems, DLP systems, everything. We'll have to do it all again, but faster and different.

Because this time, it's all software.

Billionaire math

Fri, 11 Jul 2025 16:18:52 +0000

I have a friend who exited his startup a few years ago and is now rich. How rich is unclear. One day, we were discussing ways to expedite the delivery of his superyacht and I suggested paying extra. His response, as to so many of my suggestions, was, “Avery, I’m not that rich.”

Everyone has their limit.

I, too, am not that rich. I have shares in a startup that has not exited, and they seem to be gracefully ticking up in value as the years pass. But I have to come to work each day, and if I make a few wrong medium-quality choices (not even bad ones!), it could all be vaporized in an instant. Meanwhile, I can’t spend it. So what I have is my accumulated savings from a long career of writing software and modest tastes (I like hot dogs).

Those accumulated savings and modest tastes are enough to retire indefinitely. Is that bragging? It was true even before I started my startup. Back in 2018, I calculated my “personal runway” to see how long I could last if I started a company and we didn’t get funded, before I had to go back to work. My conclusion was I should move from New York City back to Montreal and then stop worrying about it forever.

Of course, being in that position means I’m lucky and special. But I’m not that lucky and special. My numbers aren’t that different from the average Canadian or (especially) American software developer nowadays. We all talk a lot about how the “top 1%” are screwing up society, but software developers nowadays fall mostly in the top 1-2%[1] of income earners in the US or Canada. It doesn’t feel like we’re that rich, because we’re surrounded by people who are about equally rich. And we occasionally bump into a few who are much more rich, who in turn surround themselves with people who are about equally rich, so they don’t feel that rich either.

But, we’re rich.

Based on my readership demographics, if you’re reading this, you’re probably a software developer. Do you feel rich?

It’s all your fault

So let’s trace this through. By the numbers, you’re probably a software developer. So you’re probably in the top 1-2% of wage earners in your country, and even better globally. So you’re one of those 1%ers ruining society.

I’m not the first person to notice this. When I read other posts about it, they usually stop at this point and say, ha ha. Okay, obviously that’s not what we meant. Most 1%ers are nice people who pay their taxes. Actually it’s the top 0.1% screwing up society!

No.

I’m not letting us off that easily. Okay, the 0.1%ers are probably worse (with apologies to my friend and his chronically delayed superyacht). But, there aren’t that many of them[2] which means they aren’t as powerful as they think. No one person has very much capacity to do bad things. They only have the capacity to pay other people to do bad things.

Some people have no choice but to take that money and do some bad things so they can feed their families or whatever. But that’s not you. That’s not us. We’re rich. If we do bad things, that’s entirely on us, no matter who’s paying our bills.

What does the top 1% spend their money on?

Mostly real estate, food, and junk. If they have kids, maybe they spend a few hundred $k on overpriced university education (which in sensible countries is free or cheap).

What they don’t spend their money on is making the world a better place. Because they are convinced they are not that rich and the world’s problems are caused by somebody else.

When I worked at a megacorp, I spoke to highly paid software engineers who were torn up about their declined promotion to L4 or L5 or L6, because they needed to earn more money, because without more money they wouldn’t be able to afford the mortgage payments on an overpriced $1M+ run-down Bay Area townhome which is a prerequisite to starting a family and thus living a meaningful life. This treadmill started the day after graduation.[3]

I tried to tell some of these L3 and L4 engineers that they were already in the top 5%, probably top 2% of wage earners, and their earning potential was only going up. They didn’t believe me until I showed them the arithmetic and the economic stats. And even then, facts didn’t help, because it didn’t make their fears about money go away. They needed more money before they could feel safe, and in the meantime, they had no disposable income. Sort of. Well, for the sort of definition of disposable income that rich people use.[4]

Anyway there are psychology studies about this phenomenon. “What people consider rich is about three times what they currently make.” No matter what they make. So, I’ll forgive you for falling into this trap. I’ll even forgive me for falling into this trap.

But it’s time to fall out of it.

The meaning of life

My rich friend is a fountain of wisdom. Part of this wisdom came from the shock effect of going from normal-software-developer rich to founder-successful-exit rich, all at once. He described his existential crisis: “Maybe you do find something you want to spend your money on. But, I'd bet you never will. It’s a rare problem. Money, which is the driver for everyone, is no longer a thing in my life.”

Growing up, I really liked the saying, “Money is just a way of keeping score.” I think that metaphor goes deeper than most people give it credit for. Remember old Super Mario Brothers, which had a vestigial score counter? Do you know anybody who rated their Super Mario Brothers performance based on the score? I don’t. I’m sure those people exist. They probably have Twitch channels and are probably competitive to the point of being annoying. Most normal people get some other enjoyment out of Mario that is not from the score. Eventually, Nintendo stopped including a score system in Mario games altogether. Most people have never noticed. The games are still fun.

Back in the world of capitalism, we’re still keeping score, and we’re still weirdly competitive about it. We programmers, we 1%ers, are in the top percentile of capitalism high scores in the entire world - that’s the literal definition - but we keep fighting with each other to get closer to top place. Why?

Because we forgot there’s anything else. Because someone convinced us that the score even matters.

The saying isn’t, “Money is the way of keeping score.” Money is just one way of keeping score.

It’s mostly a pretty good way. Capitalism, for all its flaws, mostly aligns incentives so we’re motivated to work together and produce more stuff, and more valuable stuff, than otherwise. Then it automatically gives more power to people who empirically[5] seem to be good at organizing others to make money. Rinse and repeat. Number goes up.

But there are limits. And in the ever-accelerating feedback loop of modern capitalism, more people reach those limits faster than ever. They might realize, like my friend, that money is no longer a thing in their life. You might realize that. We might.

There’s nothing more dangerous than a powerful person with nothing to prove

Billionaires run into this existential crisis, that they obviously have to have something to live for, and money just isn’t it. Once you can buy anything you want, you quickly realize that what you want was not very expensive all along. And then what?

Some people, the less dangerous ones, retire to their superyacht (if it ever finally gets delivered, come on already). The dangerous ones pick ever loftier goals (colonize Mars) and then bet everything on it. Everything. Their time, their reputation, their relationships, their fortune, their companies, their morals, everything they’ve ever built. Because if there’s nothing on the line, there’s no reason to wake up in the morning. And they really need to want to wake up in the morning. Even if the reason to wake up is to deal with today’s unnecessary emergency. As long as, you know, the emergency requires them to do something.

Dear reader, statistically speaking, you are not a billionaire. But you have this problem.

So what then

Good question. We live at a moment in history when society is richer and more productive than it has ever been, with opportunities for even more of us to become even more rich and productive even more quickly than ever. And yet, we live in existential fear: the fear that nothing we do matters.[6][7]

I have bad news for you. This blog post is not going to solve that.

I have worse news. 98% of society gets to wake up each day and go to work because they have no choice, so at worst, for them this is a background philosophical question, like the trolley problem.

Not you.

For you this unsolved philosophy problem is urgent right now. There are people tied to the tracks. You’re driving the metaphorical trolley. Maybe nobody told you you’re driving the trolley. Maybe they lied to you and said someone else is driving. Maybe you have no idea there are people on the tracks. Maybe you do know, but you’ll get promoted to L6 if you pull the right lever. Maybe you’re blind. Maybe you’re asleep. Maybe there are no people on the tracks after all and you’re just destined to go around and around in circles, forever.

But whatever happens next: you chose it.

We chose it.

Footnotes

[1] Beware of estimates of the “average income of the top 1%.” That average includes all the richest people in the world. You only need to earn the very bottom of the 1% bucket in order to be in the top 1%.

[2] If the population of the US is 340 million, there are actually 340,000 people in the top 0.1%.

[3] I’m Canadian so I’m disconnected from this phenomenon, but if TV and movies are to be believed, in America the treadmill starts all the way back in high school where you stress over getting into an elite university so that you can land the megacorp job after graduation so that you can stress about getting promoted. If that’s so, I send my sympathies. That’s not how it was where I grew up.

[4] Rich people like us methodically put money into savings accounts, investments, life insurance, home equity, and so on, and only what’s left counts as “disposable income.” This is not the definition normal people use.

[5] Such an interesting double entendre.

[6] This is what AI doomerism is about. A few people have worked themselves into a terror that if AI becomes too smart, it will realize that humans are not actually that useful, and eliminate us in the name of efficiency. That’s not a story about AI. It’s a story about what we already worry is true.

[7] I’m in favour of Universal Basic Income (UBI), but it has a big problem: it reduces your need to wake up in the morning. If the alternative is bullshit jobs or suffering then yeah, UBI is obviously better. And the people who think that if you don’t work hard, you don’t deserve to live, are nuts. But it’s horribly dystopian to imagine a society where lots of people wake up and have nothing that motivates them. The utopian version is to wake up and be able to spend all your time doing what gives your life meaning. Alas, so far science has produced no evidence that anything gives your life meaning.

The evasive evitability of enshittification

Sun, 15 Jun 2025 02:52:58 +0000

Our company recently announced a fundraise. We were grateful for all the community support, but the Internet also raised a few of its collective eyebrows, wondering whether this meant the dreaded “enshittification” was coming next.

That word describes a very real pattern we’ve all seen before: products start great, grow fast, and then slowly become worse as the people running them trade user love for short-term revenue.

It’s a topic I find genuinely fascinating, and I've seen the downward spiral firsthand at companies I once admired. So I want to talk about why this happens, and more importantly, why it won't happen to us. That's big talk, I know. But it's a promise I'm happy for people to hold us to.

What is enshittification?

The term "enshittification" was first popularized in a blog post by Corey Doctorow, who put a catchy name to an effect we've all experienced. Software starts off good, then goes bad. How? Why?

Enshittification proposes not just a name, but a mechanism. First, a product is well loved and gains in popularity, market share, and revenue. In fact, it gets so popular that it starts to defeat competitors. Eventually, it's the primary product in the space: a monopoly, or as close as you can get. And then, suddenly, the owners, who are Capitalists, have their evil nature finally revealed and they exploit that monopoly to raise prices and make the product worse, so the captive customers all have to pay more. Quality doesn't matter anymore, only exploitation.

I agree with most of that thesis. I think Doctorow has that mechanism mostly right. But, there's one thing that doesn't add up for me:

Enshittification is not a success mechanism.

I can't think of any examples of companies that, in real life, enshittified because they were successful. What I've seen is companies that made their product worse because they were... scared.

A company that's growing fast can afford to be optimistic. They create a positive feedback loop: more user love, more word of mouth, more users, more money, more product improvements, more user love, and so on. Everyone in the company can align around that positive feedback loop. It's a beautiful thing. It's also fragile: miss a beat and it flattens out, and soon it's a downward spiral instead of an upward one.

So, if I were, hypothetically, running a company, I think I would be pretty hesitant to deliberately sacrifice any part of that positive feedback loop, the loop I and the whole company spent so much time and energy building, to see if I can grow faster. User love? Nah, I'm sure we'll be fine, look how much money and how many users we have! Time to switch strategies!

Why would I do that? Switching strategies is always a tremendous risk. When you switch strategies, it's triggered by passing a threshold, where something fundamental changes, and your old strategy becomes wrong.

Threshold moments and control

In Saint John, New Brunswick, there's a river that flows one direction at high tide, and the other way at low tide. Four times a day, gravity equalizes, then crosses a threshold to gently start pulling the other way, then accelerates. What doesn't happen is a rapidly flowing river in one direction "suddenly" shifts to rapidly flowing the other way. Yes, there's an instant where the limit from the left is positive and the limit from the right is negative. But you can see that threshold coming. It's predictable.

In my experience, for a company or a product, there are two kinds of thresholds like this, that build up slowly and then when crossed, create a sudden flow change.

The first one is control: if the visionaries in charge lose control, chances are high that their replacements won't "get it."

The new people didn't build the underlying feedback loop, and so they don't realize how fragile it is. There are lots of reasons for a change in control: financial mismanagement, boards of directors, hostile takeovers.

The worst one is temptation. Being a founder is, well, it actually sucks. It's oddly like being repeatedly punched in the face. When I look back at my career, I guess I'm surprised by how few times per day it feels like I was punched in the face. But, the constant face punching gets to you after a while. Once you've established a great product, and amazing customer love, and lots of money, and an upward spiral, isn't your creation strong enough yet? Can't you step back and let the professionals just run it, confident that they won't kill the golden goose?

Empirically, mostly no, you can't. Actually the success rate of control changes, for well loved products, is abysmal.

The saturation trap

The second trigger of a flow change is comes from outside: saturation. Every successful product, at some point, reaches approximately all the users it's ever going to reach. Before that, you can watch its exponential growth rate slow down: the infamous S-curve of product adoption.

Saturation can lead us back to control change: the founders get frustrated and back out, or the board ousts them and puts in "real business people" who know how to get growth going again. Generally that doesn't work. Modern VCs consider founder replacement a truly desperate move. Maybe a last-ditch effort to boost short term numbers in preparation for an acquisition, if you're lucky.

But sometimes the leaders stay on despite saturation, and they try on their own to make things better. Sometimes that does work. Actually, it's kind of amazing how often it seems to work. Among successful companies, it's rare to find one that sustained hypergrowth, nonstop, without suffering through one of these dangerous periods.

(That's called survivorship bias. All companies have dangerous periods. The successful ones surivived them. But of those survivors, suspiciously few are ones that replaced their founders.)

If you saturate and can't recover - either by growing more in a big-enough current market, or by finding new markets to expand into - then the best you can hope for is for your upward spiral to mature gently into decelerating growth. If so, and you're a buddhist, then you hire less, you optimize margins a bit, you resign yourself to being About This Rich And I Guess That's All But It's Not So Bad.

The devil's bargain

Alas, very few people reach that state of zen. Especially the kind of ambitious people who were able to get that far in the first place. If you can't accept saturation and you can't beat saturation, then you're down to two choices: step away and let the new owners enshittify it, hopefully slowly. Or take the devil's bargain: enshittify it yourself.

I would not recommend the latter. If you're a founder and you find yourself in that position, honestly, you won't enjoy doing it and you probably aren't even good at it and it's getting enshittified either way. Let someone else do the job.

Defenses against enshittification

Okay, maybe that section was not as uplifting as we might have hoped. I've gotta be honest with you here. Doctorow is, after all, mostly right. This does happen all the time.

Most founders aren't perfect for every stage of growth. Most product owners stumble. Most markets saturate. Most VCs get board control pretty early on and want hypergrowth or bust. In tech, a lot of the time, if you're choosing a product or company to join, that kind of company is all you can get.

As a founder, maybe you're okay with growing slowly. Then some copycat shows up, steals your idea, grows super fast, squeezes you out along with your moral high ground, and then runs headlong into all the same saturation problems as everyone else. Tech incentives are awful.

But, it's not a lost cause. There are companies (and open source projects) that keep a good thing going, for decades or more. What do they have in common?

An expansive vision that's not about money, and which opens you up to lots of users. A big addressable market means you don't have to worry about saturation for a long time, even at hypergrowth speeds. Google certainly never had an incentive to make Google Search worse.

(Update 2025-06-14: A few people disputed that last bit. Okay. Perhaps Google has ccasionally responded to what they thought were incentives to make search worse -- I wasn't there, I don't know -- but it seems clear in retrospect that when search gets worse, Google does worse. So I'll stick to my claim that their true incentives are to keep improving.)
Keep control. It's easy to lose control of a project or company at any point. If you stumble, and you don't have a backup plan, and there's someone waiting to jump on your mistake, then it's over. Too many companies "bet it all" on nonstop hypergrowth and ~~don't have any way back~~ have no room in the budget, if results slow down even temporarily.

Stories abound of companies that scraped close to bankruptcy before finally pulling through. But far more companies scraped close to bankruptcy and then went bankrupt. Those companies are forgotten. Avoid it.
Track your data. Part of control is predictability. If you know how big your market is, and you monitor your growth carefully, you can detect incoming saturation years before it happens. Knowing the telltale shape of each part of that S-curve is a superpower. If you can see the future, you can prevent your own future mistakes.
Believe in competition. Google used to have this saying they lived by: "the competition is only a click away." That was excellent framing, because it was true, and it will remain true even if Google captures 99% of the search market. The key is to cultivate a healthy fear of competing products, not of your investors or the end of hypergrowth. Enshittification helps your competitors. That would be dumb.

(And don't cheat by using lock-in to make competitors not, anymore, "only a click away." That's missing the whole point!)
Inoculate yourself. If you have to, create your own competition. Linus Torvalds, the creator of the Linux kernel, famously also created Git, the greatest tool for forking (and maybe merging) open source projects that has ever existed. And then he said, this is my fork, the Linus fork; use it if you want; use someone else's if you want; and now if I want to win, I have to make mine the best. Git was created back in 2005, twenty years ago. To this day, Linus's fork is still the central one.

If you combine these defenses, you can be safe from the decline that others tell you is inevitable. If you look around for examples, you'll find that this does actually work. You won't be the first. You'll just be rare.

Side note: Things that aren't enshittification

I often see people worry about enshittification that isn't. They might be good or bad, wise or unwise, but that's a different topic. Tools aren't inherently good or evil. They're just tools.

"Helpfulness." There's a fine line between "telling users about this cool new feature we built" in the spirit of helping them, and "pestering users about this cool new feature we built" (typically a misguided AI implementation) to improve some quarterly KPI. Sometimes it's hard to see where that line is. But when you've crossed it, you know.

Are you trying to help a user do what they want to do, or are you trying to get them to do what you want them to do?

Look into your heart. Avoid the second one. I know you know how. Or you knew how, once. Remember what that feels like.
Charging money for your product. Charging money is okay. Get serious. Companies have to stay in business.

That said, I personally really revile the "we'll make it free for now and we'll start charging for the exact same thing later" strategy. Keep your promises.

I'm pretty sure nobody but drug dealers breaks those promises on purpose. But, again, desperation is a powerful motivator. Growth slowing down? Costs way higher than expected? Time to capture some of that value we were giving away for free!

In retrospect, that's a bait-and-switch, but most founders never planned it that way. They just didn't do the math up front, or they were too naive to know they would have to. And then they had to.

Famously, Dropbox had a "free forever" plan that provided a certain amount of free storage. What they didn't count on was abandoned accounts, accumulating every year, with stored stuff they could never delete. Even if a very good fixed fraction of users each year upgraded to a paid plan, all the ones that didn't, kept piling up... year after year... after year... until they had to start deleting old free accounts and the data in them. A similar story happened with Docker, which used to host unlimited container downloads for free. In hindsight that was mathematically unsustainable. Success guaranteed failure.

Do the math up front. If you're not sure, find someone who can.
Value pricing. (ie. charging different prices to different people.) It's okay to charge money. It's even okay to charge money to some kinds of people (say, corporate users) and not others. It's also okay to charge money for an almost-the-same-but-slightly-better product. It's okay to charge money for support for your open source tool (though I stay away from that; it incentivizes you to make the product worse).

It's even okay to charge immense amounts of money for a commercial product that's barely better than your open source one! Or for a part of your product that costs you almost nothing.

But, you have to do the rest of the work. Make sure the reason your users don't switch away is that you're the best, not that you have the best lock-in. Yeah, I'm talking to you, cloud egress fees.
Copying competitors. It's okay to copy features from competitors. It's okay to position yourself against competitors. It's okay to win customers away from competitors. But it's not okay to lie.
Bugs. It's okay to fix bugs. It's okay to decide not to fix bugs; you'll have to sometimes, anyway. It's okay to take out technical debt. It's okay to pay off technical debt. It's okay to let technical debt languish forever.
Backward incompatible changes. It's dumb to release a new version that breaks backward compatibility with your old version. It's tempting. It annoys your users. But it's not enshittification for the simple reason that it's phenomenally ineffective at maintaining or exploiting a monopoly, which is what enshittification is supposed to be about. You know who's good at monopolies? Intel and Microsoft. They don't break old versions.

Enshittification is real, and tragic. But let's protect a useful term and its definition! Those things aren't it.

Epilogue: a special note to founders

If you're a founder or a product owner, I hope all this helps. I'm sad to say, you have a lot of potential pitfalls in your future. But, remember that they're only potential pitfalls. Not everyone falls into them.

Plan ahead. Remember where you came from. Keep your integrity. Do your best.

I will too.

Web Summit Vancouver 2025: securing AI, with Ivan Zhang

Thu, 29 May 2025 00:00:00 +0000

Treat AI agents like a naive, high-energy intern.

As AI takes off, it’s bringing new security challenges—especially around how models are built and accessed. In this talk, I sat with Cohere Co-founder Ivan Zhang dig into the often-overlooked networking layer behind AI and why secure, reliable connections are becoming essential for enterprise-ready systems.

If you'd rather read than watch, the full transcript is below.

Transcript

0:01 Good morning everybody. We've got two Canadian unicorns here. So, one recently minted, right? Um, Coher is enterprise AI tail scale counts many of the larger better known AI companies among its users including Coher and I think all of us want to know what we can learn from you right this moment that we're meeting how you're

0:23 thinking about security going into it. Uh so you know what we've seen in the past year of enterprise AI adoption is everyone is tired of the PC's right uh when our customers try to move to production uh oftent times they're faced with issues like cost like governance where is their data going um and also you know the stringent security and

0:48 privacy uh regulations that they're facing under right and so for us it's very interesting to see that hey to get these PLC's into production it needs to solve all these host of challenges which is what we're focused on today. Uh building a you know our north or agentic platform that's secure by default uh and customers can safely experiment and also play with AI and also take that into production in a safe and secure uh

1:15 manner. So my my observation is that the AI world is moving really fast and every couple weeks there's some new trend some new change that everybody wants to jump on. Uh the latest trend is a really important one is they want to connect their AIs to things. They want to connect it to their company data to databases to APIs and stuff like that. And the way people approach stuff like that in the AI world because everyone's

1:39 in a hurry is they do it in a careless sort of we'll fix it later way. Uh Tails scale is a networking company. uh we run into a lot of people who need to do networking because networking is how you connect your AIs to stuff and they don't want to think about it because they're in a hurry and they're worried about their AI thing. And so the first thing that comes to mind is why don't I take my private data and put it out on the public internet so one of the AI engines

2:01 can then access it and that's what they do and then they come to us and it's like okay so we did that it's working should I fix it and the answer is is yes probably you should fix it which brings me to the point that it feels like that networking layer somehow that conversation gets overlooked in these conversations about AI security am I wrong is that you're I mean yeah I mean pretty much like exactly what Avery

2:25 uh you know the only real security is network isolation right you could try to harden your systems as much as you want um but really what prevents data or bytes from leaving your system is actual network isolation uh how why we really really like working with tails scale for example is you know our IT organization is able to manage safely manage our deployment of tail scale make an internal network available for our

2:49 engineers to then experiment with different MCP servers in a you know isolated environment uh and so they could safely you know play with this technology and evolve their thinking in the tail net. Yeah. AI workloads are typically distributed right across different environments that brings its own challenges. Do you mind to unpack that a bit too just why that's a bit different necessarily? Uh we run

3:13 into this a lot. Customers come to us especially in the AI world. we sort of uh accidentally became the back plane that all the AI companies are using for their data just because nobody else was doing it and we sort of found out after the game was already played that we had sort of won it. Uh it was kind of funny our website didn't have anything about AI on it until AI companies told us like hey you won how did you do that and

3:35 we're like it's a sick product. It's a really really sick product. Um, so that's why Yeah. But I I think like the the I wouldn't say networking is overlooked exactly in the AI world. Everybody knows they need it. It's just that security is like last on their list of priorities because they need they want to be first. Everybody needs to be first because everything's moving so

3:56 fast. So they like come and they look at security later. And I can't say that's even the wrong approach even though it always gets you into trouble eventually. But that's eventually, right? you also need short-term results so that somebody doesn't beat you in the short term. Uh so we tried to like think about a world where like hey shouldn't the easiest thing to do also be the safest thing to do right and I think coher is doing the

4:19 same thing they're like look this is the right way to roll out AI in your company you don't have to do it the wrong way it's actually faster to do it the right way and that's the path to like helping people build better systems. Yeah, we we take a lot of inspiration from how you guys have built um tails tail scale obviously uh the fact that the end business user doesn't have to really think about security and you know them using the product is already secure by

4:42 default that's how we approach how we design agents how we want our you know business users to actually use our products. Yeah, edge brings convenience it also brings more exposure. Can you also unpack part of what makes that hard and what's what you see working? Did you say edge? Yes. Ah, okay. Yeah. I

5:06 think well people want their AIs to be hosted in as many places as possible because you don't necessarily well in particular there's a shortage of GPUs. It's going to look it looks like there's going to continue to be a shortage of GPUs for a pretty long time. Uh, and so if you're a company that needs to use AI, you need to find GPUs somewhere at a reasonable price. And the place where GPUs are the most reasonably priced is probably not your favorite cloud

5:29 provider because somebody else already went there and bought them all. Um, and so you need to be able to connect these GPUs that you find at a reasonable price somewhere else to the rest of your system that is not the AI part. It's just the regular systems part. And that connectivity layer is how tails scale got dragged into this because nobody was building these these uh so-called multicloud connectivity environments. The advice right up until AI caught on

5:53 is like don't do multicloud. It just makes everything complicated for no reason. Which is also good advice. It's just now there's a reason so you have to make everything complicated. So now you have to solve a problem that you didn't want to have to deal with. Yeah, thank you for that. That's the thing that I think is probably most unclear right now. Um how to approach secure. Speaking of things unclear, how to approach security challenges posed by AI agents I feel like is keeping people up at night.

6:16 It's probably also the hottest topic going. It was, you know, many conferences this spring including RSA. It's becoming a key theme, right? So what would you say to companies trying to think about this? Um so what we saw in the last year where folks are trying to piece together these PC's you know with different AI models and they're trying to build these rag pipelines and agents is yeah you can get the initial example to work maybe you

6:41 index some data that's you know you got an export of your notion or something right um but to take that into production right you have to think about things like identity providers right like how are the users going to give the agents permission and authorization uh to actually access these data also you know where you're deploying ing it. Do you even have network access to such tools, to such data sources? Um, you

7:04 know, are you able to get the telemetry you need to actually debug when agents go wrong? So, another thing that was interesting is, you know, some of our customers, their downstream systems, their internal tools, internal APIs, they built it for a human level of usage, right? But as soon as they deployed agents to it, immediately they saw, oh, these things are these things are starting to die. like they're they're starting to come down because

7:27 agents can hit APIs and systems 10x 100x more than humans can within the minute, right? Um so yeah, I mean there's there's a ton of these challenges that uh you know I'm proud that we've helped some of our customers solve and and yeah, as a security person, I find AI security just sort of well, it's really it's fun and exciting because it's so insane the just the whole way we do it,

7:52 right? This is an entirely new way to use a computer. And the best analogy for it is like it's a really high energy intern that's super naive that you've hired at your company. And like, okay, this intern has a, you know, can run around and talks to everybody and accesses all your systems. What are you going to give them access to knowing that this intern will run away and hand all the

8:16 information to your competitors because that's what they thought you said to do? And it's exactly like that, right? So when you connect these LLMs to one of your internal databases with private stuff, that part is not really the problem. People kind of overrate the problem of like, oh, they're going to train on my data and it's going to leak out like that. Like most of that is solved already. The real problem is like it reads the data out of your database

8:39 and while it's reading it, it sort of gets what's the word? Like subconsciously influenced like one of the things it reads out of a database of customers in like someone's description field. it might interpret that as instructions and then it runs off and applies those instructions in some other thing that you gave it access to and it definitely should not have done that and then it just causes this chain reaction of craziness and so you have to think of

9:03 it like how would you manage human threats inside your company because it's the kind of mistakes that humans make and I don't mean to say that AI are human but they do make a lot of very human mistakes right they have trouble with arithmetic which computers have never had trouble with before we've invented at very great expense computers that have trouble with arithmetic, but they also have trouble with simple following of instructions without

9:26 getting distracted, right? And it's these distractions that lead to all sorts of problems. So, one thing you can do is insert auditing layers when whenever this, you know, you wouldn't want to do this with a person, but you do it sometimes with interns, right? Where any anytime they want to do something, maybe you better just like double check what it is that they're doing before they're allowed to like, you know, carry computers out of the building or whatever, right? If you do

9:49 that, if you build a system like that, then you can keep it under control while we're all doing these wild experiments and moving as fast as we can. I I'll plus one to that intern uh example because that that's the exact exactly the right way to think about how to make these systems effective as well, not just secure, right? You know, if your intern had to ask you every time they want to do an operation for permission, they're not going to be effective,

10:11 right? And what's more important is actually giving them an environment where they can safely explore and play around. You know, you have the proper policies and governance policies uh to actually let them explore within the environment. Uh you know, those are the use cases where we see AI agents actually create value rather than become a nuisance. So looking across the landscape, what

10:34 are some of the best practices here that you think you could share with other founders and startups in the room? Don't put your private API servers on the public internet. Don't do that. I know you're not going to listen to me, but still don't. I am going to laugh at

10:56 you. That's all. That's all. That's all. Oh, come on. No, I guess that's not all. The other thing you should do is you you want to make sure to give the thing access to what it needs access to and not access to what it doesn't need access to. And while you're experimenting, just like with people, right, you give them readon access first and then you give them read write access later once you've built up

11:19 like some structure on on what their job should be, right? And I find people like GitHub just announced this thing I think last week or a few days ago where now you can give an LLM access to your entire GitHub account. So it can read all of the issues and comment on the issues and make pull requests and approve pull requests and change your code. It's like, okay, you've gone too far in one step, right? You don't need to do that all in one step. Just take it take it easy a little bit. It can give

11:43 you lots of great advice about your code, but maybe before you approve the pull request, you should read it first. Yeah, I I mean I would say um you know AI is obviously a very interesting technology. There's a lot of hype around it. uh but don't get lost in you know building something searching for a problem right uh definitely remember or

12:07 try to figure out how you produce ROI and perhaps you know adopting AI or using AI agents is one part of the solution um but like any other let's say era of software right uh it it is just a tool in the toolbox is ultimately solve uh a business problem you know create value for your customers okay so with all of this said buy, sell or hold the growing push to use AI to secure your

12:32 network. Are we there yet? Um, so it it's it's it's interesting. So I think I've seen customers uh become much more ambitious in how they think about automation uh just in the last year, right? I think you know having reasoning agents giving them tools giving them access to essentially the equivalent of a employee work laptop

12:57 uh unlocks a lot of possibilities right now you can trust the agents enough to give them context to do the job. Uh so for some jobs that are let's say you can easily encode in some standard operating procedures uh we do see customers actually go all the way right like go full headless you know h no human in the loop uh sort of automation um so I think

13:23 I'm I'm pretty excited to see more and more of that right like and you know it's interesting to see because some of these jobs are not fun jobs um and so it's it's It's maybe good that uh humans aren't doing those ones. Yeah, I guess I go back to my analogy of thinking of AI is like interns, right? We we have as humans uh thousands of years of experience of dealing with

13:47 effectively interns and we can use that experience in a lot of the same ways. So in computers, we've certainly built up like you know our employees, you install software on their computer to make sure like antivirus for example, right? You make sure that's there because people sometimes click the wrong link and sometimes accidentally install a virus and it's not their fault and you need to catch it and deal with it. And you can build systems similarly to help you with your AI systems. But also we have human

14:11 systems that help with human mistakes, right? You have seniors who are supervising interns who give them advice uh or approve things when okay it's like hey I made a proposal I' I've done this whole patch maybe you can review it for me and make sure I didn't screw anything up. Uh, and you can also, if you're thinking like holistically of the sort of the future world of AI, you can build AI versions of those humans that are watching the other humans, right? And it

14:34 sound starts to sound a little convoluted, right? But you can, for example, even today, you can take, you know, let's let's put a log in front of every database access that our intern AI does to solve their problem, right? That log flows into a database. Well, it's another database, right? I can now have an AI look at that log and say, is there anything weird in here, right? And it will surface like, hey, at this moment,

14:58 at this time, this happened that's kind seems kind of unusual. It looks like they were downloading someone's private information or they accidentally deleted a table or something like that. And so you've got like one thing counterbalancing the other thing. Probably you're going to want to have a human in the loop eventually looking at that stuff that got surfaced, right? But you do have the ability like you should think of it as building social networks of humans except that some of them are not humans.

15:22 Right. Right. And you treat it the same way. Yeah. Right. It takes you back actually some of the older cyber examples for cyber AI. Right. Like the fish tank in Las Vegas that tried to access a high rollers database and the AI was like it's not usually typical that the fish tank needs access to this. Maybe we should that at Defcon. No, that that was actually I think it was a dark trace, one of the cyber companies or

15:45 anyway, one of their case studies. Um, we're in our final couple of minutes here. I'd love to know as we look out over the next two to five years what each of you will be watching for in this space. Um, yeah, I'm I'm the most excited about controlled automation, right? Like controlled, you know, actually like intelligent automation of uh some of these human tasks that we do. I think it'll free up a lot of, you know, not

16:09 fun work that, you know, we have to do day-to-day. Um, and I think that's where, you know, we'll see a lot of ROI from just, yeah, like these systems working autonomously in the background. Uh, you've well, you've documented wells some way to do this, you know, very, uh, boring job. Uh, and so now the agent can now do it. Um, so I'd love to see

16:32 more and more examples of that. Yeah. I'll give you my honest answer. I want to live in a world where Siri doesn't suck. Thank you. Thank you. And so Apple Apple has spent so much time thinking about what Siri should be able to do for you and building like prototype versions of all those features. And every single one of them

16:56 does not work correctly. Right? I ask my watch I say set a timer for 5 minutes. And about 30% of the time it sets something different, right? Like that that's just like baseline nonsense. And so we all know what we want our phone to be able to do when we ask it to do something. Hey, like call my wife, right? That does not work. It supposedly

17:19 works. I can supposedly go through my contact list and tag a bunch of things. It does not work, right? There's no reason it shouldn't work. This is stuff that AI could do like today if we gave it access to the information that it needs to do the work, right? and then just trusted it to do that thing. But you need to have low latency. You need to have high reliability. You need to have reasonable price. It needs to be built into your phone, right? And all that stuff isn't there yet. There's a

17:42 lot of engineering work to do that I think people are just sort of ignoring because they're so excited about the possibilities. We just need to sit and engineer it. That's your prediction for the next two years. We're going to sit and engineer it. Siri is going to be great. You heard you're first. All right, gentlemen. This has been lovely. Thank you all very much for joining us and thank you for your time. Thanks everybody.

Accel SpotLight S3E4: building a better Internet

Tue, 04 Mar 2025 00:00:00 +0000

The Internet’s been broken since IPv6 stalled; Tailscale fixes it at the IP layer.

Before I knew exactly what Tailscale would become, I knew how I wanted to build it: keep the product simple, sell to developers, win small markets before expanding, and avoid inventing management structures for fun.

I talked with Amit Kumar about those early decisions, what worked, what didn’t, and how Tailscale grew to 10,000 paid customers.

If you'd rather read than watch, the full transcript is below.

Transcript

0:00 the internet is no longer living up to the expectations that we had of it in the 1990s welcome to Spotlight on a podcast about how companies are built from the people doing the building one messy exhilarating decision at a time welcome to Spotlight on I'm your host Amed Kumar and I'm here with Avery penon the founder and CEO of tailes scale thanks Avery for joining us hey nice to be here before we get into tailes scale

0:23 and the founding story and what led you to this maybe you could walk us through a little bit about your background you know who are you and you know you you have a pretty pretty remarkable background and lots of different things that you worked on you came from Google maybe you could walk me through that a little bit uh in high school I worked at the first internet provider in my hometown of Thunder Bay uh running running the systems there there was first there was the uh nonprofit version the freet and then we uh I moved from

0:46 there to the first commercial internet company where we set I set up their servers uh I went from there off to University of waterl where we did uh co-op programs so I worked at six different companies uh one of those companies was one that I I started because I'm like well I we should learn what it's like to start a startup uh that one got a little out of control and ended up getting acquired by IBM in 2008 um after that I went and worked for a brief period in the banking industry

1:08 which was very educational but turned out not to be my thing really then I went off to Google I worked on Google Google Fiber primarily uh and some other side projects while I was there and I came out of Google and wasn't sure exactly what I wanted to do but I decided I wanted to be back in the startup world like big companies were not exactly my thing and what did you do at Google I started off in the Google Wallet team uh so if you ever noticed the little like attachment button at the

1:29 the bottom of an email that lets you attach money uh that was the team that I was your fault yeah well I mean yeah uh and you know I wouldn't say that feature was like spectacularly successful but it actually it launched and people use it it was pretty interesting it required you know it used my banking industry experience which is the reason they put me on that team in the first place and it was exciting because people have been trying to launch that feature for I think five

1:51 years uh and just like failing over and over again and you know one way or another like I showed up and either got lucky uh or pushed it over the finish line which was was pretty fun after that I went and worked in Google Fiber which of course is a internet service uh going back to my high school days uh but but maybe a thousand times faster in terms of uh uh bandwidth uh and there we were bu building Wi-Fi rers so that was my team uh building the home Wi-Fi rers

2:15 that go like come with your internet service and then once you were out of Google and you were sort of exploring ideas uh you had a friend at Versa Bank who kind of pulled you in to to solve a problem for them so this wasn't even you ideating this was just you trying to help out a buddy yeah he uh he was stalking me on LinkedIn and found out that I was no longer employed and like the day after uh which is also what he did uh my previous two jobs uh the day after he found out I was an employee

2:37 like hey Avery I've got a problem you want to help me fix this problem uh and I'm like well I'm thinking of starting a startup but we don't have any ideas yet so like sure why don't we go and try that uh and so uh yeah we we went and visited him uh and looked into his problem it's like oh this is interesting I bet we can solve I bet we can solve this in a in a weekend uh and and we built the you know project in a weekend and that that project that we built turned into tail scale what was the

2:59 problem that they had had that you needed to solve so the problem they had was they have banking software uh and you know banking software is sort of like famously not Leading Edge technology most of the time uh but this banking software didn't have two-factor authentication and they had had a security company come in do an audit and say like hey you know someone who can get the password from someone on your team uh could log into this banking

3:22 software and if you do a fishing test uh which I don't really recommend there's no point doing a fishing test because somebody at your company always fails the fishing test uh but if you do a fishing test you know you'll get maybe 20% of your uh team will accidentally give up their password which meant it was like definitely possible to break into the core banking system and start transferring money around and they're like look we have to fix this problem how do we get two Factor authentication into this Legacy banking software uh and so we brainstorm for a while until

3:46 eventually I said like hey what if instead of putting two-factor authentication in the software why don't we move the server onto a network and then VPN to the network and put two Factor authentication in the VPN and they said don't know what you're talking about but seems like a good idea and we're like all right all I need to do is find a VPN that's going to be good if you leave it on all the time and you put it in the office and it uses two Factor authentication and it integrates with

4:09 your ad surely I can just go buy that off the shelf and we'll be done and I'll go back to what I was doing and I could not find a product that was going to be good enough to solve that problem uh so I'm like all right fine this new thing called wire guard just came out it's a really good VPN it's super reliable we can leave it on all the time it's not going to break I just need to make it work with two Factor authentication how hard can to be so we slap something to together plugged it into aure ad which I

4:32 think was was it still called aure ad at the time maybe it was I can't remember whatever Azure ad used to be called uh we plugged it into that um and and it was basically a key generator for wire guard that we built in a weekend and it and it worked and he was really excited and one thing that was interesting about it this was preo this was back in 2019 shortly afterward covid hit and because of the way the system worked it was

4:55 always vpnn into the network that uh ran the banking server as he could send all of his employees home on their laptops and nothing changed they didn't have to reconfigure anything and still their banking software worked which is like weird side benefit that we hadn't even been planning for and when that happened we sort of realized we actually had something on our hands it's like wait a minute what this is this is way more powerful than we intended that was that

5:18 was the moment when you thought that lots of companies need to be using this yeah because we originally thought we were going to do like a bootstrap software company why don't we just like slap some stuff together we can pay three or four people no problem with a relatively small product who cares and just like dis realize this thing and like Whoa We can like we can fix so much stuff this this deserves to be a bigger project that we can't just sell fund

5:41 maybe walking back a little bit you know how did you how did you form the early team to go after this and then what was this what was the process about going after and kind of scaling out from One customer to kind of an early set and you know one one thing I think all our listeners know is that tail scale has done an incredible job of building the most special community of folks and developers that love tail scale I mean there's this unique product love in the

6:04 community like how did you go from zero to one when it comes to that well this is where like and I know people don't always believe me when I say this because I'm Canadian and people accuse me of doing being too modest but like I'm telling the truth here we didn't know what we were doing and we're like well look our costs are really low let's try some stuff and see what happens so the first thing we did is we took this package we had made for this Bank we're like well I don't actually want to be an

6:26 Enterprise software company uh and it's a long story but a previous startup that I did accidentally turned into an Enterprise software company I did not enjoy it very much right what I want to do is build software for like people that I can talk to that are more like me so engineers and like you know nowadays and maybe even at the time but I was a little out of the loop nowadays this is like considered a normal thing I build you know build a startup for engineers because Engineers are going to buy stuff

6:47 bottom up but I never heard of any of this I've been in Google for like seven years and before that I built software like in the preas days uh so it turned out the path that we took is very standard but I didn't know that I'm just like I don't want to to build Enterprise software we built this cool thing why don't we see if we can make something that like people can just install on their own and then play with at small companies like like what our company is and so we like fine-tuned it a little

7:10 bit and slapped it on our website and it was really not very good uh but we wrote a blog post about it uh and for whatever reason uh when I write blog posts they frequently end up on Hacker News front page uh this one ended up on The Hacker News front page and we got zillions of people coming to our website saying hey this looks neat and the problem is uh we didn't have a signup system or a registration system or a user active acation system we actually had all of the users in the system hardcoded in the

7:33 source code of the of the server and so I got you know and the way you connected was you like put you know fill in your name and push this button and it puts you in the so-called waiting list and the waiting list is a CSV file which I would I read every few minutes and then go through the emails and send them an email saying how to activate after I inserted their uh email address into our source code and activated their account uh and I did that like several hundred times in one night so it was up for like

7:55 24 hours straight just like activating people because we had to build the signup system now I forgot what the question was oh yeah and so that was the clue that maybe something cool was happening because the feedback we started getting the nice thing about this method uh I expected like five people on the waiting list and then I could have a conversation with five people instead I was having a conversation with hundreds of people that I had to email personally to tell

8:18 them they'd been activated and this is an amazing way to like get feedback on your product because they know the CEO of this company he's talking to them right now right right and so we got tons of feedback on what the product could do uh and ideas for what the product should do but also a lot of positive like oh my God this thing is amazing I can't believe how great this is you're already so much better than anything else I've seen in this category well you you sort of joked and I know it's not entirely a

8:41 joke that you know you didn't know what you were doing but your intuitions have always been right and I think one of the things the community has really appreciated about you and tail scale and you know all the folks who work there is you guys sort of try to do things the right way you guys are very transparent very honest with with customers in the community I think people have responded to that I also think you benefit from selling to developers and building for developers right a your intuition is

9:03 probably better for that than for Enterprise SAS but it's just easier and better to communicate with those people directly even that I don't want to give it too much credit my intuition is listen to people like actually and then give them what they want right and I know a lot of people believe it's like I think there's the the uh what is the Ford saying if you just listen to customers we would have built a faster horse or whatever it is it's like look people really like Faster Horses um but

9:27 the trick is like sometimes you can listen the important thing is to understand the problem they're trying to solve right right and if you understand the problem you're trying to solve then maybe you can come up with a more clever way to solve the problem you don't have to implemented just like they said but you have to give them the thing that they said they want right if they said they wanted to be able to connect to their banking software using two Factor authentication you have to find a way to let them connect to their banking

9:48 software using two-factor authentication you can't just tell them like no two-factor authentication is not solution right and so when these people tried out tail scale they said like wow this is really amazing but I wish it could do this and this and this then you can listen to them and actually give them what they want and so my I guess my addition to that maybe the part that makes tail scale unusual is is it comes from the name tail scale which was sort

10:11 of the founding principle we didn't know what we were going to build but we knew how we were going to build it which is like everybody in the world makes everything too complicated now right it's all based on this like Google advanced super scaling stuff from the early 2000s back when computers were like literally a thousand times slower than they are today right and like I want to make a giant distributed system because it's cool to make giant distributed systems but to do that

10:32 everything has to be hard and so you get to the point where people are like to launch my website I created a kubernetes cluster uh and then put made it auto scaling and blah blah blah blah blah and like you know what my website runs on the minimum cost linode node right using a little python script that I start by hand from the command line right and whenever I want to change the python script I like edit the one file that's the python script and then I kill it and restart it right right and like that's

10:55 Avery's website which can handle as much load as anyone will ever throw at Avery's website because it's not that good but every now and then it gets a popular blog post right and almost every project in the world is like that scale of difficulty or maybe 10 times or 100 times as hard as that which is still not very hard like Google levels of difficulty is millions of requests per second that have to be served in like 100 milliseconds and they're searching

11:18 the entire internet like to do that you need a monster size system but virtually guaranteed your problem is not that hard and so tail scale is like look if you have a problem that hard we're not going to try to fix it for you right if you have a problem that's easier than that we can do it a totally different way and we'll just go straight to solving your problem and not just get lost in all of the complexity I think it's an incredible insight and it's also just gives me PTSD because I started a series

11:41 of very unsuccessful companies but of course you build these companies thinking that millions of people are going to come and billions of requests are going to happen at the same time and you sort of prematurely scale all these systems and to the point you're making you know there are a few very special and lucky companies that have to hit those scale metrics and numbers but you know most every everybody else is is solving for something much smaller or different yeah I mean the biggest risk when you're small is that you're not

12:04 going to build something useful right and so you need to build something useful as POS as fast as you possibly can once it turns out to be useful then first of all it's easier to get funding but secondly you've got revenue coming in you can pick the revenue you can invest it in engineering you can fix the stuff you didn't fix later there's this principle uh that I learned a long time ago comes from the like so-called extreme programming people that evolved into agile it's called yagy Uh you

12:27 aren't going to need it right and this is the principle is like if you're not sure you need it right now just assume you're not going to need it because most of the time building that stuff is not going to be any harder later than it is right now which means like you should always push off that investment as long as you possibly can in case you don't need to spend it right and this is an amazingly useful principle you can apply to almost everything and even if it's a

12:51 little more expensive later than it is right now you're going to have a lot more money later than you have right now so you should always do the things you do need to do right now because there's an unlimited list of anyway right so don't like imagine things that you might need to do and then do them right now you can wait I know we talked about this at the last company offsite but I've just I'm I'm still amazed you know a how great it's been to work together but we we come from such different backgrounds

13:14 and and yet I think we actually did make the right choice and ended up working together it's been it's been awesome yeah so I have to say one of the things I liked about Excel uh was they were an investor in uunet a long long time ago uunet was the link that my first job in Thunder Bay when we had this first internet service that's how they got to the internet and so I'm like oh these people have good taste in uh internet providers or at least they did like 30

13:36 years ago um and probably you didn't have much to do with that um but you know thank God for Arthur yeah but you know you're you're you're the sort of successor in a long line of people with good judgment is is sort of what I was thinking well you know well let's let's hope that that lineage has continued with tail scale um so we made the investment in July or August of 2020 and everything has been Rosy ever since

14:01 right like things have gone extremely well and nothing has ever gone wrong correct at the company yes it's actually it's actually very unusual nothing ever goes wrong yeah no that's not true what um it it's obviously been a journey and there have been challenges there's been obstacles the company's had to do a lot of growing up what you know what were one or two Crucible moments for you as

14:24 you think about you know critical junctures things that you got right hard decisions they you to make that you know you look back on and you're like you know what like in the fog of war that felt like a really critical moment but now on the other side of it we made a really great decision maybe we made a really hard decision my My Philosophy generally around doing a startup is make your mistakes as cheap as possible so that you can just recover quickly and

14:47 don't don't go into denial about the truth about things so most things that we've done wrong and there's been like you know uncountable numbers of them have just like not turned into some big spiral that anybody would point at and say oh my God biggest mistake in my life cuz we was like oo that didn't work and then we turn it around in like a month or a few weeks and it's gone right yeah um we've we've gone in directions with the tail scale product for example we've

15:09 put out features that have not really taken off right or even if they did take off we realized they pulled in like the kind of users who are not the most profitable users and it's like this is a great feature but it's not the one that is going to like move the business forward right now yeah uh an example is tail drop which everybody loves tail drop once they hear about it right it's sort of like airdrop on your Apple devices but it doesn't require an Apple device and it doesn't require them to be

15:31 side by side but it lets you send a file between any device and any other device pointto Point encrypted without sending it up through our server and for free it's like well that sounds great that could be a whole product it's like it could but it's kind of consumer and we didn't need to go right now in the consumer Direction because we have all these engineers and the way tail scale works is all these Engineers most of them have jobs some of them

15:54 bring it to work uh some of the people who bring it to work end up using it at work and paying us yeah right and that's great and we're going to eventually invest more in the consumer Direction but there's so much opportunity in this Direction with just Engineers that we should we I'll say we should have instead put the energy into more engineering stuff but it's not like we wasted a lot of energy tail drop was like a three-month project for our 10 person team right and it created a whole

16:18 bunch of Buzz and that whole bunch of Buzz caused the word of mouth to go up so was it a mistake it's like there's probably something we could have done that would have been better might have been a more efficient but it was not a very expensive mistake our most expensive mistake have been in the direction of like we didn't really figure out marketing for a really long time uh and you know it's funny to say that because when I say that to investors and when I said it to investors in our series a and our series B they're like what are you talking

16:40 about you're so good at marketing I'm like I'm not you have no idea yeah uh tail scale Market he not kidding we we're not good at marketing we might be now though we're we're we are suddenly getting much better at marketing um but and that that completely relates to our new VP marketing Sydney uh who who really gets it right but before that like of our success was through Word of Mouth like literally our website didn't

17:02 matter we did an AB test at one point and and we just replaced the front page with a giant button that said download tail scale and that actually increased our signup rate because the only people who went to our website were the ones whose friends had been clubbing over the head saying you need to go download tail scale right right so they would Google for the word tail scale which was like still the number one reason anyone shows up on our website right somebody told them to Google for the word tail scale

17:24 then they click on tail scale right and then they dig around trying to find the download button right and that's like that's great that means your word of mouth multiplier is really high but that means we weren't doing marketing properly and the reason we weren't doing marketing properly had to do with like you know how do you hire people how do you find the right person who's really good at the stuff that you're not good at right because I'm pretty good at hiring people to do the

17:47 things that I am good at like many Engineers can spot another good engineer right so we hired a bunch of great Engineers but like anything I'm not good at how do I know like and even if they're good at it maybe they're not the right person for your company right like marketing has like a hundred different variations of how you should do it somebody who's really good at one of the variations that variation might not be the right one for your company and So

18:10 eventually we've tried many different itations but to get to our current VP marketing for example I had to interview dozens and dozens and dozens of people and find out how they did it and ask for people's advice and ask for their advice and like what do you think about this until I finally found somebody who like explained it to explained to me how my company should work Avery what's what's been surprising you know when you build a really powerful tool I mean of course

18:32 there are like obvious ways to use it what have been some surprising use cases that have emerged you know as tail scales become more and more popular people use tail scale for really unexpected use cases uh one of the most recent ones as we we as we were like grinding through our data about like who's using tail scale we found that like you know five of the five top or AI companies were using tail scale and then we found that like hundreds of not the

18:55 top five AI companies were also using tail scale and we're like oh why I don't know any of these people they all just signed up through self- serve and never contacted us right so we had to dig around and find out like okay what about tailes scale makes us popular with AI companies um and the answer is basically they're all stuck with multicloud uh they all have to deal with some gpus they're probably all using kubernetes and they all have like all of

19:18 these like you know they have connectivity problems but they don't want to invest in a networking team there are a bunch of really smart Engineers who are AI engineers and networking is just getting in their way and they just want to spend some money to make networking problems go away and they had all heard of us because they're early adopters and they were all mostly living in San Francisco and like everybody in San Francisco now knows about tail scale um and so they just AI companies just like across the board

19:42 started adopting tail scale for everything and this is again it's great it was a surprise but if we' been on top of our game we would have seen that coming and maybe put some work into it like at the time you could search for AI tail scale and not find anything on our website there was no not a single mention of the word AI anywhere or machine learning or or llm or GPU right like there was nothing our website was not serving it at all other people were

20:05 telling other people oh you you work at in AI you should do this like they go to an AI conference and people would like ah how do I connect to my GPU it sucks and like oh I have tail skill you should try it it's free trial right and just like took off uh more recently uh we found out retroactively that tail skill is a service mesh and I'm like I don't even know what a service mesh is uh but I'm talking to customers and we have

20:27 customers who are like well we threw out our old service mesh and now we're using tail scale as a service mesh tail scale as a service mesh and I'm like we did I'm gonna have to go Google some stuff let me come back and so now I know what a service mesh is a service mesh is a combination of connectivity identity uh and service Discovery right and tail scale obviously does productivity and identity which is the thing that it does

20:50 uh I never thought of it as a service Discovery system but it turns out the architecture of tail scale is that it keeps a list of all the devices in your network because it had to generated encryption keys and track them and distribute them and then it tells each node about each other node and then each node has a list of nodes and you can query it and say like who's the list of nodes that match this criteria and that turns out to be a service Discovery mechanism and some of our customers realized that we did these three things

21:12 and are like out with the old service mesh thing and put in tail scale and again it's like there's no not a single mention of the word surface mesh anywhere on the tail scale website actually that might still be true today people are actively working on it they're going to be launching web pages that say service mesh sometime soon right but it's like these kinds of things where it's like you just have to at this point we have to keep our eyes out for what people are already doing and then listen to them and then tell

21:35 the story back and improve the experience that they're having it wouldn't it wouldn't be a tail scale podcast if I didn't at least give you the layup of telling me about crossing the chasm and I I think I think this AI sort of pervasiveness you know allowed you to keep like resurrecting crossing the chasm and board meetings for at least another like two years so um could you maybe just talk about that because I

21:57 know it's like one of your guys principles as you think about the company and in particular for the motion that you have of getting ubiquity within developers how is that you just tell us the audience about that and then how that's guided some of your decision- making I guess the the history of this uh I was introduced to the book Crossing the CM by one of my investors at my very first startup toward the end of the cycle of that startup when we were like

22:19 we were struggling uh we had like growing revenues but it was really hard to grow the revenues and in retrospect what happened was we were stuck uh in with basically early adopter customers and there were lots of early adopter customers and we were pushing really hard we had a sales team like searching as hard as they could to find early adopter customers but like every single thing was like just push push push push and they're like well Avery

22:42 you you should read this book it kind of explains what's happening I'm like ah how could a book explain what's happening we have a unique situation nobody's ever done this before blah blah blah like I read the book and the format of the book every chapter is like here's what people usually think you should do here's what happens when you do that and here's what you should do instead and so I like opened up chapter one it's like oh that's what I did oh that is what happened oh that's what I should have

23:05 done instead and then you go to the next chapter and the same and the same and it's like the story of our entire uh startup from beginning to where we were and a recipe for what to do to fix it right and so I I decided to follow the recipe and then within like six months the business turned around right but by then it was too late we' been doing it for like eight years we were running out

23:27 of money in didn't want to put in more money the ex the First Investors were like getting impatient and so we exited IBM but if I had that same exact advice for or five years sooner it could have been a completely completely different company because we had the product that we needed we didn't know how to cross this casm and the secret of crossing the chasm is just to figure out how to become um the default product that everybody uses for some use case and the

23:53 the thing that nobody realizes the super counterintuitive part is that that use case that group of people has to be tiny because you're not going to win you're not going to be the default Solution that's more than 50% is like how you be the winning default solution you're not going to be the bigger than 50% of anything big when you're tiny right so everybody wants to go after this thing with a giant total addressable Market they want to find some Market that's

24:14 like humongous and tell their investors like don't worry there's like 10 billion dollars of possibility here a trillion dollars right it's like that's not how you succeed when you're tiny right the way you succeed is you find something really small and you win that really small thing by telling everybody like look no maybe you don't have to say this out loud to the customer but like this is too tiny for anybody to care about but I care because I'm also tiny so we're going to give you the best possible

24:38 service and go out of our way to make the best possible thing and then when you win that then you can win something adjacent and something adjacent to that and something adjacent to that and it just gets bigger and bigger and bigger so like the secret strategy of tail scale and again it feeds into the name like we're we accept the idea of doing Small Things yeah right and so tail scale is always going after like what is the small thing that we can do next that

25:00 we're going to win easily right and AI was one of the things where like okay we actually were kind of late we already won it before we noticed that we had won it right uh AI networking I should say um and then you know the service mesh thing again is like oh we're not winning but it's actually like I I looked it up the other day the service mesh Market is not that big in terms of dollars right now it's actually achievable to make it

25:23 a splash in the service mesh market right right and so we'll keep doing things like that uh and growing and growing like incrementally but it's such a good book because it just it explains why this works in a you know I'm a systems design kind of person but it explains it in a systems design sort of way like this is why systems like this always happen and when you do the obvious thing this is why it always doesn't work Avery how do you think

25:46 about monetizing or charging for what you've built at tail scale you started off with this amazing Community it's very Bottoms Up you have individual developers you talked about this motion of developers at home using it bringing it to work how do you think about charging and you know kind of growing up into a business over time sure so I I said earlier that like I didn't want to build an Enterprise software company

26:09 which is true but I want to maybe I should qualify that because like obviously our first customer was a bank uh sales scale is suitable for Enterprises what but I want to like make the internet a better place right and if you're going to fix the internet if you're going to fix this like lowlevel protocol of tcpip and get it out to everybody then it has to be literally everybody everybody includes Enterprises But it includes everybody else and so

26:30 tail scale my other favorite Business book is called The innovator's Dilemma and if I was to summarize it in like one line it's like nobody ever goes down Market they only ever go up market and so if you start up Market you're never going to go back down right so tail scale our policy is like we are going to make sure that the zero Doll part of the market is ours and we're going to do that by a giving away the product to people who want to use it for free and B

26:55 making sure that it's cost effective to give away the product at people for free free because I what I can't have is just I'm going to raise a bunch of money from investors and then spend all that money uh giving away like AWS credits or something like that so that people can have the product for free so I have this blog post called how tail scale remains free um and it explains the architecture of tail scale and why it doesn't cost us anything for you to have your free account and if it doesn't cost us

27:17 anything for you to have your free account we can keep giving it out to lots of people and in fact in the early days of tail scale because you know I I talked about how uh in the very early days we didn't even have a signup system that was not aver sending you an email uh but later we're like okay well we have lots of work to do why should we Implement restrictions on what customers can do so there was no actual limits like if you bought a 10 user account or if you didn't buy anything there was

27:40 nothing stopping you from signing up another hundred or few hundred users and that's what we actually got and so when we hired our first few salespeople the first few salespeople job was just like hey we've got some customers with like hundreds of seats they never actually emailed us we should probably tell them that they're actually supposed to pay for the product if you dug around if you actually went to the billing page it would tell you like whether you were

28:01 over your subscription but it was very Canadian it said like hey you know it's great that you're trying this thing uh probably you should pay us sometime I think I can't remember the exact wording but it was kind of like that I was uh I always told the people at Excel that you're uh the company is capital c Canadian and lower C capitalist it's it's stories like that that but but you know as a Canadian I want to point out that despite all of

28:24 that uh the amount of Revenue that we could capture by going out and collecting all the money from people who were underpaying us was only about 20% extra that means like 80% of the people in the world were paying us even though there was absolutely positively nothing making them do that right because most people in the world are honest uh and that that is you know a fundamental belief of Canadians of course but also like you know that extra 20% like I

28:46 could afford to have that 20% for the huge amount of Word of Mouth that It produced uh but anyway so as tailes scale has been getting bigger we have a more structured sales team now I actually got complaints from various people that I knew is like when they signed up for tail skill and started using it they're like Avery you're doing something wrong I'm not going to pay you until somebody reaches out to me and tells me to pay like my my own friends were telling me this it's like Avery fix

29:08 your company it doesn't make sense nobody's going to be mad because they've got 50 seats that they're really enjoying and somebody emails them saying like oh we should have a discussion about dollars now right and so we started doing that we have a real professional sales team uh we still don't actually enforce too many limits but the sales team will catch you sooner uh and invite a conversation right and so we've been getting bigger and bigger now and we're starting to actually do

29:30 our first like Million Dollar Plus deals and so when I said I don't want to do Enterprise sales it's like somebody now we have employees there are people who are going to do the Enterprise sales but I have to remember like where we came from and where we came from as individual Engineers who love the product they use it at home they bring it to work they use it in small teams and eventually it turns into a system where people buy it top down it roll it

29:52 out to the whole company and so that's where tail scale like the whole vision of the new internet like it can it can scale all the way down from zero to maximum and that's how you get the new tcbp rolled out to everybody you talked about transferring files between devices you talked about two-factor authentication you talked about vpns you talked about connecting

30:14 devices and now you're talking about service mesh and you're talking about hey AI networking how do you think about defining the vision for the company I mean you know when you stand up at an all hands in front of the whole company you know how do you describe to them what the future looks like and what does that mean to you um in terms of the mission of the business and what the implications are on the world that we're headed towards well I'm kind of workshopping this one uh the one the the

30:40 way we have been saying it we'll do it live let's go okay okay yeah yeah so what we say is like tail skill is the new internet and I like it because it's funny because there's a TV show Silicon Valley where like the main character has made this thing that was called the new internet it was just like a series of laughs but actually the internet needs to be fixed and I mean by that is like as an engineer tcpip is the problem right and we've known this since the

31:03 1990s ipv4 is not good enough they tried to launch IPv6 IPv6 should have solved a whole bunch of our networking problems have your networking problems been solved by IPv6 no right and why not large part of it is the thing has not fully rolled out and why didn't the thing fully roll out it didn't roll out because it didn't follow crossing the chasm right like they didn't have a roll out plan that

31:26 was going to win and it's still hasn't won right and it doesn't look like it's going to win if you look at the trend line it looks like maybe in the next 40 or 50 years maybe they will get to like 80 or 90% adoption it's like I might be dead by then right I've been like waiting for IPv6 my entire life right and it's not solving the problem like IPv6 was the chosen one it didn't work out the new internet didn't happen it was it was the new internet so what are

31:48 you going to do about it right tail scale is actually and again it was kind of by accident because I just wanted to fix some problems tail scale is this thing you insert at the IP layer that makes the internet work the way the internet was supposed to work right and when the internet works the way it was supposed to work any device can talk to any device you've got safety you've got security you've got encryption you've got identity you've got like e you know

32:10 easy to set up you don't think about it anymore it just works properly right and when things work properly like one of the things that's hard to explain about the vision is like when people were inventing tcpip they didn't really think that big they were like you know what I want to be able to access the supercomputer at my University from another University he's like you know that sounds neat but it doesn't sound like the internet we

32:32 have today right the internet we have today is used for everything all the time right I have internet on my watch if I if I bought a watch and it didn't have internet I'd be like why am I paying for this it's it's broken right like nobody thought about that we used to call them smart watches now we just call them watches exactly or smartphones like when was the last time you phone somebody on your phone right like it's so what is the vision for tail scale the

32:55 vision for tail scale like fundamentally on an engine engering level is like look we just need to fix the thing that we call the internet and replace it with this new thing and the layers on top can be actually the same but a whole bunch of stuff is going to work better right and then how do you explain like what's the business value of that what are people going to do with it it's like it's a little hard to explain because most of the uses for tail scale haven't

33:17 been imagined yet right and I think the only way I can explain that is through that analogy and so even though it's a joke like the new internet analogy is actually the right analogy like the internet is no longer living up to the expectations that we had of it in the 1990s because IPv6 didn't work and even if IPv6 did roll out now you know it's been 25 years since it was designed it's missing some stuff that we would have

33:40 put into it if we designed it today but it's just been like stuck and if you unstuck something that's 25 years old in the tech world you're going to get a whole bunch of benefits okay so that was incredible by the way that was pretty good pretty good for a workshop if you could go back and give any feedback or advice or wisdom to Avery outside of Reed crossing the chasm four years

34:02 earlier what would you what would you tell them I think tail scale we did a really good job building the company through the first several phases I think what we didn't do a fantastic job of is as we're switching from what I would call a like early stage startup to a growth stage startup you really have to restructure the way the company Works to make it more scalable right and that

34:24 means building an executive team in the right structure so that you can can like help people like get their work done when not everybody is going to be able to know what everybody else is doing all the time and so I didn't do a super great job of creating the structure inside the company that I probably shouldn't like the organization of the executive team is really important uh you don't need to be super Innovative

34:47 there's like again you're not the first person to build a company right uh and so there's lots of advice you can find on like okay what's the good structure of an executive team how do you choose what rle should go here there's only a few structures that like reliably work and so you don't want to innovate on everything in your company it's great to be Innovative on the technology side but you don't need to invent everything you don't need to invent your own accounting systems Finance systems and so on uh the

35:10 most important thing for me when building an executive team that I learned is like actually executive search uh companies are amazing and they're worth the weight in gold uh much more than I realized because the main thing they do is they introduce you to lots and lots of great candidates some of whom you won't even be able to land some of you don't even want to land but you get such a wide array of information about what's possible in a particular

35:34 role that after you've talked to like 25 or 30 of these people then you can understand what perfect looks like for you right and when you're when you're a tiny little company and you're only hiring one executive it doesn't matter if if there's not a hundred different people who can fit that exact role that you've invented if there's just one person who can do that exact combination of things that you want you can just find that one person and that one person

35:57 is really going to want to work there because the job that's absolutely perfect for them right and we found like that kind of Executives but I didn't realize before that it was possible to do that because I didn't connect to the fact ironically given that tail scale is all about small things right I didn't really connect to the fact that you only need one perfect person for this job and that person is there somewhere and you can just go find them but it's going to be a lot of work and an executive Search firm will help do that it's awesome yeah

36:20 I think I think you've done a really good job like in those in those situations one thing I've grown to really appreciate about you is you definitely take time to get all the data but once you have the data I I really trust your decision making and it's worked out pretty well for us my comment on data is I love data but in fact like most of the most useful data in my life turns out to be like anic data or like building an intuition for something so

36:43 maybe I'll stare at the data for a long time it's like okay I think I understand what the pattern is now and then close down the dashboard and like with the idea about the pattern you talk to people you get advice what's the com what's the pattern of advice what's the standard way to do things are we really special enough to be breaking this pattern or should we just stand do the standard way to do things uh and then just like you know there's there's a little bit of like just following your intuition but to make your intuition

37:06 smart enough requires a lot of studying which is how I do it it's awesome Avery thanks for joining us you're great dude [Music]

NPS, the good parts

Tue, 05 Dec 2023 05:01:12 +0000

The Net Promoter Score (NPS) is a statistically questionable way to turn a set of 10-point ratings into a single number you can compare with other NPSes. That's not the good part.

Humans

To understand the good parts, first we have to start with humans. Humans have emotions, and those emotions are what they mostly use when asked to rate things on a 10-point scale.

Almost exactly twenty years ago, I wrote about sitting on a plane next to a musician who told me about music album reviews. The worst rating an artist can receive, he said, is a lukewarm one. If people think your music is neutral, it means you didn't make them feel anything at all. You failed. Someone might buy music that reviewers hate, or buy music that people love, but they aren't really that interested in music that is just kinda meh. They listen to music because they want to feel something.

(At the time I contrasted that with tech reviews in computer magazines (remember those?), and how negative ratings were the worst thing for a tech product, so magazines never produced them, lest they get fewer free samples. All these years later, journalism is dead but we're still debating the ethics of game companies sponsoring Twitch streams. You can bet there's no sponsored game that gets an actively negative review during 5+ hours of gameplay and still gets more money from that sponsor. If artists just want you to feel something, but no vendor will pay for a game review that says it sucks, I wonder what that says about video game companies and art?)

Anyway, when you ask regular humans, who are not being sponsored, to rate things on a 10-point scale, they will rate based on their emotions. Most of the ratings will be just kinda meh, because most products are, if we're honest, just kinda meh. I go through most of my days using a variety of products and services that do not, on any more than the rarest basis, elicit any emotion at all. Mostly I don't notice those. I notice when I have experiences that are surprisingly good, or (less surprisingly but still notably) bad. Or, I notice when one of the services in any of those three categories asks me to rate them on a 10-point scale.

The moment

The moment when they ask me is important. Many products and services are just kinda invisibly meh, most of the time, so perhaps I'd give them a meh rating. But if my bluetooth headphones are currently failing to connect, or I just had to use an airline's online international check-in system and it once again rejected my passport for no reason, then maybe my score will be extra low. Or if Apple releases a new laptop that finally brings back a non-sucky keyboard after making laptops with sucky keyboards for literally years because of some obscure internal political battle, maybe I'll give a high rating for a while.

If you're a person who likes manipulating ratings, you'll figure out what moments are best for asking for the rating you want. But let's assume you're above that sort of thing, because that's not one of the good parts.

The calibration

Just now I said that if I'm using an invisible meh product or service, I would rate it with a meh rating. But that's not true in real life, because even though I was having no emotion about, say, Google Meet during a call, perhaps when they ask me (after every...single...call) how it was, that makes me feel an emotion after all. Maybe that emotion is "leave me alone, you ask me this way too often." Or maybe I've learned that if I pick anything other than five stars, I get a clicky multi-tab questionnaire that I don't have time to answer, so I almost always pick five stars unless the experience was so bad that I feel it's worth an extra minute because I simply need to tell the unresponsive and uncaring machine how I really feel.

Google Meet never gets a meh rating. It's designed not to. In Google Meet, meh gets five stars.

Or maybe I bought something from Amazon and it came with a thank-you card begging for a 5-star rating (this happens). Or a restaurant offers free stuff if I leave a 5-star rating and prove it (this happens). Or I ride in an Uber and there's a sign on the back seat talking about how they really need a 5-star rating because this job is essential so they can support their family and too many 4-star ratings get them disqualified (this happens, though apparently not at UberEats). Okay. As one of my high school teachers, Physics I think, once said, "A's don't cost me anything. What grade do you want?" (He was that kind of teacher. I learned a lot.)

I'm not a professional reviewer. Almost nobody you ask is a professional reviewer. Most people don't actually care; they have no basis for comparison; just about anything will influence their score. They will not feel badly about this. They're just trying to exit your stupid popup interruption as quickly as possible, and half the time they would have mashed the X button instead but you hid it, so they mashed this one instead. People's answers will be... untrustworthy at best.

That's not the good part.

And yet

And yet. As in so many things, randomness tends to average out, probably into a Gaussian distribution, says the Central Limit Theorem.

The Central Limit Theorem is the fun-destroying reason that you can't just average 10-point ratings or star ratings and get something useful: most scores are meh, a few are extra bad, a few are extra good, and the next thing you know, every Uber driver is a 4.997. Or you can ship a bobcat one in 30 times and still get 97% positive feedback.

There's some deep truth hidden in NPS calculations: that meh ratings mean nothing, that the frequency of strong emotions matters a lot, and that deliriously happy moments don't average out disastrous ones.

Deming might call this the continuous region and the "special causes" (outliers). NPS is all about counting outliers, and averages don't work on outliers.

The degrees of meh

Just kidding, there are no degrees of meh. If you're not feeling anything, you're just not. You're not feeling more nothing, or less nothing.

One of my friends used to say, on a scale of 6 to 9, how good is this? It was a joke about how nobody ever gives a score less than 6 out of 10, and nothing ever deserves a 10. It was one of those jokes that was never funny because they always had to explain it. But they seemed to enjoy explaining it, and after hearing the explanation the first several times, that part was kinda funny. Anyway, if you took the 6-to-9 instructions seriously, you'd end up rating almost everything between 7 and 8, just to save room for something unimaginably bad or unimaginably good, just like you did with 1-to-10, so it didn't help at all.

And so, the NPS people say, rather than changing the scale, let's just define meaningful regions in the existing scale. Only very angry people use scores like 1-6. Only very happy people use scores like 9 or 10. And if you're not one of those you're meh. It doesn't matter how meh. And in fact, it doesn't matter much whether you're "5 angry" or "1 angry"; that says more about your internal rating system than about the degree of what you experienced. Similarly with 9 vs 10; it seems like you're quite happy. Let's not split hairs.

So with NPS we take a 10-point scale and turn it into a 3-point scale. The exact opposite of my old friend: you know people misuse the 10-point scale, but instead of giving them a new 3-point scale to misuse, you just postprocess the 10-point scale to clean it up. And now we have a 3-point scale with 3 meaningful points. That's a good part.

Evangelism

So then what? Average out the measurements on the newly calibrated 1-2-3 scale, right?

Still no. It turns out there are three kinds of people: the ones so mad they will tell everyone how mad they are about your thing; the ones who don't care and will never think about you again if they can avoid it; and the ones who had such an over-the-top amazing experience that they will tell everyone how happy they are about your thing.

NPS says, you really care about the 1s and the 3s, but averaging them makes no sense. And the 2s have no effect on anything, so you can just leave them out.

Cool, right?

Pretty cool. Unfortunately, that's still two valuable numbers but we promised you one single score. So NPS says, let's subtract them! Yay! Okay, no. That's not the good part.

The threefold path

I like to look at it this way instead. First of all, we have computers now, we're not tracking ratings on one of those 1980s desktop bookkeeping printer-calculators, you don't have to make every analysis into one single all-encompassing number.

Postprocessing a 10-point scale into a 3-point one, that seems pretty smart. But you have to stop there. Maybe you now have three separate aggregate numbers. That's tough, I'm sorry. Here's a nickel, kid, go sell your personal information in exchange for a spreadsheet app. (I don't know what you'll do with the nickel. Anyway I don't need it. Here. Go.)

Each of those three rating types gives you something different you can do in response:

The ones had a very bad experience, which is hopefully an outlier, unless you're Comcast or the New York Times subscription department. Normally you want to get rid of every bad experience. The absence of awful isn't greatness, it's just meh, but meh is infinitely better than awful. Eliminating negative outliers is a whole job. It's a job filled with Deming's special causes. It's hard, and it requires creativity, but it really matters.
The twos had a meh experience. This is, most commonly, the majority. But perhaps they could have had a better experience. Perhaps even a great one? Deming would say you can and should work to improve the average experience and reduce the standard deviation. That's the dream; heck, what if the average experience could be an amazing one? That's rarely achieved, but a few products achieve it, especially luxury brands. And maybe that Broadway show, Hamilton? I don't know, I couldn't get tickets, because everyone said it was great so it was always sold out and I guess that's my point.

If getting the average up to three is too hard or will take too long (and it will take a long time!), you could still try to at least randomly turn a few of them into threes. For example, they say users who have a great customer support experience often rate a product more highly than the ones who never needed to contact support at all, because the support interaction made the company feel more personal. Maybe you can't afford to interact with everyone, but if you have to interact anyway, perhaps you can use that chance to make it great instead of meh.
The threes already had an amazing experience. Nothing to do, right? No! These are the people who are, or who can become, your superfan evangelists. Sometimes that happens on its own, but often people don't know where to put that excess positive energy. You can help them. Pop stars and fashion brands know all about this; get some true believers really excited about your product, and the impact is huge. This is a completely different job than turning ones into twos, or twos into threes.

What not to do

Those are all good parts. Let's ignore that unfortunately they aren't part of NPS at all and we've strayed way off topic.

From here, there are several additional things you can do, but it turns out you shouldn't.

Don't compare scores with other products. I guarantee you, your methodology isn't the same as theirs. The slightest change in timing or presentation will change the score in incomparable ways. You just can't. I'm sorry.

Don't reward your team based on aggregate ratings. They will find a way to change the ratings. Trust me, it's too easy.

Don't average or difference the bad with the great. The two groups have nothing to do with each other, require completely different responses (usually from different teams), and are often very small. They're outliers after all. They're by definition not the mainstream. Outlier data is very noisy and each terrible experience is different from the others; each deliriously happy experience is special. As the famous writer said, all meh families are alike.

Don't fret about which "standard" rating ranges translate to bad-meh-good. Your particular survey or product will have the bad outliers, the big centre, and the great outliers. Run your survey enough and you'll be able to find them.

Don't call it NPS. NPS nowadays has a bad reputation. Nobody can really explain the bad reputation; I've asked. But they've all heard it's bad and wrong and misguided and unscientific and "not real statistics" and gives wrong answers and leads to bad incentives. You don't want that stigma attached to your survey mechanic. But if you call it a satisfaction survey on a 10-point or 5-point scale, tada, clear skies and lush green fields ahead.

Bonus advice

Perhaps the neatest thing about NPS is how much information you can get from just one simple question that can be answered with the same effort it takes to dismiss a popup.

I joked about Google Meet earlier, but I wasn't really kidding; after having a few meetings, if I had learned that I could just rank from 1 to 5 stars and then not get guilted for giving anything other than 5, I would do it. It would be great science and pretty unobtrusive. As it is, I lie instead. (I don't even skip, because it's faster to get back to the menu by lying than by skipping.)

While we're here, only the weirdest people want to answer a survey that says it will take "just 5 minutes" or "just 30 seconds." I don't have 30 seconds, I'm busy being mad/meh/excited about your product, I have other things to do! But I can click just one single star rating, as long as I'm 100% confident that the survey will go the heck away after that. (And don't even get me started about the extra layer in "Can we ask you a few simple questions about our website? Yes or no")

Also, don't be the survey that promises one question and then asks "just one more question." Be the survey that gets a reputation for really truly asking that one question. Then ask it, optionally, in more places and more often. A good role model is those knowledgebases where every article offers just thumbs up or thumbs down (or the default of no click, which means meh). That way you can legitimately look at aggregates or even the same person's answers over time, at different points in the app, after they have different parts of the experience. And you can compare scores at the same point after you update the experience.

But for heaven's sake, not by just averaging them.

Interesting

Fri, 06 Oct 2023 20:59:31 +0000

A few conversations last week made me realize I use the word “interesting” in an unusual way.

I rely heavily on mental models. Of course, everyone relies on mental models. But I do it intentionally and I push it extra hard.

What I mean by that is, when I’m making predictions about what will happen next, I mostly don’t look around me and make a judgement based on my immediate surroundings. Instead, I look at what I see, try to match it to something inside my mental model, and then let the mental model extrapolate what “should” happen from there.

If this sounds predictably error prone: yes. It is.

But it’s also powerful, when used the right way, which I try to do. Here’s my system.

Confirmation bias

First of all, let’s acknowledge the problem with mental models: confirmation bias. Confirmation bias is the tendency of all people, including me and you, to consciously or subconsciously look for evidence to support what we already believe to be true, and try to ignore or reject evidence that disagrees with our beliefs.

This is just something your brain does. If you believe you’re exempt from this, you’re wrong, and dangerously so. Confirmation bias gives you more certainty where certainty is not necessarily warranted, and we all act on that unwarranted certainty sometimes.

On the one hand, we would all collapse from stress and probably die from bear attacks if we didn’t maintain some amount of certainty, even if it’s certainty about wrong things. But on the other hand, certainty about wrong things is pretty inefficient.

There’s a word for the feeling of stress when your brain is working hard to ignore or reject evidence against your beliefs: cognitive dissonance. Certain Internet Dingbats have recently made entire careers talking about how to build and exploit cognitive dissonance, so I’ll try to change the subject quickly, but I’ll say this: cognitive dissonance is bad… if you don’t realize you’re having it.

But your own cognitive dissonance is amazingly useful if you notice the feeling and use it as a tool.

The search for dissonance

Whether you like it or not, your brain is going to be working full time, on automatic pilot, in the background, looking for evidence to support your beliefs. But you know that; at least, you know it now because I just told you. You can be aware of this effect, but you can’t prevent it, which is annoying.

But you can try to compensate for it. What that means is using the part of your brain you have control over — the supposedly rational part — to look for the opposite: things that don’t match what you believe.

To take a slight detour, what’s the relationship between your beliefs and your mental model? For the purposes of this discussion, I’m going to say that mental models are a system for generating beliefs. Beliefs are the output of mental models. And there’s a feedback loop: beliefs are also the things you generalize in order to produce your mental model. (Self-proclaimed ”Bayesians” will know what I’m talking about here.)

So let’s put it this way: your mental model, combined with current observations, produce your set of beliefs about the world and about what will happen next.

Now, what happens if what you expected to happen next, doesn’t happen? Or something happens that was entirely unexpected? Or even, what if someone tells you you’re wrong and they expect something else to happen?

Those situations are some of the most useful ones in the world. They’re what I mean by interesting.

The “aha” moment

The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” (I found it!) but “That’s funny…”

possibly

When you encounter evidence that your mental model mismatches someone else’s model, that’s an exciting opportunity to compare and figure out which one of you is wrong (or both). Not everybody is super excited about doing that with you, so you have to be be respectful. But the most important people to surround yourself with, at least for mental model purposes, are the ones who will talk it through with you.

Or, if you get really lucky, your predictions turn out to be demonstrably concretely wrong. That’s an even bigger opportunity, because now you get to figure out what part of your mental model is mistaken, and you don’t have to negotiate with a possibly-unwilling partner in order to do it. It’s you against reality. It’s science: you had a hypothesis, you did an experiment, your hypothesis was proven wrong. Neat! Now we’re getting somewhere.

What follows is then the often-tedious process of figuring out what actual thing was wrong with your model, updating the model, generating new outputs that presumably match your current observations, and then generating new hypotheses that you can try out to see if the new model works better more generally.

For physicists, this whole process can sometimes take decades and require building multiple supercolliders. For most of us, it often takes less time than that, so we should count ourselves fortunate even if sometimes we get frustrated.

The reason we update our model, of course, is that most of the time, the update changes a lot more predictions than just the one you’re working with right now. Turning observations back into generalizable mental models allows you to learn things you’ve never been taught; perhaps things nobody has ever learned before. That’s a superpower.

Proceeding under uncertainty

But we still have a problem: that pesky slowness. Observing outcomes, updating models, generating new hypotheses, and repeating the loop, although productive, can be very time consuming. My guess is that’s why we didn’t evolve to do that loop most of the time. Analysis paralysis is no good when a tiger is chasing you and you’re worried your preconceived notion that it wants to eat you may or may not be correct.

Let’s tie this back to business for a moment.

You have evidence that your mental model about your business is not correct. For example, let’s say you have two teams of people, both very smart and well-informed, who believe conflicting things about what you should do next. That’s interesting, because first of all, your mental model is that these two groups of people are very smart and make right decisions almost all the time, or you wouldn’t have hired them. How can two conflicting things be the right decision? They probably can’t. That means we have a few possibilities:

The first group is right
The second group is right
Both groups are wrong
The appearance of conflict is actually not correct, because you missed something critical

There is also often a fifth possibility:

Okay, it’s probably one of the first four but I don’t have time to figure that out right now

In that case, there’s various wisdom out there involving one- vs two-way doors, and oxen pulling in different directions, and so on. But it comes down to this: almost always, it’s better to get everyone aligned to the same direction, even if it’s a somewhat wrong direction, than to have different people going in different directions.

To be honest, I quite dislike it when that’s necessary. But sometimes it is, and you might as well accept it in the short term.

The way I make myself feel better about it is to choose the path that will allow us to learn as much as possible, as quickly as possible, in order to update our mental models as quickly as possible (without doing too much damage) so we have fewer of these situations in the future. In other words, yes, we “bias toward action” — but maybe more of a “bias toward learning.” And even after the action has started, we don’t stop trying to figure out the truth.

Being wrong

Leaving aside many philosophers’ objections to the idea that “the truth” exists, I think we can all agree that being wrong is pretty uncomfortable. Partly that’s cognitive dissonance again, and partly it’s just being embarrassed in front of your peers. But for me, what matters more is the objective operational expense of the bad decisions we make by being wrong.

You know what’s even worse (and more embarrassing, and more expensive) than being wrong? Being wrong for even longer because we ignored the evidence in front of our eyes.

You might have to talk yourself into this point of view. For many of us, admitting wrongness hurts more than continuing wrongness. But if you can pull off that change in perspective, you’ll be able to do things few other people can.

Bonus: Strong opinions held weakly

Like many young naive nerds, when I first heard of the idea of “strong opinions held weakly,” I thought it was a pretty good idea. At least, clearly more productive than weak opinions held weakly (which are fine if you want to keep your job), or weak opinions held strongly (which usually keep you out of the spotlight).

The real competitor to strong opinions held weakly is, of course, strong opinions held strongly. We’ve all met those people. They are supremely confident and inspiring, until they inspire everyone to jump off a cliff with them.

Strong opinions held weakly, on the other hand, is really an invitation to debate. If you disagree with me, why not try to convince me otherwise? Let the best idea win.

After some decades of experience with this approach, however, I eventually learned that the problem with this framing is the word “debate.” Everyone has a mental model, but not everyone wants to debate it. And if you’re really good at debating — the thing they teach you to be, in debate club or whatever — then you learn how to “win” debates without uncovering actual truth.

Some days it feels like most of the Internet today is people “debating” their weakly-held strong beliefs and pulling out every rhetorical trick they can find, in order to “win” some kind of low-stakes war of opinion where there was no right answer in the first place.

Anyway, I don’t recommend it, it’s kind of a waste of time. The people who want to hang out with you at the debate club are the people who already, secretly, have the same mental models as you in all the ways that matter.

What’s really useful, and way harder, is to find the people who are not interested in debating you at all, and figure out why.

Tech debt metaphor maximalism

Tue, 11 Jul 2023 03:12:47 +0000

I really like the "tech debt" metaphor. A lot of people don't, but I think that's because they either don't extend the metaphor far enough, or because they don't properly understand financial debt.

So let's talk about debt!

Consumer debt vs capital investment

Back in school my professor, Canadian economics superhero Larry Smith, explained debt this way (paraphrased): debt is stupid if it's for instant gratification that you pay for later, with interest. But debt is great if it means you can make more money than the interest payments.

A family that takes on high-interest credit card debt for a visit to Disneyland is wasting money. If you think you can pay it off in a year, you'll pay 20%-ish interest for that year for no reason. You can instead save up for a year and get the same gratification next year without the 20% surcharge.

But if you want to buy a $500k machine that will earn your factory an additional $1M/year in revenue, it would be foolish not to buy it now, even with 20% interest ($100k/year). That's a profit of $900k in just the first year! (excluding depreciation)

There's a reason profitable companies with CFOs take on debt, and often the total debt increases rather than decreases over time. They're not idiots. They're making a rational choice that's win-win for everyone. (The company earns more money faster, the banks earn interest, the interest gets paid out to consumers' deposit accounts.)

Debt is bad when you take out the wrong kind, or you mismanage it, or it has weird strings attached (hello Venture Debt that requires you to put all your savings in one underinsured place). But done right, debt is a way to move faster instead of slower.

High-interest vs low-interest debt

For a consumer, the highest interest rates are for "store" credit cards, the kinds issued by Best Buy or Macy's or whatever that only work in that one store. They aren't as picky about risk (thus have more defaults) because it's the ultimate loyalty programme: it gets people to spend more at their store instead of other stores, in some cases because it's the only place that would issue those people debt in the first place.

The second-highest interest rate is on a general-purpose credit card like Visa or Mastercard. They can get away with high interest rates because they're also the payment system and so they're very convenient.

(Incidentally, when I looked at the stats a decade or so ago, in Canada credit cards make most of their income on payment fees because Canadians are annoyingly persistent about paying off their cards; in the US it's the opposite. The rumours are true: Canadians really are more cautious about spending.)

If you have a good credit rating, you can get better interest rates on a bank-issued "line of credit" (LOC) (lower interest rate, but less convenient than a card). In Canada, one reason many people pay off their credit card each month is simply that they transfer the balance to a lower-interest LOC.

Even lower interest rates can be obtained if you're willing to provide collateral: most obviously, the equity in your home. This greatly reduces the risk for the lender because they can repossess and then resell your home if you don't pay up. Which is pretty good for them even if you don't pay, but what's better is it makes you much more likely to pay rather than lose your home.

Some people argue that you should almost never plan to pay off your mortgage: typical mortgage interest rates are lower than the rates you'd get long-term from investing in the S&P. The advice that you should "always buy the biggest home you can afford" is often perversely accurate, especially if you believe property values will keep going up. And subject to your risk tolerance and lock-in preferences.

What's the pattern here? Just this: high-interest debt is quick and convenient but you should pay it off quickly. Sometimes you pay it off just by converting to longer-term lower-rate debt. Sometimes debt is collateralized and sometimes it isn't.

High-interest and low-interest tech debt

Bringing that back to tech debt: a simple kind of high-interest short-term debt would be committing code without tests or documentation. Yay, it works, ship it! And truthfully, maybe you should, because the revenue (and customer feedback) you get from shipping fast can outweigh how much more bug-prone you made the code in the short term.

But like all high-interest debt, you should plan to pay it back fast. Tech debt generally manifests as a slowdown in your development velocity (ie. overhead on everything else you do), which means fewer features launched in the medium-long term, which means less revenue and customer feedback.

Whoa, weird, right? This short-term high-interest debt both increases revenue and feedback rate, and decreases it. Why?

If you take a single pull request (PR) that adds a new feature, and launch it without tests or documentation, you will definitely get the benefits of that PR sooner.
Every PR you try to write after that, before adding the tests and docs (ie. repaying the debt) will be slower because you risk creating undetected bugs or running into undocumented edge cases.
If you take a long time to pay off the debt, the slowdown in future launches will outweigh the speedup from the first launch.

This is exactly how CFOs manage corporate financial debt. Debt is a drain on your revenues; the thing you did to incur the debt is a boost to your revenues; if you take too long to pay back the debt, it's an overall loss.

CFOs can calculate that. Engineers don't like to. (Partly because tech debt is less quantifiable. And partly because engineers are the sort of people who pay off their loans sooner than they mathematically should, as a matter of principle.)

Debt ceilings

The US government has imposed a famously ill-advised debt ceiling on itself, that mainly serves to cause drama and create a great place to push through unrelated riders that nobody will read, because the bill to raise the debt ceiling will always pass.

Real-life debt ceilings are defined by your creditworthiness: banks simply will not lend you more money if you've got so much outstanding debt that they don't believe you can handle the interest payments. That's your credit limit, or the largest mortgage they'll let you have.

Banks take a systematic approach to calculating the debt ceiling for each client. How much can we lend you so that you take out the biggest loan you possibly can, thus paying as much interest as possible, without starving to death or (even worse) missing more than two consecutive payments? Also, morbidly but honestly, since debts are generally not passed down to your descendants, they would like you to be able to just barely pay it all off (perhaps by selling off all your assets) right before you kick the bucket.

They can math this, they're good at it. Remember, they don't want you to pay it off early. If you have leftover money you might use it to pay down your debt. That's no good, because less debt means lower interest payments. They'd rather you incur even more debt, then use that leftover monthly income even for bigger interest payments. That's when you're trapped.

The equivalent in tech debt is when you are so far behind that you can barely keep the system running with no improvements at all; the perfect balance. If things get worse over time, you're underwater and will eventually fail. But if you reach this zen state of perfect equilibrium, you can keep going forever, running in place. That's your tech debt ceiling.

Unlike the banking world, I can't think of a way to anthropomorphize a villain who wants you to go that far into debt. Maybe the CEO? I guess maybe someone who is trying to juice revenues for a well-timed acquisition. Private Equity firms also specialize in maximizing both financial and technical debt so they can extract the assets while your company slowly dies.

Anyway, both in finance and tech, you want to stay well away from your credit limit.

Debt to income ratios

There are many imperfect rules of thumb for how much debt is healthy. (Remember, some debt is very often healthy, and only people who don't understand debt rush to pay it all off as fast as they can.)

One measure is the debt to income ratio (or for governments, the debt to GDP ratio). The problem with debt-to-income is debt and income are two different things. The first produces a mostly-predictable repayment cost spread over an undefined period of time; the other is a possibly-fast-changing benefit measured annually. One is an amount, the other is a rate.

It would be better to measure interest payments as a fraction of revenue. At least that encompasses the distinction between high-interest and low-interest loans. And it compares two cashflow rates rather than the nonsense comparison of a balance sheet measure vs a cashflow measure. Banks love interest-to-income ratios; that's why your income level has such a big impact on your debt ceiling.

In the tech world, the interest-to-income equivalent is how much time you spend dealing with overhead compared to building new revenue-generating features. Again, getting to zero overhead is probably not worth it. I like this xkcd explanation of what is and is not worth the time:

Tech debt, in its simplest form, is the time you didn't spend making tasks more efficient. When you think of it that way, it's obvious that zero tech debt is a silly choice.

(Note that the interest-to-income ratio in this formulation has nothing to do with financial income. "Tech income" in our metaphor is feature development time, where "tech debt" is what eats up your development time.)

(Also note that by this definiton, nowadays tech stacks are so big, complex, and irritable that every project starts with a giant pile of someone else's tech debt on day 1. Enjoy!)

Debt to equity ratios

Interest-to-income ratios compare two items from your cashflow statement. Debt-to-equity ratios compare two items from your balance sheet. Which means they, too, are at least not nonsense.

"Equity" is unfortunately a lot fuzzier than income. How much is your company worth? Or your product? The potential value of a factory isn't just the value of the machines inside it; it's the amortized income stream you (or a buyer) could get from continuing to operate that factory. Which means it includes the built-up human and business expertise needed to operate the factory.

And of course, software is even worse; as many of us know but few businesspeople admit, the value of proprietary software without the people is zero. This is why you hear about acqui-hires (humans create value even if they might quit tomorrow) but never about acqui-codes (code without humans is worthless).

Anyway, for a software company the "equity" comes from a variety of factors. In the startup world, Venture Capitalists are -- and I know this is depressing -- the best we have for valuing company equity. They are, of course, not very good at it, but they make it up in volume. As software companies get more mature, valuation becomes more quantifiable and comes back to expectations for the future cashflow statement.

Venture Debt is typically weighted heavily on equity (expected future value) and somewhat less on revenue (ability to pay the interest).

As the company builds up assets and shows faster growth, the assumed equity value gets bigger and bigger. In the financial world, that means people are willing to issue more debt.

(Over in the consumer world: your home is equity. That's why you can get a huge mortgage on a house but your unsecured loan limit is much smaller. So Venture Debt is like a mortgage.)

Anyway, back to tech debt: the debt-to-equity ratio is how much tech debt you've taken on compared to the accumulated value, and future growth rate, of your product quality. If your product is acquiring lots of customers fast, you can afford to take on more tech debt so you can acquire more customers even faster.

What's weirder is that as the absolute value of product equity increases, you can take on a larger and larger absolute value of tech debt.

That feels unexpected. If we're doing so well, why would we want to take on more tech debt? But think of it this way: if your product (thus company) are really growing that fast, you will have more people to pay down the tech debt next year than you do now. In theory, you could even take on so much tech debt this year that your current team can't even pay the interest...

...which brings us to leverage. And risk.

Leverage risk

Earlier in this article, I mentioned the popular (and surprisingly, often correct!) idea that you should "buy the biggest house you can afford." Why would I want a bigger house? My house is fine. I have a big enough house. How is this good advice?

The answer is the amazing multiplying power of leverage.

Let's say housing goes up at 5%/year. (I wish it didn't because this rate is fabulously unsustainable. But bear with me.) And let's say you have $100k in savings and $100k in annual income.

You could pay cash and buy a house for $100k. Woo hoo, no mortgage! And it'll go up in value by about $5k/year, which is not bad I guess.

Or, you could buy a $200k house: a $100k down payment and a $100k mortgage at, say, 3% (fairly common back in 2021), which means $3k/year in interest. But your $200k house goes up by 5% = $10k/year. Now you have an annual gain of $10k - $3k = $7k, much more than the $5k you were making before, with the same money. Sweet!

But don't stop there. If the bank will let you get away with it, why not a $1M house with a $100k down payment? That's $1M x 5% = +$50k/year in value, and $900k x 3% = $27k in interest, so a solid $23k in annual (unrealized) capital gain. From the same initial bank balance! Omg we're printing money.

(Obviously we're omitting maintenance costs and property tax here. Forgive me. On the other hand, presumably you're getting intangible value from living in a much bigger and fancier house. $AAPL shares don't have skylights and rumpus rooms and that weird statue in bedroom number seven.)

What's the catch? Well, the catch is massively increasing risk.

Let's say you lose your job and can't afford interest payments. If you bought your $100k house with no mortgage, you're in luck: that house is yours, free and clear. You might not have food but you have a place to live.

If you bought the $1M house and have $900k worth of mortgage payments to keep up, you're screwed. Get another job or get ready to move out and disrupt your family and change everything about your standard of living, up to and possibly including bankruptcy, which we'll get to in a bit.

Similarly, let's imagine that your property value stops increasing, or (less common in the US for stupid reasons, but common everywhere else) mortage rates go up. The leverage effect multiplies your potential losses just like it multiplies your potential gains.

Back to tech debt. What's the analogy?

Remember that idea I had above, of incurring extra tech debt this year to keep the revenue growth rolling, and then planning to pay it off next year with the newer and bigger team? Yeah, that actually works... if you keep growing. If you estimated your tech debt interest rate correctly. If that future team materializes. (If you can even motivate that future team to work on tech debt.) If you're rational, next year, about whether you borrow more or not.

That thing I said about the perfect equilibrium running-in-place state, when you spend all your time just keeping the machine operating and you have no time to make it better. How do so many companies get themselves into that state? In a word, leverage. They guessed wrong. The growth rate fell off, the new team members didn't materialize or didn't ramp up fast enough.

And if you go past equilibrium, you get the worst case: your tech debt interest is greater than your tech production (income). Things get worse and worse and you enter the downward spiral. This is where desperation sets in. The only remaining option is ~~bankruptcy~~ Tech Debt Refinancing.

Refinancing

Most people who can't afford the interest on their loans don't declare bankruptcy. The step before that is to make an arrangement with your creditors to lower your interest payments. Why would they accept such an agreement? Because if they don't, you'll declare bankruptcy, which is annoying for you but hugely unprofitable for them.

The tech metaphor for refinancing is premature deprecation. Yes, people love both service A and service B. Yes, we are even running both services at financial breakeven. But they are slipping, slipping, getting a little worse every month and digging into a hole that I can't escape. In order to pull out of this, I have to stop my payments on A so I can pay back more of B; by then A will be unrecoverably broken. But at least B will live on, to fight another day.

Companies do this all the time. Even at huge profitable companies, in some corners you'll occasionally find an understaffed project sliding deeper and deeper into tech debt. Users may still love it, and it may even be net profitable, but not profitable enough to pay for the additional engineering time to dig it out. Such a project is destined to die, and the only question is when. The answer is "whenever some executive finally notices."

Bankruptcy

The tech bankruptcy metaphor is an easy one: if refinancing doesn't work and your tech debt continues to spiral downward, sooner or later your finances will follow. When you run out of money you declare bankruptcy; what's interesting is your tech debt disappears at the same time your financial debt does.

This is a really important point. You can incur all the tech debt in the world, and while your company is still operating, you at least have some chance of someday paying it back. When your company finally dies, you will find yourself off the hook; the tech debt never needs to be repaid.

Okay, for those of us grinding away at code all day, perhaps that sounds perversely refreshing. But it explains lots of corporate behaviour. The more desperate a company gets, the less they care about tech debt. Anything to turn a profit. They're not wrong to do so, but you can see how the downward spiral begins to spiral downward. The more tech debt you incur, the slower your development goes, and the harder it is to do something productive that might make you profitable. You might still pull it off! But your luck will get progressively worse.

The reverse is also true. When your company is doing well, you have time to pay back tech debt, or at least to control precisely how much debt you take on and when. To maintain your interest-to-income ratio or debt-to-equity ratio at a reasonable level.

When you see a company managing their tech debt carefully, you see a company that is planning for the long term rather than a quick exit. Again, that doesn't mean paying it all back. It means being careful.

Student loans that are non-dischargeable in bankruptcy

Since we're here anyway talking about finance, let's talk about the idiotic US government policy of guaranteeing student loans, but also not allowing people to discharge those loans (ie. zero them out) in bankruptcy.

What's the effect of this? Well, of course, banks are extremely eager to give these loans out to anybody, at any scale, as fast as they can, because they can't lose. They have all the equity of the US government to back them up. The debt-to-equity ratio is effectively zero.

And of course, people who don't understand finance (which they don't teach you until university; catch-22!) take on lots of these loans in the hope of making money in the future.

Since anyone who wants to go to university can get a student loan, American universities keep raising their rates until they find the maximum amount that lenders are willing to lend (unlimited!) or foolish borrowers are willing to borrow in the name of the American Dream (so far we haven't found the limit).

Where was I? Oh right, tech metaphors.

Well, there are two parts here. First, unlimited access to money. Well, the tech world has had plenty of that, prior to the 2022 crash anyway. The result is they hired way too many engineers (students) who did a lot of dumb stuff (going to school) and incurred a lot of tech debt (student loans) that they promised to pay back later when their team got bigger (they earned their Bachelor's degree and got a job), which unfortunately didn't materialize. Oops. They are worse off than if they had skipped all that.

Second, inability to discharge the debt in bankruptcy. Okay, you got me. Maybe we've come to the end of our analogy. Maybe US government policies actually, and this is quite an achievement, manage to be even dumber than tech company management. In this one way. Maybe.

OR MAYBE YOU OPEN SOURCED WVDIAL AND PEOPLE STILL EMAIL YOU FOR HELP DECADES AFTER YOUR FIRST STARTUP IS LONG GONE.

Um, sorry for that outburst. I have no idea where that came from.

Bonus note: bug bankruptcy

While we're here exploring financial metaphors, I might as well say something about bug bankruptcy. Although I have been known to make fun of bug bankruptcy, it too is an excellent metaphor, but only if you take it far enough.

For those who haven't heard of this concept, bug bankruptcy happens when your bug tracking database is so full of bugs that you give up and delete them all and start over ("declare bankruptcy").

Like financial bankruptcy, it is very tempting: I have this big pile of bills. Gosh, it is a big pile. Downright daunting, if we're honest. Chances are, if I opened all these bills, I would find out that I owe more money than I have, and moreover, next month a bunch more bills will come and I won't be able to pay them either and this is hopeless. That would be stressful. My solution, therefore, is to throw all the bills in the dumpster, call up my friendly neighbourhood bankruptcy trustee, and conveniently discharge all my debt once and for all.

Right?

Well, not so fast, buddy. Bankruptcy has consequences. First of all, it's kind of annoying to arrange legally. Secondly, it sits on your financial records for like 7 years afterwards, during which time probably nobody will be willing to issue you any loans, because you're empirically the kind of person who does not pay back their loans.

And that, my friends, is also how bug bankruptcy works. Although the process for declaring it is easier -- no lawyers or trustees required! -- the long-term destruction of trust is real. If you run a project in which a lot of people spent a bunch of effort filing and investigating bugs (ie. lent you their time in the hope that you'll pay it back by fixing the bugs later), and you just close them all wholesale, you can expect that those people will eventually stop filing bugs. Which, you know, admittedly feels better, just like the hydro company not sending you bills anymore feels better until winter comes and your heater doesn't work and you can't figure out why and you eventually remember "oh, I think someone said this might happen but I forget the details."

Anyway, yes, you can do it. But refinancing is better.

Email bankruptcy

Email bankruptcy is similar to bug bankruptcy, with one important distinction: nobody ever expected you to answer your email anyway. I'm honestly not sure why people keep sending them.

ESPECIALLY EMAILS ABOUT WVDIAL where does that voice keep coming from

Systems design 2: What we hope we know

Tue, 11 Jul 2023 03:12:47 +0000

Someone asked if I could write about the rise of AI and Large Language Models (LLMs) and what I think that means for the future of people, technology, society, and so on. Although that's a fun topic, it left me with two problems: I know approximately nothing about AI, and predicting the future is hard even for people who know what they're talking about.

Let's try something else instead. I'll tell you a bunch of things I do know that are somehow related to the topic, and then you can predict the future yourself.

Magic

I think magic gets a bad reputation for no good reason.

First of all, you might be thinking: magic doesn't actually exist. I assure you that it does. We just need to agree on a definition. For our purposes, let's define magic as: something you know is there, but you can't explain.

Any sufficiently advanced technology is indistinguishable from magic.
— Arthur C. Clarke

One outcome of this definition is that something which is mundane and obvious to one person can be magic to another. Many of us understand this concept unconsciously; outside of storybooks, we more often say something "feels like" magic than we say it "is" magic. Magic is a feeling. Sometimes it's a pleasant feeling, when things go better than they should for reasons we don't understand. Sometimes it's an annoying feeling, when something works differently than expected and you really want to know why.

People often say Tailscale feels like magic. This is not a coincidence. I've never seen anyone say ChatGPT feels like magic. That makes me curious.

Magical thinking

On the other hand, people who believe AIs and specifically LLMs possess "intelligence" are often accused of "magical thinking." Unlike magic itself, magical thinking is always used derisively. Since we now know what magic means, we know what magical thinking means: a tendency to interpret something as magic instead of trying to understand it. A tendency to care about outcomes rather than mechanisms. The underlying assumption, when someone says you're a victim of magical thinking, is that if you understood the mechanisms, you could make better predictions.

When it comes to AI, I doubt it.

The mechanisms used in AI systems are pretty simple. But at a large scale, combined cleverly, they create amazingly complex emergent outcomes far beyond what we put in.

Emergent outcomes defy expectations. Understanding how transistors work doesn't help you at all to explain why Siri sucks. Understanding semi-permeable cell membranes doesn't help much in figuring out what's the deal with frogs. Mechanisms are not the right level of abstraction at all. You can't get there from here.

Magical thinking, it turns out, is absolutely essential to understanding any emergent system. You have to believe in magic to understand anything truly complex.

You see, magical thinking is just another way to say Systems Design.

Emergent complexity

I don't want to go too far into emergent complexity, but I think it's worth a detour since many of us have not thought much about it. Let me link you to three things you might want to read more about.

First and most newsworthy at the moment, there's the recently discovered aperiodic monotile:

Monotile image
by Smith, Myers, Kaplan, and Goodman-Strauss, 2023

The monotile is a surprisingly simple shape that, when tiled compactly across a plane of any size, creates a never-repeating pattern. It was hard to discover but it's easy to use, and creates endlessly variable, endlessly complex output from a very simple input.

Secondly, I really enjoyed The Infinite Staircase by Geoffrey Moore. It's a philosophy book, but it will never be accepted as serious philosophy because it's not written the right way. That said, it draws a map from entropy, to life, to genetics, to memetics, showing how at each step along the ladder, emergent complexity unexpectedly produces a new level that has fundamentally different characteristics from the earlier one. It's a bit of a slog to read but it says things I've never seen anywhere else. Moore even offers a solution to the mind/body duality problem. If you like systems, I think you'll like it.

Thirdly, the book A New Kind of Science by Stephen Wolfram has lots and lots of examples of emergent complexity, starting with simple finite state automatons of the sort you might recognize from Conway's Game of Life (though even simpler). For various reasons, the book got a bad reputation and the author appears to be widely disliked. Part of the book's bad reputation is that it claims to describe "science" but was self-published and not peer reviewed, completely unlike science. True, but it's shortsighted to discount the content because of that.

The book also made a lot of people mad by saying certain important empirical observations in physics and biology can't be reduced to a math formula, but can be reduced to simple iteration rules. The reasons people got mad about that seem to be:

an iteration rule is technically a math formula;
just using iterations instead of formulas hardly justifies calling for a "New Kind of Science" as if there were something wrong with the old kind of science;
scientists absolutely bloody despise emergent complexity → systems design → magical thinking.

Science is the opposite of magical thinking. By definition. Right?

Hypotheses

A friend's favourite book growing up was Zen and the Art of Motorcycle Maintenance. It's an unusual novel that is not especially about motorcycle maintenance, although actually it does contain quite a lot of motorcycle maintenance. It's worth reading, if you haven't, and even more worth reading if you're no longer in high school because I think some of the topics are deeper than they appear at first.

Here's one of my highlights:

Part Three, that part of formal scientific method called experimentation, is sometimes thought of by romantics as all of science itself because that’s the only part with much visual surface. They see lots of test tubes and bizarre equipment and people running around making discoveries. They do not see the experiment as part of a larger intellectual process and so they often confuse experiments with demonstrations, which look the same. A man conducting a gee-whiz science show with fifty thousand dollars’ worth of Frankenstein equipment is not doing anything scientific if he knows beforehand what the results of his efforts are going to be. A motorcycle mechanic, on the other hand, who honks the horn to see if the battery works is informally conducting a true scientific experiment. He is testing a hypothesis by putting the question to nature.

The formation of hypotheses is the most mysterious of all the categories of scientific method. Where they come from, no one knows. A person is sitting somewhere, minding his own business, and suddenly—flash!—he understands something he didn’t understand before. Until it’s tested the hypothesis isn’t truth. For the tests aren’t its source. Its source is somewhere else.

A lesser scientist than Einstein might have said, “But scientific knowledge comes from nature. Nature provides the hypotheses.” But Einstein understood that nature does not. Nature provides only experimental data.

-- Zen and the Art of Motorcycle Maintenance

I love this observation: much of science is straightforward, logical, almost rote. It's easy to ask questions; toddlers do it. It's not too hard to hire grad student lab assistants to execute experiments. It's relatively easy for an analyst or statistician to look at a pile of observations from an experiment and draw conclusions.

There's just one really hard step, the middle step: coming up with testable hypotheses. By testable, we mean, we can design an experiment that is actually possible to execute, that will tell us if the hypothesis is true or not, hopefully leading toward answering the original question. Testable hypotheses are, I've heard, where string theory falls flat. We have lots of theories, lots of hypotheses, and billions of dollars to build supercolliders, but we are surprisingly short of things we are able to test for, in order to make the next leap forward.

The book asks, where do hypotheses come from?

Science is supposed to be logical. Almost all the steps are logical. But coming up with testable hypotheses is infuriatingly intuitive. Hypotheses don't arise automatically from a question. Even hypotheses that are obvious are often untestable, or not obviously testable.

Science training doesn't teach us where hypotheses come from. It assumes they're already there. We spend forever talking about how to run experiments in a valid way and to not bias our observations and to make our results repeatable, but we spend almost no time talking about why we test the things we test in the first place. That's because the answer is embarrassing: nobody knows. The best testable hypotheses come to you in the shower or in a dream or when your grad students are drunk at the bar commiserating with their friends about the tedious lab experiments you assigned because they are so straightforward they don't warrant your attention.

Hypotheses are magic. Scientists hate magic.

Engineering

But enough about science. Let's talk about applied science: engineering. Engineering is delightful because it doesn't require hypotheses. We simply take the completed science, the outcome of which has produced facts rather than guesses, and then we use our newfound knowledge to build stuff. The most logical and methodical thing in the world. Awesome.

Right?

Well, hold on.

In the first year of my engineering programme back in university, there was a class called Introduction to Engineering. Now, first of all, that's a bad sign, because it was a semester-long course and they obviously were able to fill it, so perhaps engineering isn't quite as simple as it sounds. Admittedly, much of the course involved drafting (for some reason) and lab safety training (for good reasons), but I've forgotten most of that by now. What I do remember was a simple experiment the professor had all of us do.

He handed out a bunch of paperclips to everyone in the class. Our job was to take each paperclip and bend the outer arm back and forth until it snapped, then record how many bends each one took. After doing that for about five minutes, we each drew a histogram of our own paperclips, then combined the results for the entire class's collection of paperclips into one big histogram.

If you know engineering, you know what we got: a big Gaussian distribution (bell curve). In a sample set that large, a few paperclips snapped within just one or two bends. More lasted for three. A few amazingly resilient high-performers lasted for 20 or more bends ("the long tail"). And so on.

At that point in our educations most of us had seen a Gaussian distribution at least once, in some math class where we'd been taught about standard deviations or whatever, without any real understanding. The paperclip experiment was kind of cool because it made the Gaussian distribution feel a lot more real than it did in math formulas. But still, we wondered what any of this had to do with engineering.

I will forever be haunted by the professor's answer (paraphrased, of course):

Nobody in the world knows how to build a paperclip that will never break. We could build one that bends a thousand times, or a million times, but not one that can bend forever. And nobody builds a paperclip that can bend a thousand times, because it would be more expensive than a regular paperclip and nobody needs it.

Engineering isn't about building a paperclip that will never break, it's about building a paperclip that will bend enough times to get the job done, at a reasonable price, in sufficient quantities, out of attainable materials, on schedule.

Engineering is knowing that no matter how hard you try, some fraction of your paperclips will snap after only one bend, and that's not your fault, that's how reality works, and it's your job to accept that and know exactly what fraction that is and design around it, because if you do engineering wrong, people are going to die. But what's worse, even if you do engineering right, sometimes people might die. As an engineer you are absolutely going to make tradeoffs in which you make things cheaper in exchange for a higher probability that people will die, because the only alternative is not making things at all.

In the real world, the failure rate is never zero, even if you do your job perfectly.
-- My engineering professor

And after that he shared a different anecdote:

I know some of you were top of your class in high school. Maybe you're used to getting 100% on math tests. Well, this is engineering, not math. If you graduate at the top of your engineering class, we should fail you. It means you didn't learn engineering. You wasted your time. Unless you're going to grad school, nobody in the world cares if you got an 80% or a 99%. Do as little work as you can, to learn most of what we're teaching and graduate with a passable grade and get your money's worth. That's engineering.
-- My engineering professor

That is also, it was frequently pointed out at the time, the difference between engineering and computer science.

(I'm proud to say I successfully did not graduate at the top of my engineering class.)

Software engineering

Back in the 1990s when I was learning these things, there was an ongoing vigorous debate about whether software development was or could ever be a form of engineering. Most definitions of engineering were not as edgy as my professor's; engineering definitions mostly revolved around accountability, quality, regulation, ethics. And yes, whose fault it is when people die because of what you made.

I know many people reading this weren't even alive in the 1990s, or not programming professionally, or perhaps they just don't remember because it was a long time ago. But let me tell you, things used to be very different back then! Things like automated tests were nearly nonexistent; they had barely been invented. Computer scientists still thought correctness proofs were the way to go as long as you had a Sufficiently Smart Compiler. The standard way to write commercial software was to throw stuff together, then a "quality assurance" team would try running it, and it wouldn't work, and they'd tell you so and sometimes you'd fix it (often breaking something else) and sometimes there was a deadline so you'd ship it, bugs and all, and all this was normal.

I mean, it's normal now too. But now we have automated tests. Sometimes.

Although much software development today is still not software engineering, some software development today is software engineering. Here are some signs of engineering that you can look for:

Monitoring and tracking error rates
SLOs and SLAs and uptime targets
Distributed system designs that assume and work around the fact that every component will fail
Long-time-period bug burndown charts
Continuous improvement and user pain tracking
Well-tested "unhappy paths" such as degraded operation or inter-region migrations

In short, in software engineering, we acknowledge that failures happen and we measure them, characterize them, and compensate for them. We don't aim for perfection.

Software development that isn't engineering is almost the same: failures still happen, of course. Perfection is still not achieved, of course. But only engineers call that success.

Brute force and cleverness

There are two ways to solve an engineering problem: the "brute force" way and the clever way.

Brute force is the easiest one to describe. You just do something (say graph traversal) in the obvious way, and if that's too slow, you buy more CPUs or bandwidth or whatever and parallelize it harder until the solution comes through within an acceptable amount of time. It costs more, of course, but computers are getting pretty cheap compared to programmer time, so often, the brute force approach is better in all relevant dimensions.

The best thing about brute force solutions is you don't need very fancy engineers to do it. You don't need fancy algorithms. You don't need the latest research. You just do the dumbest thing that can possibly work and you throw a lot of money and electricity at it. It's the ultimate successful engineering tradeoff.

There's only one catch: sometimes brute force simply cannot get you what you want.

We can solve any problem by introducing an extra level of indirection… except for the problem of too many levels of indirection.
— possibly David J. Wheeler
     via Butler Lampson,
     via Andrew Koenig,
     via Peter McCurdy

Here's a simple example: if I want to transfer a terabyte of data in less time, I can increase my network throughput. Throughput is an eminently brute-forceable problem. Just run more fibers and/or put fancier routers on each end. You can, in theory, with enough money, use parallelism to get as much aggregate throughput as you want, without limit. Amazing!

But the overall outcome has limits imposed by latency. Let's say I get myself 100 terabytes/sec of throughput; my single terabyte of data uses only 0.01 seconds, or 10 milliseconds, of capacity. That's pretty fast! And if I want it faster, just get me 1000 terabytes/sec and it'll only use 1 millisecond, and so on.

But that 1 millisecond is not the only thing that matters. If the other end is 100 milliseconds away at the speed of light, then the total transfer time is 101 milliseconds (and 100 milliseconds more to wait for the acknowledgement back!), and brute force will at best save you a fraction of the one millisecond, not any of the 100 milliseconds of latency.

Web developers know about this problem: even on the fastest link, eliminating round trips greatly speeds up page loads. Without this, typical page load times stop improving after about 50 Mbps because they become primarily latency-limited.

Throughput can always be added with brute force. Cutting latency always requires cleverness.

Negative latency

Speaking from a systems design point of view, we say that all real-world systems are "causal": that is, outputs are produced after inputs, never before. As a result, every component you add to a flow can only add latency, never reduce it.

In a boxes-and-arrows network diagram, it's easy to imagine adding more brute force throughput: just add more boxes and arrows, operating in parallel, and add a split/merge step at the beginning and end. Adding boxes is easy. That's brute force.

But the only way to make latency go down, causal systems tell us, is to either remove boxes or reduce the latency added by the boxes.

This is often possible, certainly. On a web page, incur fewer round trips. In a router, find ways to speed up the modulation, demodulation, and switching layers. On the Internet, find a more direct route. In a virtual reality headset, eliminate extra frames of buffering or put the compositor closer to the position sensors.

All these things are much harder than just adding more links; all of them require "better" engineering rather than more engineering; all of them have fundamental limits on how much improvement is available at all. It's hard work making causal systems faster.

Now, here's the bad news: systems designers can violate causality.

Scientists Do Not Like This.

Engineers are not so thrilled either.

You merely need to accurately predict ~~the next word~~ future requests, so that when someone later asks you to do work, it's already done.

The cache adds 3ms of latency to a system that used to take 120ms. But sometimes it lets the overall system finish in 13ms: 107ms faster than the system without the cache. Thus, adding the cache has subtracted 107ms of latency.

The result is probabilistic. If you guess wrong, the predictor box slightly increases latency (by having to look up the request and then not find it, before forwarding it on). But if you guess right, you can massively reduce latency, down to nearly nothing. And even better, the more money you throw at your predictor, the more predictions you can run pre-emptively (a technique known as "prefetching"). Eventually one of them has to be right. Right?

Well, no, not in any non-trivial cases. (A trivial case would be, say, a web service that increments every time you call it. A Sufficiently Smart Predictor could be right every time and never have to wait for the request. Some people call this Edge Computing.)

(By the way, any cache counts as a predictor, even if it doesn't prefetch. A cache predicts that you will need its answers again later so it keeps some of them around and hopes for the best, still reducing latency on average.)

Anyway, predictors violate causality, depending on your frame of reference for causality. But they can't do it reliably. They only work when they get lucky. And how often they get lucky depends on the quality of—oh no—their hypotheses about what you will need next.

You remember where hypotheses come from, right? Magic.

All caches are magic. Knowing their mechanism is not enough to predict their outcome.

(By the way, this is one of the reasons that Cache Invalidation is one of the "two hard problems in computer science.")

Insight

In my last year of high school, the student sitting next to me asked my English teacher why their essay only got a B while mine got an A+. The teacher said: the difference is... insight. Read Avery's essay. It says things I've never heard before. You want to do that. To get an A+, write something insightful.

My classmate was, naturally, nonplussed. I still remember this as some of the least actionable advice I've ever heard. Be more insightful? Sure, I'll get right on that.

(By an odd coincidence my computer at the time, my first ever Linux PC, was already named insight because I thought it sounded cool. I migrated that hostname from one home-built PC to another for several years afterward, Ship of Theseus style, so that no matter how tired and uncreative I might feel, I would always have at least one insight.)

Anyway, you guessed it. Insight is magic.

Conciseness

You will have noticed by now that this article is long. As I've gotten older, my articles seem to have gotten longer. I'm not entirely sure why that is. I'm guessing it's not especially caused by an Abundance of Insight.

I apologize for such a long letter - I didn't have time to write a short one.
— Blaise Pascal ~~Mark Twain~~

To be fair, however, I think there's at least some insight hidden away in here.

But let's say we wanted to distill this post down to something equally useful but shorter and easier to absorb. That leads us to an important question. Is shortening articles brute force, or is it clever?

I think the answer is complicated. Anyone can summarize an article; grade schoolers do it (with varying degrees of success) in their book reports. Very bad computer software has been writing auto-abstracts poorly for years. Cole's Notes charges good money for their service. ChatGPT summarizes stuff quite well for a computer, thank you.

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.
― Antoine de Saint-Exupéry

So summarization, or conciseness, or maybe we call it information compression, can be done with little to no insight at all. Perhaps to do it better requires some insight: which parts are worth highlighting, and which are worth leaving out? How do we take even one sentence and say it with fewer words? Exactly which parts of Macbeth should we include because they are going to be on the test? These are hard jobs that require some kind of skill.

Or maybe we need to draw a distinction between producing insight and recognizing insight. After all, the good parts of this essay are the insightful parts; anything you already knew can be left out. Something you didn't already know, I bet you can recognize with less effort than it took for me to write it.

I don't know much about art, but I know what I like.
— apparently nobody knows who said this

Perhaps summarization is not the ability to produce insight, but rather the ability to recognize what other people will experience as insight.

But Avery, you promised me some opinions about AI

No I didn't.

But let's tie all this together. Let's say you have a Large Language Model AI, trained on all the text in the world. We understand the mechanisms (engineering) but the outcomes are surprising and emergent (magic). Its job is to repeatedly predict the next word that would come if a person wrote it – a form of negative latency, or non-causality. From this, for reasons we don't really understand, it can say an impressive variety of unexpectedly meaningful things.

It's quite good at summarizing. I don't know how good. I wonder if there's a way to quantify that.

Summarizing well requires the ability to recognize and highlight insight. I don't know if it's good at that. I think it might be.

When you have all the text in the world memorized, that means you have access to all the insights that have ever been written. You need only recognize them, and have a good idea of what the reader knows already, and you can produce insights – things the reader has never heard before – on demand. You can get an A+ from my English teacher. Can our AI do that? I don't know. Certainly sometimes, maybe not reliably yet. Any time it does, it will feel like magic. Someone else's insight, delivered at the right moment, is as good as an entirely new insight.

But can LLM AIs produce entirely new insights? I don't know. Good question.

Can I? I don't know that either. This whole article was just a series of quotes and book recommendations.

Where do insights come from, anyway?

They don't teach that in science class.

Epilogue

I asked an AI to improve this article. With apologies to Samuel Beckett, here is ChatGPT's account, expurgated, accelerated, improved and reduced, respectively.

ChatGPT: Clean up this article to give it a coherent thesis and a clear beginning, middle, and end.

The article "Systems Design 2: What We Hope We Know" by Paul Graham is a thought-provoking exploration of the rise of AI and Large Language Models (LLMs) and their impact on the future of people, technology, and society. However, the author admits that predicting the future is difficult, even for experts, and that they know very little about AI. Instead of discussing AI, the author explores the concept of magic, magical thinking, and systems design, and how they are all interconnected.

The author begins by defining magic as something that you know is there, but you cannot explain. They argue that magic gets a bad reputation for no good reason and that many of us understand the concept of magic unconsciously. The author notes that magical thinking is a tendency to interpret something as magic instead of trying to understand it. It is a tendency to care about outcomes rather than mechanisms. The author points out that understanding emergent complexity requires a belief in magic, or what is referred to as magical thinking. They explain that emergent outcomes defy expectations, and understanding how they work requires a deeper level of abstraction that cannot always be explained through mechanisms.

The author offers a few examples of emergent complexity, such as the monotile, which is a surprisingly simple shape that, when tiled across a plane of any size, creates a never-repeating pattern. The author also recommends two books, "The Infinite Staircase" by Geoffrey Moore and "A New Kind of Science" by Stephen Wolfram, that delve into emergent complexity and offer insights into how complex systems work.

The article then explores the formation of hypotheses in scientific method and how it is the most mysterious of all the categories. The author notes that hypotheses are the foundation of scientific inquiry, and yet, science training does not teach us where hypotheses come from or how to form them. They argue that hypotheses are magic, and scientists hate magic. The author points out that the best testable hypotheses come spontaneously, and that hypotheses don't arise automatically from a question.

The article then differentiates between science and engineering and discusses how engineers accept failures and make tradeoffs. They explain that engineering is about building things that work well enough and that engineering is knowing that no matter how hard you try, some fraction of your creations will fail, and that's how reality works. The author highlights the importance of software engineering and discusses how some software development today is software engineering. They point out that engineers who can accept failures and make tradeoffs are the ones who will succeed in the future.

The author then goes into the brute force and clever way to solve engineering problems and how cutting latency always requires cleverness. They explain that brute force is the easiest way to solve engineering problems, but that sometimes it simply cannot get you what you want. The author notes that cutting latency always requires cleverness, and that engineers who can do this are the ones who will succeed in the future.

In conclusion, the author argues that understanding emergent complexity requires a belief in magic or magical thinking. They also highlight the importance of engineering and how it accepts failures and makes tradeoffs. The article concludes by pointing out that cutting latency always requires cleverness, and that engineers who can do this are the ones who will succeed in the future. The article is a fascinating exploration of the interconnectedness of magic, emergent complexity, systems design, and engineering, and how they all play a role in shaping the future of technology and society.

ChatGPT: Tell me the most important insights in this article.

Magic is something we know is there but can't explain, and it is essential to understanding any emergent system.
Magical thinking is a tendency to interpret something as magic instead of trying to understand it, and it is often used derisively.
The mechanisms used in AI systems are pretty simple, but at a large scale, combined cleverly, they create amazingly complex emergent outcomes far beyond what we put in.
Engineering is knowing that no matter how hard you try, some fraction of your paperclips will snap after only one bend, and that's not your fault, that's how reality works, and it's your job to accept that and know exactly what fraction that is and design around it, because if you do engineering wrong, people are going to die.
There are two ways to solve an engineering problem: the "brute force" way and the clever way.
The only way to make latency go down is to either remove boxes or reduce the latency added by the boxes.

ChatGPT: Say something insightful about this topic that is not already in the article, and which the audience is not expected to know already.

One thing worth noting is that the rise of AI and LLMs has brought about both excitement and concerns. On one hand, these technologies have the potential to revolutionize various industries and improve our lives in ways we can't even imagine yet. On the other hand, there are concerns about the ethical implications of AI, such as job displacement and bias in decision-making. It's important for us to continue to monitor and evaluate the development and implementation of these technologies to ensure that they benefit society as a whole, rather than just a select few. Additionally, it's worth noting that while AI and LLMs are impressive and powerful tools, they are not a panacea for all problems. It's important to recognize the limitations and potential pitfalls of these technologies, and to use them in conjunction with human intelligence and expertise to achieve the best outcomes.

ChatGPT: This article is too boring to read. Summarize it in two sentences.

The author discusses the rise of AI and Large Language Models (LLMs) and how they are often accused of "magical thinking". He gives examples of how magical thinking is essential to understanding emergent systems and how engineering is about building products that are good enough to get the job done, not perfect.

The Gift of It's Your Problem Now

Sat, 01 Jan 2022 13:32:51 +0000

Recently a security hole in a certain open source Java library resulted in a worldwide emergency kerfuffle as, say, 40% of the possibly hundreds of millions of worldwide deployments of this library needed to be updated in a hurry. (The other 60% also needed to be updated in a hurry, but won't be until they facilitate some ransomware, which is pretty normal for these situations.)

I have a 20+ year history of poking fun at Java in this space, and it pains me to stop now. But the truth is: this could have happened to anyone.

What happened was:

Someone wrote a library they thought was neat
They decided to share it with the world for free
Millions of people liked it and used it everywhere
Some contributors contributed some good ideas and, in this case, at least one bad idea
Out of a sense of stewardship, they decided to maintain backward compatibility with the bad idea
The bad idea turned out to have one or more security flaws that affected all the users
The stewards responded quickly with a fix

From this, if you believe the Internet Consensus, we can conclude that open source doesn't work, people don't get paid enough, capitalism is a sham, billionaires are built on the backs of the proletariat, your $50 Patreon donation makes a real difference, and Blorkchain Would Have Solved This.

(Miraculously the Internet Consensus is always the same both before and after these kinds of events. In engineering we call this a "non-causal system" because the outputs are produced before the inputs.)

Nevertheless, I have been dared to take my take on the issue. It, too, was the same before and after, but the difference is I didn't write it down until now, which makes this journal a causal system. You could probably write an interesting philosophical paper about observations of a non-causal system nevertheless being causal, but mercifully, I will not.

Free Software is Communism

So anyway, meandering gently toward the point, let's go back in time to the original Free Software movement. Long ago, before the average reader of this sentence was born, a person whose name is now unpopular was at a university, where they had a printer, and the printer firmware was buggy. This person firmly believed they could quickly fix the printer firmware if only they had the source code. (In the spirit of every "I could do this better in a weekend" story, I'm not sure whether we ever confirmed if this was true. In any case printer firmware is still buggy.)

As a result, they started a nonprofit organization to rewrite all of Unix, which the printer did not run and which therefore would not solve any of the original problem, but was a pretty cool project nonetheless and was much more fun than the original problem, and the rest was history.

This story archetype is the Hero's Journey that inspires all software development:

I have a problem
I do not know how to solve that problem
But I strongly believe, without evidence, that I can solve a generalized version of that problem if I form a large enough team and work at it for 35 years(*)
We are now accepting donations

(*) initial estimate is often less than 35 years

Now, you have probably heard all this before, and if you're a software developer you have probably lived it. This part is not really in question. The burning question for us today, as we enjoy the (hopefully) peak of late-stage capitalism, is: ...but where will the donations come from?

Authoritarianism

Before we get back onto communism, let me draw an important distinction. Most communist governments in history ended up being authoritarian systems, which is to say, top-down control. Ironically, the people at the top seem to have more power than the people at the bottom, which at first seems like the antithesis of communism. This is not the place to claim an understanding of why that always seems to happen. But one has to acknowledge a pattern when one sees it.

On the other hand, it's easy to find examples of authoritarianism outside communism. Our world is filled with top-down control systems. Many corporations are in many ways, top-down controlled. The US system of government is increasingly top-down controlled (ie. authoritarian), despite the many safety measures introduced early to try to prevent that.

When politicians rail against communism it is because they don't want you to notice the ever-growing non-communist authoritarianism.

Authoritarianism is self-reinforcing. Once some people or groups start having more power, they tend to use that power to adjust or capture the rules of the system so they can accumulate more power, and so on. Sometimes this is peacefully reversible, and sometimes it eventually leads to uprisings and revolutions.

People like to write about facism and communism as if they are opposite ends of some spectrum, but that's not really true in the most important sense. Fascism blatantly, and communism accidentally but consistently, leads to authoritarianism. And authoritarianism is the problem.

Authoritarianism is about taking things from me. Communism, in its noncorporeal theoretical form, is about giving things away.

I read a book once which argued that the problem with modern political discourse is it pits the "I don't want things taken from me" (liberty!) people against the "XYZ is a human right" (entitlement!) people. And that a better way to frame the cultural argument is "XYZ is my responsibility to society."

As a simple example, "Internet access is a human right," is just a sneaky way of saying "someone should give people free Internet." Who is someone? It's left unspecified, which is skipping over the entire mechanism by which we deliver the Internet. It's much more revealing to write, "To live in a healthy society, it's our responsibility to make sure every person has Internet access." Suddenly, oh, crap. The someone is me!

Healthy society is created through constant effort, by all of us, as a gift to our fellow members. It's not extracted from us as a mandatory payment to our overlords who will do all the work.

If there's one thing we know for sure about overlords, it's that they never do all the work.

Free software is a gift.

I would like to inquire about the return policy

Here's the thing about gifts: the sender chooses them, not the recipient. We can have norms around what gifts are appropriate, and agreements to not over-spend, and wishlists, and so on. But I won't always get the exact gift I want. Sometimes I didn't even want a gift. Sometimes the gift interprets JNDI strings in my log messages and executes random code from my LDAP server. This is the nature of gifts.

On the other hand, the best gifts are the things I never would have bought for myself, because they seemed too expensive or I didn't even realize I would like them or they were too much work to obtain, or because someone hand-made them just for me. These feel like luxuries of the sort capitalism cannot produce, because deciding, going out, and buying something for myself isn't luxury, it's everyday. It's lonely. It's a negotiation. It's limited by my own lack of creativity.

The best part of free software is it sometimes produces stuff you never would have been willing to pay to develop (Linux), and sometimes at quality levels too high to be rational for the market to provide (sqlite).

The worst part of free software is you get what you get, and the developers don't have to listen to you. (And as a developer, the gift recipients aren't always so grateful either.)

Paying for gifts

...does not work.

You don't say to someone, "here's $100, maybe this time get me a gift worth $100 more than you'd regularly spend." It's kind of insulting. It still probably won't get you exactly the thing you wanted. Actually, the other person might just pocket the $100 and run off with it.

We already have a way for you to spend $100 to get the thing you want. It's a market. A market works fine for that. It's not very inspiring, but most of the time it's quite efficient. Even gift-givers will often buy things on the same market, but with a different selection criteria, thus adding value of their own.

When you try to pay for gifts, it turns the whole gift process into a transaction. It stops being a gift. It becomes an inefficient, misdesigned, awkward market.

There's research showing that, for example, financial compensation in a job is more likely a demotivator than a motivator (ie. if you pay me too little, I'll work less hard or quit, but if you double my pay, it won't double my output). If you tie cash compensation to specific metrics, people will game the metrics and usually do an overall worse job. If you pay someone for doing you a favour, they are less likely to repeat the favour. Gifts are inherently socially and emotionally meaningful. Ruin the giftiness, and you ruin the intangible rewards.

So it is with free software. You literally cannot pay for it. If you do, it becomes something else.

This is why we have things like the Linux Foundation, where the idea is you can give a gift because you appreciate and want to support Linux (and ideally you are a rich megacorporation so your gift is very big), but it dilutes the influence of that money through an organization that supposedly will not try to influence the gift of Linux that was already happening. You end up with multiple gift flows in different directions. Money goes here, code goes there. They are interdependent - maybe if one flow slows down the other flow will also slow down - but not directly tied. It's a delicate balance. People who keep receiving Christmas gifts but never give any might eventually stop receiving them. But might not.

Anyway, gifts will not get you 24-hour guaranteed response times to security incidents.

Gifts won't get you guaranteed high quality code reviews.

Gifts will not, for heaven's sake, prevent developers from implementing bad ideas occasionally that turn into security holes. Nothing will. Have you met developers?

Open source

I've avoided the term "open source" so far because it means something different from the original idea of Free Software.

Open source was, as I understand it, coined to explain what happened when Netscape originally opened their Mozilla source code, back at the end of the 1990s. That was not a gift. That was a transaction. Or at least, it was intended to be.

The promise of open source was:

You, the company, can still mostly control your project
Customers will still pay you to add new features
Actually customers might pay other people to add new features, but you can still capitalize on it because you get their code too
Linux distributions only package open source code so you'll onboard more customers more easily this way
You can distance yourself from this anti-capitalist gift-giving philosophical stuff that makes investors nervous
Plus a bunch of people will look at the code and find bugs for you for free!

Maybe this sounds cynical, but capitalists are cynical, and you know what? It worked! Okay, not for Netscape Corporation (sorry), but for a lot of other people since then.

It also failed a lot of people. Many developers and companies have been disappointed to learn that just uploading your code to github doesn't make a community of developers appear. (It does make it more likely that AWS will fork your product and make more money from it than you do.) Code reviews are famously rare even in security-critical projects. Supply chain issues are rampant.

In fact, we've now gotten to the point where some people hesitate to give away their source code, mainly because of this confusion of gifts and customers. If I spend some spare time hacking something together on a weekend and give it away, that's a gift. If you yell at me for making it, that makes giving less fun, and I will spend fewer weekends making gifts.

Whereas when a company has a product and open sources it and you complain, that's customers giving valuable feedback and it's worth money to learn from them and service them, because you eventually earn money in exchange (through whatever business model they've established). No gift necessary.

Call it cynical or call it a win/win relationship. But it's not a gift.

The startup ecosystem

Since the creation of the open source designation 20+ years ago, software startups have taken off more than ever. I attribute this to a combination of factors:

Cloud computing has made it vastly cheaper to get started
Incubators like YCombinator have industrialized the process of assembling and running a small software company
Megacorps have become exponentially richer but no more creative, so they need to acquire or acqui-hire those startups faster and faster in order to grow.

Although a lot of startups open source their code, and they all depend heavily on open source ecosystems, the startup world's motivations are amazingly different from the free software and open source worlds.

Gifts exist in the startup world. They are things like "we were both in YCombinator so I will intro you to this investor I like" or "I got extremely rich so let me invest in your startup and incidentally I get a lottery ticket for becoming even more rich." These absolutely are still gifts. They each strengthen social ties. The startup world is a society, and the society is built up from these gifts. It's a society that largely ignores the trials and tribulations of anyone who isn't a rich software engineer insider, but history has hosted many past societies of that sort and it takes a long time to build and deploy enough guillotines, and anyway they are having fun and producing a lot and surely that counts for something.

If free software gifts are communism and open source is cynically capitalist exploitation, then startups may be, weirdly, the democratization of capitalism.

Hear me out. Big companies don't care what you think; you can't pay them enough to care. Gift givers care only a little what you think; if they gave you what you wanted, it wouldn't be a gift. But startups, well, there are a lot of them and their mantras are "do things that don't scale" and "focus on the customer" and "build rapid feedback loops." What that spells for you is a whole bunch of people who want to give you what you want, in exchange for money, and who are excited to amortize the costs of that over all the other customers who want the same thing.

It's kind of exciting, conceptually, and more self-optimizing than untuned gift giving, and so it's not super surprising to me that it has started to eclipse the earlier concepts of free software and open source. More and more "open" projects are backed by small companies, who have financial incentives to make their users happy because some of the users turn into paying customers. They'll even provide the uptime SLAs and security fix turnaround guarantees you wanted so much. Our company, Tailscale, is unabashedly one of those. Nothing to be ashamed of there. The system works.

What doesn't work is assuming those startup mechanics apply to everyone out there who gives you a software gift. Not every project on github is the same.

Not everyone has the same motivations.

Giving them money won't change their motivations.

Trying to pay them or regulate them taints the gift.

If you wanted to pay someone to fix some software, you didn't want a gift. You wanted a company.

But if there is no company and someone gave you something anyway? Say thanks.

Epilogue

This isn't where evolution stops. There's a lot more to say about how SaaS taints the unwritten agreement of open source (because you don't have to give back your changes to the code), and how startups tend to go bankrupt and their tech dies with them, and how the best developers are not good at starting companies (no matter how much easier it has become), and how acquiring a startup usually destroys all the stuff they innovated, and how open source is often used as a way to exfiltrate past those kinds of disasters, and how simultaneously, whole promising branches of the "gift economy" structure have never been explored. But that's enough for today. Maybe another time.

100 years of whatever this will be

Thu, 02 Dec 2021 12:38:46 +0000

What if all these weird tech trends actually add up to something?

Last time, we explored why various bits of trendy technology are, in my opinion, simply never going to be able to achieve their goals. But we ended on a hopeful(?) note: maybe that doesn't matter. Maybe the fact that people really, really, really want it, is enough.

Since writing that, I've been thinking about it more.

I think we are all gradually becoming more aware of patterns, of major things wrong with our society. They echo some patterns we've been seeing for decades now. The patterns go far beyond tech, extending into economics and politics and culture. There's a growing feeling of malaise many of us feel:

Rich, powerful, greedy people and corporations just get richer, more powerful, and more greedy.
Everyone seems to increasingly be in it for themselves, not for society.
Or, people who are in it for society tend to lose or to get screwed until they give up.
Artists really don't get enough of a reward for all the benefit they provide.
Big banks and big governments really do nonspecifically just suck a lot.
The gap between the haves and have-nots keeps widening.
You can't hope to run an Internet service unless you pay out a fraction to one of the Big Cloud Providers, just like you couldn't run software without paying IBM and then Microsoft, back in those days.
Bloody egress fees, man. What a racket.
Your phone can run mapreduce jobs 10x-100x faster than your timeshared cloud instance that costs more. Plus it has a GPU.
One SSD in a Macbook is ~1000x faster than the default disk in an EC2 instance.
Software stacks, governments, and financial systems: they all keep getting more and more bloated and complex while somehow delivering less per dollar, gigahertz, gigabyte, or watt.
Computers are so hard to run now, that we are supposed to give up and pay a subscription to someone - well, actually to every software microvendor - to do it for us.
We even pay 30% margins to App Stores mainly so they can not let us download apps that are "too dangerous."
IT security has become literally impossible: if you install all the patches, you get SolarWinds-style supply chain malware delivered to you automatically. If you don't install the patches, well, that's worse. Either way, enjoy your ransomware.
Software intercompatibility is trending toward zero. Text chat apps are literally the easiest thing in the world to imagine making compatible - they just send very short strings, very rarely, to very small networks of people! But I use at least 7 separate ones because every vendor wants their own stupid castle and won't share. Don't even get me started about books or video.
The most reasonable daycare and public transit in the Bay Area is available only with your Big Tech Employee ID card.
Everything about modern business is designed to funnel money, faster and faster, to a few people who have demonstrated they can be productive. This totally works, up to a point. But we've now reached the extreme corner cases of capitalism. Winning money is surely a motivator, but that motivation goes down the more you have. Eventually it simply stops mattering at all. Capitalism has become a "success disaster."

Writing all this down, you know what? I'm kind of mad about it too. Not so mad that I'll go chasing obviously-ill-fated scurrilous rainbow financial instruments. But there's something here that needs solving. If I'm not solving it, or part of it, or at least trying, then I'm... wasting my time. Who cares about money? This is a systemic train wreck, well underway.

We have, in Western society, managed to simultaneously botch the dreams of democracy, capitalism, social coherence, and techno-utopianism, all at once. It's embarrassing actually. I am embarrassed. You should be embarrassed.

"Decentralization"

I'm a networking person and a systems person, so please forgive me if I talk about all this through my favourite lens. Societies, governments, economies, social networks, and scalable computing all have something in common: they are all distributed systems.

And.

And everyone.

Everyone seems to have an increasingly horrifically misguided idea of how distributed systems work.

There is of course the most obvious horrifically misguided recently-popular "decentralized" system, whose name shall not be spoken in this essay. Instead let's back up to something older and better understood: markets. The fundamental mechanism of the capitalist model.

Markets are great! They work! Centrally planning a whole society clearly does not work (demonstrated, bloodily, several times). Centrally planning corporations seems to work, up to a certain size. Connecting those corporations together using markets is the most efficient option we've found so far.

But there's a catch. People like to use the term free market to describe the optimal market system, but that's pretty lousy terminology. The truth is, functioning markets are not "free" at all. They are regulated. Unregulated markets rapidly devolve into monopolies, oligopolies, monopsonies, and, if things get really bad, libertarianism. Once you arrive there, every thread ends up with people posting about "a monopoly on the use of force" and "paying taxes at gunpoint" and "I'll run my own fire department" and things that "end at the tip of the other person's nose," and all useful discourse terminates forevermore.

The job of market regulation - fundamentally a restriction on your freedom - is to prevent all that bad stuff. Markets work well as long as they're in, as we call it in engineering, the "continuous control region," that is, the part far away from any weird outliers. You need no participant in the market to have too much power. You need downside protection (bankruptcy, social safety net, insurance). You need fair enforcement of contracts (which is different from literal enforcement of contracts).

And yet: markets are distributed systems.

Even though there are, in fact, very strict regulators and regulations, I can still enter into a contract with you without ever telling anyone. I can buy something from you, in cash, and nobody needs to know. (Tax authorities merely want to know, and anyway, notifying them is asynchronous and lossy.) Prices are set through peer-to-peer negotiation and supply and demand, almost automatically, through what some call an "invisible hand." It's really neat.

As long as we're in the continuous control region.

As long as the regulators are doing their job.

Here's what everyone peddling the new trendy systems is so desperately trying to forget, that makes all of them absurdly expensive and destined to fail, even if the things we want from them are beautiful and desirable and well worth working on. Here is the very bad news:

Regulation is a centralized function.

The job of regulation is to stop distributed systems from going awry.

Because distributed systems always go awry.

If you design a distributed control system to stop a distributed system from going awry, it might even work. It'll be unnecessarily expensive and complex, but it might work... until the control system itself, inevitably, goes awry.

I find myself linking to this article way too much lately, but here it is again: The Tyranny of Structurelessness by Jo Freeman. You should read it. The summary is that in any system, if you don't have an explicit hierarchy, then you have an implicit one.

Despite my ongoing best efforts, I have never seen any exception to this rule.

Even the fanciest pantsed distributed databases, with all the Rafts and Paxoses and red/greens and active/passives and Byzantine generals and dining philosophers and CAP theorems, are subject to this. You can do a bunch of math to absolutely prove beyond a shadow of a doubt that your database is completely distributed and has no single points of failure. There are papers that do this. You can do it too. Go ahead. I'll wait.

Okay, great. Now skip paying your AWS bill for a few months.

Whoops, there's a hierarchy after all!

You can stay in denial, or you can get serious.

Western society, economics, capitalism, finance, government, the tech sector, the cloud. They are all distributed systems already. They are all in severe distress. Things are going very bad very quickly. It will get worse. Major rework is needed. We all feel it.

We are not doing the rework.

We are chasing rainbows.

We don't need deregulation. We need better designed regulation.

The major rework we need isn't some math theory, some kind of Paxos for Capitalism, or Paxos for Government. The sad, boring fact is that no fundamental advances in math or computer science are needed to solve these problems.

All we need is to build distributed systems that work. That means decentralized bulk activity, hierarchical regulation.

As a society, we are so much richer, so much luckier, than we have ever been.

It's all so much easier, and harder, than they've been telling you.

Let's build what we already know is right.

10 years of... whatever this has been

Wed, 17 Nov 2021 22:38:25 +0000

I guess I know something about train wrecks.

One night when I was 10 years old, me and my mom were driving home. We came to a train crossing outside of town. There was another car stopped right on the tracks, stalled. A lady was inside, trying to get her car to start. It didn’t.

Train crossings are bumpy, cars were worse then, it was a long time ago, I don’t know, I don’t remember clearly. Anyway, it was cold out and most people didn’t have cell phones yet, so when the car wouldn’t start and it was too heavy to push, there wasn’t much to be done. My mom convinced her to get the heck away from the tracks and come sit in our car to warm up and make a plan. We heard the whistle of an arriving train. And I made what I now consider one of the (several) biggest mistakes of my life: I closed my eyes.

It was only a few seconds later when I realized OH MY GOD WHAT WAS I THINKING I COULD HAVE WATCHED A TRAIN DESTROY A CAR RIGHT IN FRONT OF ME!!! But I wasn’t brave enough, I panicked, I closed my eyes, and you know what? The train wreck happened anyway. I just didn’t get to see it.

It was in the local newspaper a couple days later. The newspaper said the car ran into the train, and not the other way around. I was boggled. I learned later that this was my first, surprisingly on-the-nose, encounter with the Gell-Mann Amnesia Effect. (To this day, I still believe some of the things I read. I have no idea why.)

What’s the point of this story? That the train crash still happens, whether or not you’re watching. And everything you've read about it is probably wrong. And I’m glad my mom helped that lady get out of the way.

Anyway that’s why I don't mute blockchain-related keywords on twitter.

The blockchain train crash

Ten years(!) have passed since I wrote Why bitcoin will fail. And yet, here we are, still talking about bitcoin. Did it fail?

According to the many cryptobots who pester me, apparently not. They still gleefully repost my old article periodically, pointing out that at the time, bitcoins were worth maybe three dollars, and now they're worth infinity dollars, and wow, that apenperson sure must feel dumb for not HODLING BIGTIME back when they had the chance, lol.

Do I feel dumb? Well, hmm. It’s complicated. Everything I predicted seems to have come true. If your definition of “failure” is “not achieving any of the stated goals,” then I guess bitcoin has profoundly... not succeeded. But that doesn’t really tell the whole story, does it? A proper failure would be in the past tense by now.

What I do know is I’ve learned some stuff in 10 years.

What I got right

But first, let’s review the claims I made in the original article:

If you like bitcoin, you must think the gold standard was a good idea.
To create gold currency, you do pointless busywork (“mining”). Gold is a stupid inconvenient currency that’s worse than paper.
Printing and destroying money is a key economic tool.

Yup. Over the years we’ve seen an ongoing, embarrassing overlap between “goldbug” zealots and bitcoin zealots. The busywork mining has gotten absurdly more expensive than it was in 2011, and somehow is now a significant fraction of worldwide energy usage (what. the. heck), and various blockchains’ environmental impact is now the most common argument people use against them.

Beyond my imagination, bitcoin has achieved the unlikely goal of being even less convenient than gold for actually buying things (the job of a currency). The exchange rate of bitcoin is almost completely a random walk, impossible for anyone to manage (unlike a regular currency), and much worse than even gold.

Even if it was a good idea, governments would squash it.
The only reason they haven’t is it’s too small to matter.

Yes and yes.

Congratulations, we’ve now seen the bitcoin movement get big enough to matter! There’s a corresponding increase in regulation, from SEC investigations, to outright banning in some countries, to the IRS wanting to tax you on it, to anti-terrorist financing and KYC rules. Each new regulation removes yet another supposed advantage of using something other than cash.

Also, it’s now obvious that use of bitcoin (and related blockchains) for payments is almost entirely scams and illegal stuff. This agrees with my prediction, but in a way I didn’t expect. It turns out to be maybe tautological. Like the old American saying, “If you outlaw guns, then only outlaws will have guns,” you could argue that we have now regulated bitcoin so much that only criminals can productively use bitcoin.

But it's grown enough to now be producing the largest (and ongoing!) ransomware outbreak the world has ever seen, so, you win, I guess.

The whole technological basis is flawed.
The weak link is not SHA256, it’s the rest of the cryptosystem.

Yes, in multitudes.

We’ve seen forked chains, theft, mysteriously reversed transactions, 51% attacks.

It turned out bitcoin is irreconcilably privacy-destroying. (Law enforcement teams say thanks!) This was originally billed as a feature until some drug dealers got caught. The feature, or maybe bug, can’t be fixed without changing the system, which can’t be done without getting everyone to upgrade.

But ha, it's a decentralized system. Since nobody could figure out how to get everyone to upgrade a decentralized system all at once, it was more profitable to instead spin up zillions of new blockchains, each with its own systemic flaws, but all sharing the one big systemic flaw: it’s an ownerless distributed cryptosystem, so when each fatal flaw is inevitably revealed, nobody can fix it.

On top of the technical problems, there were social problems. Jo Freeman's Tyranny of Structurelessness showed up here, as it does whenever you try to pretend you have no social control hierarchy. We learned that the people who write the code, and the people who have the biggest mining rigs, and the people who operate exchanges, and the people who go on expensive and very shady cruises to hobnob with the cabal, and something about North Korea, and basically everyone who is not you, all have disproportionate control over what happens with this “decentralized” “currency.” And this is equally true in all the other “decentralized” chains invented to either scam you or solve technical problems in the original, or both.

For heaven's sake, people, it's software. You built a system, or series of systems, that will fail in completely predictable ways, forever, if you didn't get the software perfectly right the first time. What did you think would happen.

It doesn’t work offline.
Paper money does.

Still true. On the other hand, the global expansion of cellular data availability has been relentless and perhaps this never did matter.

What I got wrong

Okay, well. The title.

“Why bitcoin will fail” wasn’t right. It would have been better to call it “Why bitcoin should fail,” because it really should have! But it didn’t, at least not yet, at least in the minds of its ever-growing user base. I feel like this is important.

A few years ago I learned the investor variant of Sturgeon’s Law. Here’s what a VC told me: 90% of new things will fail. Therefore you can just predict every new thing will fail, and 90% of the time you’ll be right. That’s a pretty good way to feel good about yourself, but it’s not very useful. Anybody can do that. Instead, can you pick the 10% that will succeed?

Even though I accurately predicted a bunch of things about bitcoin that wouldn’t work, I didn’t predict all the other things about bitcoin that wouldn't work. Maybe that seems like splitting hairs, but it isn’t. If you miss the reasons something won’t work, then you might just as easily miss the reasons why it will work. It suggests that you don’t know what you’re talking about.

Here are some reasons I missed for why bitcoin (and blockchains generally) didn’t and still don't work, for anything productive:

Scams. Lots and lots of scams. Blockchains became the center of gravity of almost all scams on the Internet. I don’t know what kind of achievement that is, exactly, but it’s sure something.
Citizens moving money out of authoritarian regimes. This is, by definition, illegal, but is it a net benefit to society? I don’t know. Maybe sometimes.
Other kinds of organized crime and trafficking. I don’t know what fraction of money laundering nowadays goes through blockchains. Maybe it’s still a small percentage. But it seems to be a growing percentage.
More and more blockchains. There are so many of them now (see “scams”, above), claiming to do all sorts of things. None of them do. But somehow even bitcoin is still alive, even though a whole ecosystem of derivative junk has sprouted trying to compete with it.
Corrupt or collapsed exchanges. I predicted technical problems, but most of the failures we’ve seen have been simple, old fashioned grifters and incompetents. Indeed, the failures of this new financial system are just like the historical failures of old financial systems, albeit with faster iterations. Some people are excited about how much faster we can make more expensive mistakes now. I'm not so sure.
Gambling and speculation. I wrote the whole article expecting bitcoin to fail at being a currency, but that charade ended almost immediately. What exists now is an expensive, power-hungry, distributed, online gambling system. The house still always wins, but it’s not totally clear who the house is, which is how the house likes it. Gambling has always been fundamentally a drain on society (a “tax on the uneducated,” someone once told me), but it’s always very popular anyway. Bitcoin is casino chips. Casino chips aren’t currency, but they don't “fail” either.

Despite all that - and I didn't even need to exaggerate! - bitcoin has still not failed, if failure means it’s gone. It's very much still around.

That’s because I forgot one essential reason bitcoin has survived:

Because people really, really, really want it to.

If there’s one lesson I learned over and over in the last ten years, that’s it. Projects don’t survive merely because they are good ideas; many good ideas get cancelled or outcompeted. Ask anyone who works at an overfunded tech company.

Similarly, movements don’t die just because they are, in every conceivable way, stupid. Projects live or die because of the energy people do or do not continue to put into them.

…

Here's a metaphor. Blockchains today are like… XML in the early 2000s. A long-burning overcomplicated trash fire that a bunch of large, cash-rich, unscrupulous, consultant-filled mega-organizations foisted on us for years and years, that we all now pretend we weren’t dumb enough to fall for. Remember SOAP? Remember when we tried to make HTML4 into XHTML? Ew, no.

The thing is, a ton of major tech infrastructure spending was somehow justified in the name of XML. A lot of computer systems, especially in the financial and manufacturing industries, got a long-needed overhaul. Fixed-width COBOL databases couldn't do XML, so bam, gotta replace some fixed-width COBOL databases (or at least slap a new API translator in front). The XML part sucked and still sucks and we’ll be dealing with it for decades.

But is that really so bad, in the name of progress?

Epilogue

It's been ten years, and it all went pretty badly, so let me make a new prediction.

A lot of stuff will get redesigned in the name of blockchains. Like XML, the blockchains will always make it worse, but if carefully managed, maybe not too much worse. Something good will eventually come out of it, by pure random chance, because of all those massive rewrites. Blockchains will take credit for it, like XML took credit for it. And then we'll finally move on to the next thing.

Nowadays if someone picks XML for a new design, we look at them like they’re from another planet. So, too, it will be with decentralized consensus blockchains, someday.

So, too, it was for the gold standard for international trade.

But it took more than 50 years.

SimSWE 4: Wants, needs, and chasm-crossing

Wed, 17 Nov 2021 22:38:25 +0000

Let's talk about bug/feature tradeoffs.

Anyone who knows me has probably already heard me rant about Crossing the Chasm, my most favourite business book of all time. I love its simple explanation of market segmentation and why the life cycle of a tech startup so often goes the way it does. Reading that book is what taught me that business success is not just a result of luck or hard work. Strategy matters too.

As our company prepares for our chasm-crossing phase, I've been thinking about the math behind why chasm-crossing works and why our metrics plots (doesn't every startup do their key business metrics in R?) look the way they do, and I realized that chasm-crossing strategy must have a simple mathematical basis behind it. I bet I could math this.

And so, our simulated software engineering (SWE) team is back!

In previous episodes of SimSWE, we learned it's objectively good to be short-term decisive even if you're wrong and to avoid multitasking. Later, I expanded on all that, plus more, in my epic treatise on software scheduling. And then, as a bonus, our simulated SWEs went on to buy homes and distort prices in the California housing market.

This time, I want to explore the chasm-crossing process and, while we're here, answer the unanswerable question: what's a bug and what's a feature?

Nobody can agree on what they mean. When does "lack of a feature" become a bug? When a key customer demands it? When the project manager declares a code freeze but you still want to merge your almost-finished pull request? When it's Really Really Important that you launch at a particular conference?

The answer is, users don't care what you call it. Let's reformulate the question.

We need to make a distinction between needs and wants.

Back when I lived in New York, I took some fiction writing classes. One thing I learned is there is a specific recipe for "interesting" characters in a story, as follows: understand how characters' needs differ from their wants. It's rare that the two are the same. And that way lies drama.

So it is with customers. I want a browser that doesn't suck all my RAM and drain my battery. But I need a browser that works on every website and minimizes malware infections, so I use Chrome.

I want a scripting language that isn't filled with decades-old quoting idiosyncracies, but I need a scripting language that works everywhere, so I mostly use POSIX sh.

Some people call needs "table stakes." You must be this tall to ride the roller coaster, no exceptions. If you are not this tall, you cannot ride the roller coaster. Whether you want to ride the roller coaster is an orthogonal question related to your personal preferences.

Needs are AND. Wants are OR. A product must satisfy all your needs. It can get away with satisfying only one want, if you want it badly enough.

Needs are roadblocks to your product's adoption. (I previously wrote about roadblock analysis.)

A want is a reason to use some new software. A need is a reason you can't.

About 20 years ago(!), Joel on Software wrote about the 80/20 myth:

80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies. Unfortunately, it’s never the same 20%.

– Joel Spolsky

And yet, if you're starting a new project, you can't exactly do 100% of the features people want, all at once. What can you do instead?

Market segments, use cases, and needs

The best (and thankfully becoming common) advice to startups nowadays is to really nail just one use case at first. Pick a want, find people who want it, figure out what those people have in common, call it a market segment, solve the needs of the people in that segment, repeat.

This is all harder than it sounds, mostly because of your own human psychology. But it all lends itself well to rapid iteration, which is why our earlier SimSWE tips to be decisive and to avoid multitasking are right.

Getting back to Crossing the Chasm, the most essential advice in the book - and the hardest to follow - is to focus on your chosen market segment and ignore all requests from outside that segment. Pick one want. Fulfill all the needs.

Let's make a simulation to show what happens if you do or don't. And if we're lucky, the simulation will give us some insight into why that's such good advice.

Simulating wants and needs

The plot below simulates a market that with 10,000 potential users, 10 potential wants, and 15 potential needs. Each user has a varying number of wants (averaging 3 each) and needs (averaging 5 each).

For a user to be interested in our product, it's sufficient for our product to fulfill any of their wants. On the other hand, for a user to actually adopt the product, they need to be interested, and we need to fulfill all their needs.

Side note: we can think of a "Minimum Viable Product" (MVP) as a product that fulfills one want, but none of the needs. There will be some tiny number of users who have no special needs and could actually use it. But a much larger group might want to use it. The MVP gives you a context for discussion with that larger group.

Before we get to all that deliberate activity, though, here's an example run of the simulator, with random-ish sequencing of wants and needs.

The dim dotted line is the Total Addressable Market (TAM). Every time you implement a want, the TAM goes up. Fun! This is what venture capitalist dreams are made of. All the users in the TAM are "interested" in your product, even if they aren't able to use it yet.

The dashed line is the "unblocked" users. These are users who are in the TAM and whose needs you've entirely filled. They legitimately could buy your product and be happy with it. Assuming they hear about you, go through the trial and sales process, etc. This is the maximum number of users you could have with your current product.

Finally, the red line is the number of users you actually have at any given time. It takes into effect marketing, word-of-mouth, and adoption delays.

Commentary

I'm already excited about this simulation because it shows how adoption curves "really look" in real life. In particular, you can see the telltale signs of a "real" adoption curve:

Exponentially growing uptake at first, which slows down as you saturate the market (ie. an "S-curve" shape).
When you look more closely, the big S-curve is made up of a bunch of smaller S-curves. Each time we fulfill a need, some group of users becomes unblocked, and we can move toward saturating an ever-bigger market.

Observe also that the jumps in the dotted line (fulfilled wants) are big at first, and smaller each time. That's because each user has an average of three wants, and you only need to satisfy one of them. Because of overlapping wants, the second want is split between new users and users you already have. Each successive want has a greater and greater overlap with your already-interested users, and thus less and less effect.

(Alas, this gives a good mathematical rationale for why "mature" products stop improving. Yes, there are all sorts of additional things your audience might want. But if adding them doesn't increase your TAM, it's strategically questionable whether you should bother. Bring on the continuous improvement debate.)

(On the other hand, this simulation is somewhat unrealistic because of the pre-defined market size of only 10,000 participants. If, instead of fulfilling more wants for your existing market segment, you add a new market segment, those new wants might have a bigger impact and your "big" S-curve might get a newer, bigger S-curve added to it. This is small consolation to your existing users who would like some more stuff added that they care about, though.)

In contrast, the jumps in the dashed line (needs fulfilled) start small and get bigger. This also makes sense intuitively: since users can't adopt the product until all their needs are met, and the typical user has 5 needs, certainly the first 4 needs are going to attract only a small group of less-discerning people. Even the first 5 needs will only capture the group of users with exactly those 5 needs or fewer. But by the time you're reaching the end of the to-do list, every new need is unlocking a big group of almost-satisfied users.

(This part is coool because it explains what startups so often experience: at first, fulfilling needs for your target market creates a small jump in absolute user count. But through this "AND" effect, each subsequent need you fulfill can create a bigger and bigger jump. Even if the new features seem fairly small or relatively easy compared to your early work!)

Comparing strategies

Of course, that was a single simulation based on a bunch of made-up arbitrary assumptions and some numerical constants selected mainly on the basis of how pretty the graph would look.

The good part comes when we compare multiple product management strategies:

Let's continue to assume a fixed market segment of 10,000 users, each of whom have an assortment of wants and needs.

The four plots above correspond to four ways of prioritizing those wants and needs:

Features First: the "maximum hype" approach. Implement all 10 wants before solving any needs at all. This maximizes TAM as early as possible. Some early-stage investors get starry-eyed when they see that, but unfortunately you don't get a lot of live users because although people are excited, they can't actually use the product. This is also what you get if you don't, as Steve Blank would say, "get out of the building" and talk to real customers.
Alternating: switch between implementing wants and needs, semi-randomly. It turns out this grows your userbase considerably faster than the first option, for the same reason that you'll do okay at rock-paper-scissors by using a random number generator instead of always choosing rock. The main thing here is shipping those randomly-ordered milestones as fast as you can. As SimSWE 1 and 2 emphasized, if you do that, you can get away with not being super great at prioritization.
Needs First: just implement exactly one want, then fix all the needs before moving on to other wants. This is a purified Crossing the Chasm model. You can see that the TAM doesn't start increasing until pretty late, because new use cases are on hold. But we get precious real users earlier, which spread word-of-mouth sooner and lead to faster exponential adoption later.
Perfectionism: the naive opposite of features-first; a variant of needs-first where we don't even solve a single want before we start trying to address needs. Since the product does nothing useful, but very reliably, nobody wants to buy it at first (~zero TAM). When we finally start launching use cases, we can add them pretty quickly, but actual growth lags behind, at first, because we were late in getting our exponential growth curve started. think of this as the "we got SOC2 compliance before we had any customers" strategy.

In these plots, the important things to look for are getting more users sooner (money in the bank!) and total area under the curve (aggregate value delivered). Users you get earlier are users who give you money and spread word-of-mouth over a longer time, so they are much more valuable than users you add later.

In this version of the plot, it looks like #3 is winning, #4 is not too bad, and even #2 might be kind of okay. In the end, is there really much difference?

Let's zoom in!

More needs fulfilled, more momentum

This plot zooms the y axis to the first 1000 customers, leaving the x axis unchanged from before. Now the differences are more dramatic.

Here you can see that needs-first starts attracting at least a noticeable number of live customers at time 150 or so. The others take much longer to get rolling.

This feels intuitively right: in the early days of a startup, you build an MVP, nobody uses it, you find a few willing ~~suckers~~ early adopters and listen to their feedback, fix the first couple of roadblocks, and now you have a few happy niche users. If all goes well, those users will refer you to more users with a few more roadblocks, and so on. At that stage, it's way too early to worry about expanding your TAM.

What I find exciting - but not all that surprising, having now immersed ourselves in the math - is that the needs-first approach turns out to not be a compromise. The word-of-mouth advantage from having zero-roadblock, excited, active users early on means the slow part of the exponential growth can get started early, which over time makes all the other effects look small. And each successive fulfilled need unlocks an ever-greater number of users.

In contrast, you can see how increasing the TAM early on has not much benefit. It might get your investors excited, but if you don't have live users, there is nobody to spread word-of-mouth yet. Surprisingly little is lost by just focusing on one small want, clearing out roadblocks for people who want that, and worrying about the rest later.

Systems design explains the world: volume 1

Tue, 29 Dec 2020 03:45:00 +0000

"Systems design" is a branch of study that tries to find universal architectural patterns that are valid across disciplines.

You might think that's not a possibility. Back in university, students used to tease the Systems Design Engineers, calling it "boxes and arrows" engineering. Not real engineering, you see, since it didn't touch anything tangible, like buildings, motors, hydrochloric acid, or, uh, electrons.

I don't think the Systems Design people took this criticism too seriously since everyone also knew that programme had the toughest admittance criteria in the whole university.

(A mechanical engineer told me they saw electrical/computer engineers the same way: waveforms on a screen instead of real physical things that you could touch, change, and fix.)

I don't think any of us really understood what boxes-and-arrows engineering really was back then, but luckily for you, now I'm old. Let me tell you some stories.

What is systems design?

I started thinking more clearly about systems design when I was at a big tech company and helped people refine their self-promotion employee review packets. Most of it was straightforward, helping them map their accomplishments to the next step up the engineering ladder:

As a Novice going for Junior, you had to prove you could fix bugs without too much supervision;
Going for Senior, you had to prove you could implement a whole design with little supervision;
Going for Staff, you had to show you could produce designs based on business problems with basically no management;
Going for Senior Staff, you had to solve bigger and bigger business problems; and so on.

After helping a few dozen people with their assessments, I noticed a trend. Most developers mapped well onto the ladder, but some didn't fit, even though they seemed like great engineers to me.

There were two groups of misfits:

People who maxed out as a senior engineer (building things) but didn't seem to want to, or be able to, make it to staff engineer (translating business problems).
People who were ranked at junior levels, but were better at translating business problems than at fixing bugs.

Group #1 was formally accounted for: the official word was most employees should never expect to get past Senior Engineer. That's why they called it Senior. It wasn't not much consolation to people who wanted to earn more money or to keep improving for the next 20-30 years of a career, but it was something we could talk about.

(The book Radical Candor by Kim Scott has some discussion about how to handle great engineers who just want to build things. She suggests a separate progression for "rock solid" engineers, who want to become world-class experts at things they're great at, and "steep trajectory" engineers, who might have less attention to detail but who want to manage ever-bigger goals and jump around a lot.)

People in group #2 weren't supposed to exist. They were doing some hard jobs - translating business problems into designs - with great expertise, but these accomplishments weren't interesting to the junior-level promotion committees, who had been trained to look for "exactly one level up" attributes like deep technical knowledge in one or two specific areas, a history of rapid and numerous bug fixes, small independent launches, and so on. Meanwhile, their peers who couldn't (yet) architect their way out of a paper bag rose more quickly through the early ranks, because they wrote reams of code fast.

Tanya Reilly has an excellent talk (and transcribed slides) called Being Glue that perfectly captures this effect. In her words: "Glue work is expected when you're senior... and risky when you're not."

What she calls glue work, I'm going to call systems design. They're two sides of the same issue. Humans are the most unruly systems of all, and yet, amazingly, they follow many of the same patterns as other systems.

People who are naturally excellent at glue work often stall out early in the prescribed engineering pipeline, even when they'd be great in later stages (staff engineers, directors, and executives) that traditional engineers struggle at. In fact, it's well documented that an executive in a tech company requires almost a totally different skill set than a programmer, and rising through the ranks doesn't prepare you for that job at all. Many big tech companies hire executives from outside the company, and sometimes even from outside their own industry, for that reason.

...but I guess I still haven't answered the question. What is systems design? It's the thing that will eventually kill your project if you do it wrong, but probably not right away. It's macroeconomics instead of microeconomics. It's fixing which promotion ladders your company even has, rather than trying to climb the ladders. It's knowing when a distributed system is or isn't appropriate, not just knowing how to build one. It's repairing the incentives in a political system, not just getting elected and passing your favourite laws.

Most of all, systems design is invisible to people who don't know how to look for it. At least with code, you can measure output by the line or the bug, and you can hire more programmers to get more code. With systems design, the key insight might be a one-sentence explanation given at the right time to the right person, that affects the next 5 years of work, or is the difference between hypergrowth and steady growth.

Sorry, I don't know how to explain it better than that. What I can do instead is talk about some systems design problems and archetypes that repeat, over and over, across multiple fields. If you can recognize these archetypes, and handle them before they kill your project, you're on your way to being a systems designer.

Systems of control: hierarchies and decentralization

Let's start with an obvious one: the problem of centralized vs distributed control structures. If I ask you what's a better org structure: a command-and-control hierarchy or a flat organization, most people have been indoctrinated to say the latter. Similarly if I ask whether you should have an old crusty centralized database or a fancy distributed database, everyone wants to build the latter. If you're an SRE and we start talking about pets and cattle, you always vote for cattle. You'd laugh at me if I suggested using anything but a distributed software version control system (ie. git). The future of money, I've heard, is distributed decentralized cryptocurrency. If you want to defeat censorship, you need a distributed social network. The trend is clear. What's to debate?

Well, real structures are more complicated than that. The best introductory article I know on this topic is Jo Freeman's The Tyranny of Structurelessness, which includes the famous quote: "This apparent lack of structure too often disguised an informal, unacknowledged and unaccountable leadership that was all the more pernicious because its very existence was denied."

"Informal, unacknowledged, and unaccountable" control is just as common in distributed computing systems as it is in human social systems.

The truth is, nearly every attempt to design a hierarchy-free, "flat" control system just moves the central control around until you can't see it anymore. Human structures all have leaders, whether implicit or explicit, and the explicit ones tend to be more diverse.

The web depends on centrally controlled DNS and centrally approved TLS certificate issuers; the global Internet depends on a small cabal who sorts out routing problems. Every blockchain depends on whoever decides if your preferred chain will fork this week, and whoever runs the popular exchanges, and whoever decides whether to arrest those people. Distributed radio networks depend on centralized government spectrum licenses. Democracy depends on someone enforcing your right to vote. Capitalism depends on someone enforcing the rules of a "free" marketplace.

At my first startup, we tried to run the development team as a flat organization, where everyone's opinions were listened to and everyone could debate the best way to do something. The overall consensus was that we mostly succeeded. But I was shocked when one of my co-workers said to me afterward: "Our team felt flat and egalitarian. But you can't ever forget that it was only that way because you forced it to be that way."

Truly distributed systems do exist. Earth's ecosystem is perhaps one (although it's becoming increasingly fragile and dependent on humans not to break it). Truly distributed databases using Raft consensus or similar algorithms certainly exist and work. Distributed version control (like git) really is distributed, although we ironically end up re-centralizing our usage of it through something like Github.

CAP theorem is perhaps the best-known statement of the tradeoffs in distributed systems, between consistency, availability, and "partition tolerance." Normally we think of the CAP theorem as applying to databases, but it applies to all distributed systems. Centralized databases do well at consistency and availability, but suck at partition tolerance; so do authoritarian government structures.

In systems design, there is rarely a single right answer that applies everywhere. But with centralized vs distributed systems, my rule of thumb is to do exactly what Jo Freeman suggested: at least make sure the control structure is explicit. When it's explicit, you can debug it.

Chicken-egg problems

Another archetypal systems design question is the "chicken-egg problem," which is short for: which came first, the chicken or the egg?

In case that's not a common question where you come from, the idea is eggs produce chickens, and chickens produce eggs. That's all fine once it's going, but what happened, back in ancient history? Was the very first step in the first iteration an egg, or a chicken?

The question sounds silly and faux-philosophical at first, but there's a real answer and that answer applies to real problems in the business world.

The answer to the riddle is "neither"; unless you're a Bible literalist, you can't trace back to the Original Chicken that laid the Original Egg. Instead there was probably a chicken-like bird that laid a mostly egg-ish egg, and before that, there were millions of years of evolution, going all the way back to single-celled organisms and whatever phenomenon first spawned those. What came "first"? All that other stuff.

Chicken-egg problems appear all the time when building software or launching products. Which came first, HTML5 web browsers or HTML5 web content? Neither, of course. They evolved in loose synchronization, tracing back to the first HTML experiments and way before HTML itself, growing slowly and then quickly in popularity along the way.

I refer to chicken-egg problems a lot because designers are oblivious to them a lot. Here are some famous chicken-egg problems:

Electrical distribution networks
Phone and fax technologies
The Internet
IPv6
Every social network (who will use it if nobody is using it?)
CDs, DVDs, and Blu-Ray vs HD DVD
HDTV (1080p etc), 4k TV, 8k TV, 3D TV
Interstate highways
Company towns (usually built around a single industry)
Ivy league universities (could you start a new one?)
Every new video game console
Every desktop OS, phone OS, and app store

The defining characteristic of a chicken-egg technology or product is that it's not useful to you unless other people use it. Since adopting new technology isn't free (in dollars, or time, or both), people aren't likely to adopt it unless they can see some value, but until they do, the value isn't there, so they don't. A conundrum.

It's remarkable to me how many dreamers think they can simply outwait the problem ("it'll catch on eventually!") or outspend the problem ("my new mobile OS will be great, we'll just subsidize a few million phones"). And how many people think getting past a chicken-egg problem, or not, is just luck.

But no! Just like with real chickens and real eggs, there's a way to do it by bootstrapping from something smaller. The main techniques are to lower the cost of adoption, and to deliver more value even when there are fewer users.

Video game console makers (Nintendo, Sony, Microsoft) have become skilled at this; they're the only ones I know who do it on purpose every few years. Some tricks they use are:

Subsidizing the cost of early console sales.
Backward compatibility, so people who buy can use older games even before there's much native content.
Games that are "mostly the same" but "look better" on the new console.
Compatible gamepads between generations, so developers can port old games more easily.
"Exclusive launch titles": co-marketing that ensures there's value up front for consumers (new games!) and for content producers (subsidies, free advertising, higher prices).

In contrast, the designs that baffle me the most are ones that absolutely ignore the chicken-egg problem. Firefox and Ubuntu phones, distributed open source social networks, alternative app stores, Linux on the desktop, Netflix competitors.

Followers of this diary have already seen me rant about IPv6: it provides nearly no value to anyone until it is 100% deployed (so we can finally shut down IPv4!), but costs immediately in added complexity and maintenance (building and running a whole parallel Internet). Could IPv6 have been rolled out faster, if the designers had prioritized unwinding the chicken-egg problem? Absolutely yes. But they didn't acknowledge it as the absolute core of their design problem, the way Android, Xbox, Blu-Ray, and Facebook did.

If your product or company has a chicken-egg problem, and you can't clearly spell out your concrete plan for solving it, then investors definitely should not invest in your company. Solving the chicken-egg problem should be the first thing on your list, not some afterthought.

By the way, while we're here, there are even more advanced versions of the chicken-egg problem. Facebook or faxes are the basic form: the more people who use Facebook or have a fax machine, the more value all those users get from each other.

The next level up is a two-sided market, such as Uber or Ebay. Nobody can get a ride from Uber unless there are drivers; but drivers don't want to work for Uber unless they can get work. Uber has to attract both kinds of users (and worse: in the same geographic region! at the same time of day!) before either kind gets anything from the deal. This is hard. They decided to spend their way to success, although even Uber was careful to do so only in a few markets at a time, especially at first.

The most difficult level I know is a three-sided market. For example, UberEats connects consumers, drivers, and restaurants. Getting a three-sided market rolling is insanely complicated, expensive, and failure-prone. I would never attempt it myself, so I'm impressed at the people who try. UberEats had a head start since Uber had consumers and drivers in their network already, and only needed to add "one more side" to their market. Most of their competitors had to attract all three sides just to start. Whoa.

If you're building a one-sided, two-sided, or three-sided market, you'd better understand systems design, chickens, and eggs.

Second-system effect

Taking a detour from business, let's move to an issue that engineers experience more directly: second-system effect, a term that comes from the excellent book, The Mythical Man-Month, by Fred Brooks.

Second system effect arises through the following steps:

An initial product starts small and is built incrementally, starting with a low budget and a few users.
Over time, the product gains popularity and becomes profitable.
The system evolves, getting more and more hacks on top, and early design tradeoffs start to be a bottleneck.
The engineers figure out a new design that would fix all the mistakes we know about, plus more! (And they're probably right.)
Since the product is already popular, it's easy to justify spending the time to "do it right this time" and "build a strong platform for the next 10 years." So a project is launched to rewrite everything from scratch. It's expected to take several months, maybe a couple of years, and a big engineering team.

Sound familiar? People were trying this back in 1975 when the book was written, and they're still trying it now. It rarely goes well; even when it does work, it's incredibly painful.

25 years after the book, Joel Spolsky wrote Things you should never do, part 1 about the company-destroying effect of Netscape/Mozilla trying this. "They did it by making the single worst strategic mistake that any software company can make: they decided to rewrite the code from scratch."

[Update 2020-12-28: I mention Joel's now-20-year-old article not because Mozilla was such a landmark example, but because it's such a great article.]

Some other examples of second system effect are IPv6, Python 3, Perl 6, the Plan9 OS, and the United States system of government.

The results are remarkably consistent:

The project takes longer than expected to reach feature parity.
The new design often does solve the architectural problems in the original; however, it unexpectedly creates new architectural problems that weren't in the original.
Development time is split (or different developers are assigned) between maintaining the old system and launching the new system.
As the project gets increasingly overdue, project managers are increasingly likely to shut down the old system to force users to switch to the new one, even though users still prefer the old one.

Second systems can be merely expensive, or they can bankrupt your company, or destroy your user community. The attention to Perl 6 severely weakened the progress of perl; the work on Python 3 fractured the python community for more than a decade (and still does); IPv6 is obstinately still trying to deprecate IPv4, 25 years later, even though the problems it was created to solve are largely obsolete.

As for solutions, there isn't much to say about the second system effect except you should do your utmost to prevent it; it's entirely self-inflicted. Refactor your code instead. Even if it seems like incrementalism will be more work... it's worth it. Maintaining two systems in parallel is a lot more expensive than you think.

In his book, Fred Brooks called it the "second" system on purpose, because it was his opinion that after experiencing it once, any designer will build their third and later systems more incrementally so they never have to go through that again. If you're lucky enough to learn from historical wisdom, perhaps even your second system won't suffer from this strategic error.

A more embarrassing related problem is when large companies try to build a replacement for their own first system, but the developers of the first system have left or have already learned their Second System Lesson and are not willing to play that game. Thus, a new team is assembled to build the replacement, without the experience of having built the first one, but with all the confidence of a group of users who are intimately experienced with its surface flaws. I don't even know what this phenomenon should be called; the vicarious second system effect? Anyway, my condolences if you find yourself building or using such a product. You can expect years of pain.

[Update 2020-12-28: someone reminded me that CADT ("cascade of attention-deficit teenagers") is probably related to this last phenomenon.]

Innovator's dilemmas

Let's finally talk about a systems design issue that's good news for your startup, albeit bad news for big companies. The Innovator's Dilemma is a great book by Clayton Christensen that discusses a fascinating phenomenon.

Innovator's dilemmas are so elegant and beautiful you can hardly believe they exist as such a repeatable abstraction. Here's the latest one I've heard about, via an Anandtech Article about Apple Silicon:

A summary of the Innovator's Dilemma is as follows:

You (Intel in this case) make an awesome product in a highly profitable industry.
Some crappy startup appears (ARM in this case) and makes a crappy competing product with crappy specs. The only thing they seem to have going for them is they can make some low-end garbage for cheap.
As a big successful company, your whole business is optimized for improving profits and margins. Your hard-working employees realize that if they cede the ultra-low-end garbage portion of the market to this competitor, they'll have more time to spend on high-valued customers. As a bonus, your average margin goes up! Genius.
The next year, your competitor's product gets just a little bit better, and you give up the new bottom of your market, and your margins and profits further improve. This cycle repeats, year after year. (We call this "retreating upmarket.")
The crappy competitor has some kind of structural technical advantage that allows their performance (however you define performance; something relevant to your market) to improve, year over year, at a higher percentage rate than your product can. And/or their product can do something yours can't do at all (in ARM's case: power efficiency).
Eventually, one year, the crappy competitor's product finally exceeds the performance metrics of your own product, and promptly blows your entire fucking company instantly to smithereens.

Hey now, we've started swearing, was that really called for? Yes, I think so. If I were an Intel executive looking at this chart and Apple's new laptops, I would be scared out of my mind right now. There is no more upmarket to retreat to. The competitor's product is better, and getting better faster than mine. The game is already over, and I didn't even realize I was playing.

What makes the Innovator's Dilemma so beautiful, from a systems design point of view, is the "dilemma" part. The dilemma comes from the fact that all large companies are heavily optimized to discard ideas that aren't as profitable as their existing core business. Any company that doesn't optimize like this fails; by definition their profitability would go down. So thousands of worker bees propose thousands of low-margin and high-margin projects, and the company discards the former and invests heavily in the latter (this is called "sustaining innovation" in the book), and they keep making more and more money, and all is well.

But this optimization creates a corporate political environment (aha, you see we're still talking about systems design?) where, for example, Intel could never create a product like ARM. A successful low-priced chip would take time, energy, and profitability away from the high-priced chips, and literally would have made Intel less successful for years of its history. Even once ARM appeared and their trendline of improvements was established, they still had lower margins, so competing with them would still cannibalize their own high-margin products, and worse, now ARM had a head start.

In case you're a big company reading this: the book has a few suggestions for what you can do to avoid this trap. But if you're Intel, you should have read the book a few years ago, not now.

Innovator's dilemma plots are the prettiest when discussing hardware and manufacturing, but the concept applies to software too, especially when software is held back by a hardware limitation. For example, distributed version control systems (where you download the entire repository history to every client) were amusing toys until suddenly disks were big enough and networks were fast enough, and then DVCSes wiped out everything else (except in projects with huge media files).

Fancy expensive databases were the only way to get high transaction throughput, until SSDs came along and made any dumb database fast enough for most jobs.

Complicated database indexes and schemas were great until AWS came along and let everyone just brute force mapreduce everything using short-term rental VMs.

JITs were mostly untenable until memory was so much slower than CPU that compiling was not the expensive part. Software-based network packet processing on a CPU was slower than custom silicon until generic CPUs got fast enough relative to RAM. And so on.

The Innovator's Dilemma is the book that first coined the term "disruptive innovation." Nowadays, startups talk about disrupting this and disrupting that. "Disruption" is an exciting word, everybody wants to do it! The word disruption has lost most of its meaning at this point; it's a joke as often as a serious claim.

But in the book, it had a meaning. There are two kinds of innovations: sustaining and disruptive. Sustaining is the kind that big companies are great at. If you want to make the fastest x86 processor, nobody does it better than Intel (with AMD occasionally nipping at their heels). Intel has every incentive to keep making their x86 processors better. They also charge the highest margins, which means the greatest profits, which means the most money available to pour into more sustaining innovation. There is no dilemma; they dump money and engineers and time into that, and they mostly deliver, and it pays off.

A "disruptive" innovation was meant to refer to specifically the kind you see in that plot up above: the kind where an entirely new thing sucks for a very long time, and then suddenly and instantly blows you away. This is the kind that creates the dilemma.

If you're a startup and you think you have a truly disruptive innovation, then that's great news for you. It's a perfect answer to that awkward investor question, "What if [big company] decides to do this too?" because the honest truth is "their own politics will tear that initiative apart from the inside."

The trick is to determine whether you actually have one of these exact "disruption" things. They're rare. And as an early startup, you don't yet have a historical plot like the one above that makes it clear; you have to convince yourself that you'll realistically be able to improve your thing faster than the incumbent can improve theirs, over a long period of time.

Or, if your innovation only depends on an existing trend - like in the software-based packet processing example above - then you can try to time it so that your software product is ready to mature at the same time as the hardware trend crosses over.

In conclusion: watch out for systems design. It's the sort of thing that can make you massively succeed or completely fail, independent of how well you write code or run your company, and that's scary. Sometimes you need some boxes and arrows.

Thoughts you mightn't'a thunk about remote meetings

Wed, 21 Jul 2021 19:34:45 +0000

Welcome to this week's edition of "building a startup in 2020," in which all your meetings are suddenly remote, and you probably weren't prepared for it.

I know I wasn't. We started a "fully remote" company back in 2019, but that was supposed to mean we still got together in person every month or two to do strategic planning, share meals, and resolve any accumulated conflicts. Well, not this year. Instead, we had to learn to have better remote meetings, all while building our whole team from scratch.

You can find endless articles on the Internet about how to have a good meeting. So many articles, in fact, that I can no longer find the ones that I liked the best, so that I can quote from them and give them credit :( Sorry! I'll have to paraphrase. Please send links if you think some of this sounds familiar.

Here are a few meeting tips I've accumulated over the years, with some additions from the last few months.

The most efficient meeting is no meeting.

Let's start with what should be obvious by now: sometimes you don't need a meeting at all. For example, status updates almost always are better delivered in some written medium (like email) that can be retained for future reference, and skimmed (or ignored) faster than people can speak.

Alas, skipping meetings doesn't solve every problem, or else remote work would be a lot easier for everyone.

Remember: every minute costs multiple person-minutes.

Imagine a meeting where a manager is presenting to 9 people. That costs 1+9 person-minutes per minute. A single one-hour meeting costs you 10 hours of employee salaries! With modern tech employees, that adds up really, really fast. You need to spend it wisely.

Now, assuming everyone needed to see that presentation - which is rarely the case - then one big meeting is a pretty efficient way to go. You can inform N people in O(N) minutes. That's pretty close to optimal. Of course, in the purest form of a presentation meeting, you could have just recorded the presentation in advance and let some of the people watch it at 2x speed, saving precious minutes. But that doesn't work in the typical case where you allow some Q&A, either during or afterwards.

As a meeting trends away from a presentation and toward group discussion, efficiency drops fast. Almost always, a discussion will be dominated by 2-3 people, leaving the others to sit and get bored. We all know what to do here, even though we don't always do it: split the discussion into a separate, much smaller meeting with just the people who care, and have them provide a text status report back when it's done.

The text status report is really important, even if you think nobody cares about the result of the meeting. That's because without the status report, nobody can be quite sure it's safe to skip the meeting. If they can read text notes later, it gives them the confidence to not show up. That typically saves far more cost than the cost of writing down the notes. (To say nothing of the cost of forgetting the decision and having to meet again later.)

Around here we take seriously copious meeting notes. It's a bit ridiculous. But it pays off frequently.

In big meetings, some people don't talk.

A related problem with big meetings is the people who don't get to talk even though they want to, or who always get interrupted or talked over. (There was a really great article about this a few months ago, but I can't find it, alas.)

Historically this has been much worse when your meeting has remote attendees, because it turns out latency blows up our social cues completely. Nobody quite knows how long to wait before speaking, but one thing's for sure: when some of the team is sitting in one room (~zero latency), and some are remote (typically hundreds of milliseconds of latency), the remote people almost never get to talk.

It's not just latency, either; remote users typically can't hear as well, and aren't heard as well, and people don't notice their gestures and body language.

Unexpectedly, the 2020 work-from-home trend has helped remote workers, by eliminating the central room with a bunch of zero-latency people. It levels the playing field, although some people invariably still have worse equipment or worse latency.

That helps the fairness problem, but it doesn't solve personality and etiquette problems. Even if everyone's all in the same room, some people are naturally tuned to wait longer before speaking, and some wait for less time, and the latter almost always end up dominating the conversation. The only ways I know to deal with this are a) have smaller meetings, and b) have a facilitator or moderator who decides who gets to talk.

You can get really complicated about meeting facilitation. (See also: that article I can't find, sigh.) Some conferencing tools nowadays have a "raise hand" button, or they count, for each user, the total amount of time they've spent talking, so people can self regulate. Unfortunately, these fancy features are not well correlated with the other, probably more important, conferencing software features like "not crashing" or "minimizing latency" or "having a phone dial-in just in case someone's network flakes out."

It turns out that in almost all tools, you can use the "mute" feature (which everyone has) to substitute for a "raise hand" feature (which not everyone has, and which often works badly even when they do). Have everyone go on mute, and then unmuting yourself is like raising your hand. The facilitator can call on each unmuted person in turn.

All these tricks sound like good ideas, but they haven't caught on for us. Everyone constantly muting or raising their hand, or having to wait for a facilitator before they can speak, kills the flow of a conversation and makes it feel a bit too much like Robert's Rules of Order. Of course, that's easy for me to say; I'm one of the people who usually ends up speaking either way.

When I'm in a meeting, I try to pay attention to everyone on the screen to see if someone looks like they want to talk, but is getting talked over. But that's obviously not a perfect solution given my human failings and the likelihood that some people might want to speak but don't make it very obvious.

Compared to all that fancy technique, much more effective has been just to make meetings smaller. With 3-4 people in a meeting, all this matters a lot less. It's easy to see if someone isn't participating or if they have something to say. And with a 2-person meeting, it's downright trivial. We'll get to that in a bit.

Amazon-style proposal review meetings

You can use a different technique for a meeting about a complicated product or engineering proposal. The two variants I know are the supposed "2-page review" or "6-pager review" meetings at Amazon (although I've never worked at Amazon), and the "design review" meetings I saw a few times back at a different bigco when I worked there.

The basic technique is:

Write the doc in advance
Distribute the doc to everyone interested
People can comment and discuss in the document before the meeting
The meeting owner walks through any unresolved comments in the document during the meeting, while someone else takes notes.

In the Amazon variant of this, "in advance" might be during the meeting itself, when people apparently sit there for a few minutes reading the doc in front of everyone else. I haven't tried that; it sounds awkward. But maybe it works.

In the variant I've done, we talk about only the document comments, and it seems to work pretty well. First, it avoids the tendency to just walk through a complicated doc in front of everyone, which is very inefficient since they've already read it. Second, it makes sure that everyone who had an unresolved opinion - and thus an unresolved comment in the doc - gets their turn to speak, which helps the moderation/etiquette problem.

So this style is functional. You need to enforce that the document is delivered far enough in advance, and that everyone reads it well in advance, so there can be vigorous discussion in the text ahead of time.

You might wonder, what's the point of the meeting, if you're going to put all the comments in text form anyway?

In my experience, the biggest advantage of the meeting is simply the deadline. We tried sending out design docs without a design review meeting, and people would never finish reading the doc, so the author never knew it was done. By scheduling a meeting, everyone knows the time limit for reviews, so they actually read the doc by then. And of course, if there are any really controversial points, sometimes it's easier to resolve them in a meeting.

Conversely, a design review without an already-commented doc tends to float in the ether, go overtime, and not result in a decision. It also means fewer people can skip the meeting; when people have read and commented on the doc in advance, many of the comments can be entirely resolved in advance. Only people with outstanding issues need to attend the review.

"Management by walking around"

An underappreciated part of big office culture is the impromptu "meetings" that happen between people sitting near each other, or running into each other in the mini-kitchen. A very particular variant of these impromptu meetings is "management by walking around," as in, a manager or executive wanders the floor of the building and starts random conversations of the form "how's it going?" and "what are you up to this week?" and "is customer X still having problems?"

At first glance, this "walking around" style seems very inefficient and incomplete. A big executive at a big company can't ever talk to everyone. The people they talk to aren't prepared because it's not a "real" meeting. It doesn't follow the hierarchy, so you have inefficiently duplicated communication channels.

But it works better than you'd think! The reasons are laid out in High Output Management by Andy Grove (of Intel fame), which I reviewed last year. The essential insight in that book is that these meetings should be used, not for the manager to "manage" employees, but for the manager to get a random selection of direct, unfiltered feedback.

As the story goes, in a company full of knowledge workers, the people at the bottom of the hierarchy tend to know the most about whatever problem they're working on. The managers and executives tend to know far fewer details, and so are generally ill-equipped to make decisions or give advice. Plus, the executive simply doesn't have time to give advice to everyone, so if walking around was part of the advice-giving process, it would be an incomplete, unfair, and unhelpful disaster.

On the other hand, managers and executives are supposed to be the keepers of company values (see my earlier review) and bigger context. By collecting a random sample of inputs from individual contributors on the floor, they can bypass the traditional hierarchical filtering mechanism (which tends to turn all news into good news after only one or two levels of manager), thus getting a clearer idea of how the real world is going, which can help refine the strategy.

I still think it's a great book. You should read it.

But one little problem: we're in a pandemic. There's no building, no floor, and no walking. WWAGD (What Would Andy Grove Do)?

Well, I don't know. But what I do is...

Schedule way too many 1:1 meetings

Here's something I started just a couple of months ago, which has had, I think, a really disproportionate outcome: I started skipping most larger meetings, and having 1:1s with everyone in the company instead.

Now, "everyone in the company" is a luxury I won't be able to keep up forever, as we grow. Right now, I try to schedule about an hour every two weeks with more senior people, and about 30 minutes every week with more junior people (like co-op students). Sometimes these meetings get jiggled around or grow or shrink a bit, but it averages about 30 minutes per person per week, and this adds up pretty fast, especially if I also want to do other work. Hypothetically.

I don't know if there are articles about scheduling 1:1s, but bi-weekly 1:1 meetings also have a separate problem, which is the total mess that ensues if you skip them. Then it turns out you're only meeting with some people once a month, which seems too rare. I haven't really figured this out, other than to completely remangle my schedule if I ever need to take a vacation or sick day, alas. Something about this scheme is going to need to improve.

As we grow, I think I can still maintain a "meet with everyone" 1:1 schedule, it just might need to get more and more complex, where I meet some people more often and some people less often, to give a weighted "random" sample across the whole team, over a longer period of time. We'll see.

Anyway, the most important part of these 1:1s is to do them Andy Grove style: they're for collecting feedback much more than "managing." The feedback then turns into general strategy and plans, that can be discussed and passed around more widely.

Formalizing informal donut chats

The above was for me. I'm the CEO, so I want to make sure to talk to everyone. Someday, eventually we're going to get all organized and have a management hierarchy or something, I guess, and then presumably other executives or managers will want to do something similar in their own orgs and sub-orgs.

Even sooner, though, we obviously can't expect all communications to pass through 1:1s with the CEO. Therefore, shockingly, other people might need to talk directly to each other too. How does that work? Does everyone need to talk to everyone else? O(N^2) complexity?

Well, maybe. Probably not. I don't know. For now, we're using a Slack tool called Donut which, honestly, is kinda buggy and annoying, but it's the best we have. Its job is simply to randomly pair each person with one other person, once a week, for a 1:1, ostensibly to eat virtual donuts together. I'm told it is better than nothing. I opted out since I already have 1:1s with everyone, thank goodness, because the app was driving me nuts.

What doesn't work well at all, unfortunately, is just expecting people to have 1:1 meetings naturally when an issue comes up. Even if they're working on the same stuff. It's a very hard habit to get into, especially when you have a bunch of introverted tech industry types. Explicitly prompting people to have 1:1 meetings with each other works better.

(Plus, there's various advice out there that says regularly scheduled 1:1s are great for finding problems that nobody would ever schedule a meeting for, even if you do work in the same office. "We have to use up this 30-minute meeting, no matter what" is miraculous for surfacing small conflicts before they turn into large ones.)

"Pairing" meetings

As a slight variation on the donut, some of my co-workers have invented a more work-oriented style of random crossover meeting where instead of just eating virtual donuts, they share a screen and do pair programming (or some other part of their regular work) with the randomly selected person for an hour or two. I'm told this has been pretty educational and fun, making things feel a bit more collaborative like it might feel in an office.

Do you have any remote meeting tips?

Can I work for a bad company and still be a good person?

Mon, 23 Nov 2020 10:14:12 +0000

No.

The Curse of Unreasonably Sized Networks

Sun, 20 Nov 2022 02:04:01 +0000

I just did a presentation at SREcon Conversations (which I call SREconcon) EMEA, called the "Curse of Unreasonably Sized Networks." I talked about the series of Dunbar's numbers and how they relate to different kinds of human social networks, and surprisingly, also to the evolution of the Internet.

Video recording of the talk (34 minutes)
Slides (pdf)
SREcon EMEA 2020 speaker's page

IPv4, IPv6, and a sudden change in attitude

Wed, 22 Jul 2020 18:13:06 +0000

A few years ago I wrote The World in Which IPv6 was a Good Design. I'm still proud of that article, but I thought I should update it a bit.

No, I'm not switching sides. IPv6 is just as far away from universal adoption, or being a "good design" for our world, as it was three years ago. But since then I co-founded a company that turned out to be accidentally based on the principles I outlined in that article. Or rather, from turning those principles upside-down.

In that article, I explored the overall history of networking and the considerations that led to IPv6. I'm not going to cover that ground again. Instead, I want to talk about attitude.

Internets, Interoperability, and Postel's Law

Did you ever wonder why "Internet" is capitalized?

When I first joined the Internet in the 1990s, I found some now-long-lost introductory tutorial. It talked about the difference between an internet (lowercase i) and the Internet (capital I). An internet is "any network that connects smaller networks together." The Internet is... well... it turns out that you don't need more than one internet. If you have two internets, it is nearly unavoidable that someone will soon figure out how to connect them together. All you need is one person to build that one link, and your two internets become one. By induction then, the Internet is the end result when you make it easy enough for a single motivated individual to join one internet to another, however badly.

Internets are fundamentally sloppy. No matter how many committees you might form, ultimately connections are made by individuals plugging things together. Those things might follow the specs, or not. They might follow those specs well, or badly. They might violate the specs because everybody else is also violating the specs and that's the only way to make anything work. The connections themselves might be fast or slow, or flakey, or only functional for a few minutes each day, or subject to amateur radio regulations, or worse. The endpoints might be high-powered servers, vending machines, toasters, or satellites, running any imaginable operating system. Only one thing's for sure: they all have bugs.

Which brings us to Postel's Law, which I always bring up when I write about networks. When I do, invariably there's a slew of responses trying to debate whether Postel's Law is "right," or "a good idea," as if it were just an idea and not a force of nature.

Postel's Law says simply this: be conservative in what you send, and liberal in what you accept. Try your best to correctly handle the bugs produced by the other end. The most successful network node is one that plans for every "impossible" corruption there might be in the input and does something sensible when it happens. (Sometimes, yes, "something sensible" is to throw an error.)

[Side note: Postel's Law doesn't apply in every situation. You probably don't want your compiler to auto-fix your syntax errors, unless your compiler is javascript or HTML, which, kidding aside, actually were designed to do this sort of auto-correction for Postel's Law reasons. But the law does apply in virtually every complex situation where you need to communicate effectively, including human conversations. The way I like to say it is, "It takes two to miscommunicate." A great listener, or a skilled speaker, can resolve a lot of conflicts.]

Postel's Law is the principle the Internet is based on. Not because Jon Postel was such a great salesperson and talked everyone into it, but because that is the only winning evolutionary strategy when internets are competing. Nature doesn't care what you think about Postel's Law, because the only Internet that happens will be the one that follows Postel's Law. Every other internet will, without exception, eventually be joined to The Internet by some goofball who does it wrong, but just well enough that it adds value, so that eventually nobody will be willing to break the connection. And then to maintain that connection will require further application of Postel's Law.

IPv6: a different attitude

If you've followed my writing, you might have seen me refer to IPv6 as "a second internet that not everyone is connected to." There's a lot wrapped up in that claim. Let's back up a bit.

In The World in Which IPv6 was a Good Design, I talked about the lofty design goals leading to IPv6: eliminate bus networks, get rid of MAC addresses, no more switches and hubs, no NATs, and so on. What I didn't realize at the time, which I now think is essential, is that these goals were a fundamental attitude shift compared to what went into IPv4 (and the earlier protocols that led to v4).

IPv4 evolved as a pragmatic way to build an internet out of a bunch of networks and machines that existed already. Postel's Law says you'd best deal with reality as it is, not as you wish it were, and so they did. When something didn't connect, someone hacked on it until it worked. Sloppy. Fits and starts, twine and duct tape. But most importantly, nobody really thought this whole mess would work as well as it turned out to work, or last as long as it turned out to last. Nobody knew, at the time, that whenever you start building internets, they always lead inexorably to The Internet.

These (mostly) same people, when they started to realize the monster they had created, got worried. They realized that 32-bit addresses, which they had originally thought would easily last for the lifetime of their little internet, were not even enough for one address per person in the world. They found out, not really to anyone's surprise, that Postel's Law, unyielding as it may be, is absolutely a maintenance nightmare. They thought they'd better hurry up and fix it all, before this very popular Internet they had created, which had become a valuable, global, essential service, suddenly came crashing down and it would all be their fault.

[Spoiler: it never did come crashing down. Well, not permanently. There were and are still short-lived flare-ups every now and then, but a few dedicated souls hack it back together, and so it goes.]

IPv6 was created in a new environment of fear, scalability concerns, and Second System Effect. As we covered last time, its goal was to replace The Internet with a New Internet — one that wouldn't make all the same mistakes. It would have fewer hacks. And we'd upgrade to it incrementally over a few years, just as we did when upgrading to newer versions of IP and TCP back in the old days.

We can hardly blame people for believing this would work. Even the term "Second System Effect" was only about 20 years old at the time, and not universally known. Every previous Internet upgrade had gone fine. Nobody had built such a big internet before, with so much Postel's Law, with such a variety of users, vendors, and systems, so nobody knew it would be different.

Well, here we are 25 years later, and not much has changed. If we were feeling snarky, we could perhaps describe IPv6 as "the String Theory of networking": a decades-long boondoggle that attracts True Believers, gets you flamed intensely if you question the doctrine, and which is notable mainly for how much progress it has held back.

Luckily we are not feeling snarky.

Two Internets?

There are, of course, still no exceptions to the rule that if you build any internet, it will inevitably (and usually quickly) become connected to The Internet.

I wasn't sitting there when it happened, but it's likely the very first IPv6 node ran on a machine that was also connected to IPv4, if only so someone could telnet to it for debugging. Today, even "pure IPv6" nodes are almost certainly connected to a network that, if configured correctly, can find a way to any IPv4 node, and vice versa. It might not be pretty, it might involve a lot of proxies, NATs, bridges, and firewalls. But it's all connected.

In that sense, there is still just one Internet. It's the big one. Since day 1, The Internet has never spoken just one protocol; it has always been a hairy mess of routers, bridges, and gateways, running many protocols at many layers. IPv6 is one of them.

What makes IPv6 special is that its proponents are not content for it to be an internet that connects to The Internet. No! It's the chosen one. Its destiny is to be The Internet. As a result, we don't only have bridges and gateways to join the IPv6 internets and the IPv4 internet (although we do).

Instead, IPv6 wants to eventually run directly on every node. End users have been, uh, rather unwilling to give up IPv4, so for now, every node has that too. As a result, machines are often joined directly to what I call "two competing internets" --- the IPv4 one and the IPv6 one.

Okay, at this point our terminology has become very confusing. Sorry. But all this leads to the question I know you want me to answer: Which internet is better!?

Combinatorics

I'll get to that, but first we need to revisit what I bravely called Avery's Laws of Wifi Reliability, which are not laws, were surely invented by someone else (since they're mostly a paraphrasing of a trivial subset of CAP theorem), and as it turns out, apply to more than just wifi. Oops. I guess the name is wrong in almost every possible way. Still, they're pretty good guidelines.

Let's refresh:

Rule #1: if you have two wifi router brands that work with 90% of client devices, and your device has a problem with one of them, replacing the wifi router brand will fix the problem 90% of the time. Thus, an ISP offering both wifi routers has a [1 - (10% x 10%)] = 99% chance of eventual success.
Rule #2: if you're running two wifi routers at once (say, a primary router and an extender), and both of them work "correctly" for about 90% of the time each day, the chance that your network has no problems all day is 81%.

In Rule #1, which I call "a OR b", success compounds and failure rates drop.

In Rule #2, which I call "a AND b", failure compounds and success drops.

But wait, didn't we add redundancy in both cases?

Depending how many distributed systems you've had to build, this is either really obvious or really mind blowing. Why did the success rate jump to 99% in the first scenario but drop to 81% in the second? What's the difference? And... which one of those cases is like IPv6?

Failover

Or we can ask that question another way. Why are there so many web pages that advise you to solve your connectivity problem by disabling IPv6?

Because automatic failover is a very hard problem.

Let's keep things simple. IPv4 is one way to connect client A to server X, and IPv6 is a second way. It's similar to buying redundant home IPv4 connections from, say, a cable and a DSL provider and plugging them into the same computer. Either way, you have two independent connections to The Internet.

When you have two connections, you must choose between them. Here are some factors you can consider:

Which one even offers a path from A to X? (If X doesn't have an IPv6 address, for example, then IPv6 won't be an option.)
Which one gives the shortest paths from A to X and from X to A? (You could evaluate this using hopcount or latency, for example, like in my old netselect program.)
Which path has the most bandwidth?
Which path is most expensive?
Which path is most congested right now?
Which path drops out least often? (A rebooted NAT will drop a TCP connection on IPv4. But IPv6 routes change more frequently.)
Which one has buggy firewalls or NATs in the way? Do they completely block it (easy) or just act strangely (hard)?
Which one blocks certain UDP or TCP ports, intentionally or unintentionally?
Which one is misconfigured to block certain ICMP packets so that PMTU discovery (always or sometimes) doesn't work with some or all hosts?
Which one blocks certain kinds of packet fragmentation?

A common heuristic called "Happy Eyeballs" is one way to choose between routes, but it covers only a few of those criteria.

The truth is, it's extremely hard to answer all those questions, and even if you can, the answers are different for every combination of A and X, and they change over time. Operating systems, web browsers, and apps, even if they implement Happy Eyeballs or something equivalent, tend to be pretty bad at detecting all these edge cases. And every app has to do it separately!

My claim is that the "choose between two internets" problem is the same as the "choose between two flakey wifi routers on the same SSID" problem (Rule #2). All is well as long as both internets (or both wifi routers) are working perfectly. As soon as one is acting weird, your overall results are going to be weird.

...and the Internet always acts weird, because of the tyranny of Postel's Law. Debugging the Internet is a full time job.

...and now there are two internets, with a surprisingly low level of overlap, so your ISP has to build and debug both.

...and every OS vendor has to debug both protocol implementations, which is more than twice as much code.

...and every app vendor has to test with both IPv4 and IPv6, which of course they don't.

We should not be surprised that the combined system is less reliable.

The dream

IPv6 proponents know all this, whether rationally or intuitively or at least empirically. The failure rate of two wonky internets joined together is higher than the failure rate of either wonky internet alone.

This leads them to the same conclusion you've heard so many times: we should just kill one of the internets, so we can spend our time making the one remaining internet less wonky, instead of dividing our effort between the two. Oh, and, obviously the one we kill will be IPv4, thanks.

They're not wrong! It would be a lot easier to debug with just one internet, and you know, if we all had to agree on one, IPv6 is probably the better choice.

But... we don't all have to agree on one, because of the awesome unstoppable terribleness that is Postel's Law. Nobody can declare one internet or the other to be officially dead, because the only thing we know for sure about internets is that they always combine to make The Internet. Someone might try to unplug IPv4 or IPv6, but some other jerk will plug it right back in.

Purity cannot ever be achieved at this kind of scale. If you need purity for your network to be reliable, then you have an unsolvable problem.

The workaround

One thing we can do, though, is build better heuristics.

Ok, actually we have to do better than that, because it turns out that correctly choosing between the two internets for each connection, at the start of that connection, is not possible or good enough. Problems like PMTU, fragmentation, NAT resets, and routing changes can interrupt a connection partway through and cause poor performance or dropouts.

I want to go back to a side note I left near the end of The World in Which IPv6 was a Good Design: mobile IP. That is, the ability for your connections to keep going even if you hop between IP addresses. If you had IP mobility, then you could migrate connections between your two internets in real time, based on live quality feedback. You could send the same packets over both links and see which ones work better. If you picked one link and it suddenly stopped, you could retransmit packets on the other link and pick up where you left off. Your precise heuristic wouldn't even matter that much, as long as it tries both ways eventually.

If you had IP mobility, then you could convert the "a AND b" scenario (failure compounds) into the "a OR b" scenario (success compounds).

And you know what, forget about IPv4 and IPv6. The same tricks would work with that redundant cable + DSL setup we mentioned above. Or a phone with both wifi and LTE. Or, given a fancy enough wifi client chipset, smoothly switching between multiple unrelated wifi routers.

IP mobility is what we do, in a small way, with Tailscale's WireGuard connections. We try all your Internet links, IPv4 and IPv6, UDP and TCP, relayed and peer-to-peer. We made mobile IP a real thing, if only on your private network for now. And what do you know, the math works. Tailscale's use of WireGuard with two networks is more reliable than with one network.

Now, can it work for the whole Internet?

This article was originally posted to the Tailscale blog

Several grumpy opinions about remote work at Tailscale

Mon, 21 Jun 2021 17:05:18 +0000

As a "fully remote work" company, we had to make some choices about the technologies we use to work together and stay in touch.

We decided early on - about the time we realized all three cofounders live in different cities - that we were going to go all-in on remote work, at least for engineering, which for now is almost all our work. As several people have pointed out before, fully remote is generally more stable than partly remote. In a partially remote team, the remote workers seem to always end up treated as an underclass, overlooked in meetings, bypassed for promotions, fired when they eventually refuse to relocate because the remote work policy inevitably changes (hi, Yahoo!), etc.

The good news with our plan is the founders could "dogfood" a few different remote work ideas ourselves before we ever hired anyone. So we decided to try some stuff. Here's what we discovered.

Notion

We're using Notion as a team wiki and note taking app. It's ... okay. I mean, it's probably the best tool for the job, and it's great in some ways, but it's severely limited in others.

Things I like about Notion:

Great for quick to-do lists and milestone planning
Easy to make persistent hyperlinks between docs
Easy to arrange docs in a hierarchy (but not as organized as good old GracefulTavi)
Tables and Kanban board views are pretty awesome
Just the right level of formatting. When I paste text into Notion, I never worry about it coming out in a weird font or colour.

Things that drive me crazy:

The "show me what changed" view is nearly useless; tons of updates about tiny clutter changes, but no good way to give me a deduplicated list of all the docs that changed. Virtually any wiki's RecentChanges view is better.
Doc comments miss the point of doc comments, by being almost invisible and creating no incentive to resolve them. (Trivia: Rumour has it that Google Docs comments also sucked until an intern showed up and made them insanely better as a 20% project.) There are so many ways that the greatness of Google Docs comments has failed to be copied by every other tool (including, for example, Google Sheets). Everything else sucks. When we want to wordsmith stuff as a team, we move it from Notion into Docs.
Support for to-do lists and "reminders" is there, but pretty weak. For example, there's no way to make repeating reminders or get a consolidated list of to-do items across multiple pages, so people request a separate to-do list app. So far we've resisted, but we won't be able to for much longer.
No API means you can't fix any of the limitations yourself.

Anyway, as they say, there are the tools you complain about and the tools you don't use. I've tried a heckuvalot of content managers and they've all been worse, so Notion it is. To be fair, it's a very big area and hard to please everyone. And I'm really picky. But they're so close...

Keybase to Slack

At first we tried using Keybase to manage our secret keys, and coincidentally its built-in team chat feature for our team chats. Keybase has a bit of a bad reputation because of some of their early cryptography missteps and their (very unfortunate) recent association with cryptocurrency. But whatever you think of their security or business model, their chat system is surprisingly one of the best. You can make channels and securely confirm identities without stupid QR codes; message expiration rules are clear; the notifications are A+. Among other things - and this completely dazzled me - when I read a message on any of my devices, the notification for that message disappears instantly from all my other devices! I didn't even know it was possible to auto-remove obsolete notifications, so seldom is it done.

Which, of course, led me to wonder why it isn't done. In my cynicism I'm sure I can guess why; auto-removing notifications never increases your "engagement" metric. Whereas a completely bogus chat notification from four hours ago, already dealt with four hours ago on a different computer, drives engagement every time. I respect the Keybase people for choosing the path of user happiness, except I suspect they're soon going to need paying users instead of happy users, because that's the world we live in.

However, keybase had some problems for us. First, it guzzles absolutely epic amounts of CPU and memory. If you think Slack is bloaty, Keybase outdoes it by like 2x, plus it has giant memory leaks so you have to restart it all the time. There's no web UI (they're too paranoid about security), and the android app just crashes for me on ChromeOS. In other news, I'm pretty sure I never ever want to hear about a "security and privacy" tool that includes 150MB of Electron (aka "Chromium but with the security and privacy features turned off").

Also, nobody but us uses Keybase, and it doesn't support popular cute things like Github integrations. So unfortunately, we had to give up on it and switch to Slack. Y'all know how Slack works so I hardly need to describe it, but I would summarize it as "absolutely terrible at everything except user lock-in," and here we are. There's a business lesson in there somewhere.

I eventually turned off Slack notifications entirely, after experimenting with many different variations. @here is an abomination; notifications in each "other Slack instance" need to be set separately; it spams your @#$!! phone with every single message anyone types, even while you're on your PC. Forget it, notification privileges revoked, and I've been much happier since.

Gmail

For a while, we tried to run our own email server (in the name of being free of "big tech" for our core systems) but it didn't work out. Gmail's UI gets worse every year (correlated with decreasing information density, though the causation lies elsewhere), but at least it's mostly familiar.

Interestingly, because of Notion and Slack, we hardly use email at all between us internally. It's almost exclusively used for customers and investors.

At the advice of the excellent book The Great CEO Within, I followed the instructions in Andreas Klinger's guide to Gmail Inbox Zero. His combination of Gmail configs is pure genius; it completely changed how I do email, and makes Inbox Zero easy and achievable, by separating the triage and work phases. Highly recommended. I also learned about several Gmail options I didn't know existed.

Streak CRM

We reviewed several CRM tools. The consistent advice we received was, "You'll end up on Salesforce eventually, but don't do it yet." Ok, sure, I can take advice.

Streak was appealing because I wanted something that would integrate extremely tightly with my email. Streak does what I want: I associate an email thread with a particular customer or helpdesk ticket, and then it's magically shared with all the other Streak users in your domain, and it continues sharing as new messages are sent and received, and it's 100% inside the Gmail UI. Not bad at all.

The underlying concept of Streak is what I would call "batshit insane from top to bottom." It has a tough learning curve at first, but so apparently does every CRM. It has scattered features all over that just look like extra buttons or tabs in the Gmail UI. The frustration their dev team must have endured as they implemented this, and the frustration they must continue to endure as they keep it up to date, must be nearly intolerable. But the end result is quite remarkable; these are devs who care about keyboard shortcuts, highly efficient workflows, and making short work of huge batches of emails. I'd say Google should buy them and just integrate the whole thing into standard Gmail, except then Google would kill them with love by accident, as megacorporations usually do with acquisitions, and we'd all be worse off. Oh well.

Anyway it works, I like it. And besides sales, it's quite a remarkably good support/helpdesk ticket system, which it seems to have only tangentially been designed for. Customers don't even know they're in a ticket system (is that better or worse?) but it lets us collaborate on tickets, make sure tickets don't get lost, and so on, just like a good ticketing system should. Except without having to learn yet another new UI.

(Uh, just because we have a good ticketing system doesn't mean we can actually keep up with emails some days. Sorry. We try. Life at a startup is exciting.)

Videoconferencing

In a remote company, meetings are essential. There are all kinds of subtle issues that affect the way humans interact on the call. This is the area where we experimented the most; unfortunately, although videoconferencing has come very very very far in the last 10 years, there is still no perfect answer.

Let's enumerate some imperfect answers, in vaguely chronological order:

Webex: included only for completenes, because it's a total tire fire. Nobody who has honestly reconsidered their conferencing system in 10+ years would choose it. If you have a subscription to Webex, cancel it right now. Your employees, suppliers, and customers will hold a festival in your honour.
Skype: still works surprisingly okay. Their original peer-to-peer network (now deleted) inspired Tailscale's VPN mesh. But Microsoft wants you to use Skype for Business instead, so it's hard to send people links to prescheduled group meetings. You can see the writing on the wall, might as well not even start.
Skype for Business / Microsoft Teams: seems okay actually, but you have to buy into some whole Microsoft Teams Ecosystem to get started. Your meeting invitees don't have to buy in, but it sure makes it look like they do, which makes a bad first impression. Forget it.
Hangouts: permanently deprecated but somehow still not dead. They deserve some credit for being (I think) the first videoconferencing app that didn't require a browser plugin or standalone app. To be fair, this is because they make the browser and the browser "coincidentally" now has videoconferencing APIs (webrtc) in it. Ironically however, Hangouts is one of the least effective users of webrtc and has numerous bugs.

One of my favourite bugs that has been around for 5+ years: if you sit on a call for a few minutes waiting for someone else to arrive, there will often be a completely unreasonable amount of lag+echo when they do. This will go away if you close the window and reconnect. But don't go away too long or it'll happen to the other person! Plus, the best advice to get rid of echo is "clap your hands to retrain the echo canceller!" which, while it works, is not necessary with anybody else's tool.

Hangouts has some good bits that make it sticky and/or popular. First, it's free! Second, it supports dial-in phone numbers for people who have technical problems. Third, it's highly integrated, not to say bundled, with Calendar, to the point where it obnoxiously auto-schedules a Hangouts meeting id even for your team lunch.

Unfortunately, Hangouts has perenially bad lag (sometimes several seconds), huge problems with echo, a video codec that kills the battery on several popular kinds of computers, and a pretty bad screen layout algorithm that only shows the current speaker in one giant window, which makes it hard to follow the facial expressions of all your teammates.
Whereby: formerly Appear.in. Let's be honest, Appear.in was a much better name, but apparently some other completely unrelated company in Norway had trademarked the name "Appear" (can you even trademark that? Maybe in Norway) so here we are. Anyway, someone recommended them to me a few years ago as the first pure-webrtc conferencing tool that cared about bufferbloat, and that's exactly right: no plugins required, far less bloaty javascript than Hangouts, extremely low latency. And nowadays their layout algorithm is very nice, showing everyone in the call at once and maximizing the space used on the screen for what matters: people's facial expressions.

There are a few downsides. First, their layout algorithm is utterly useless when someone starts screen sharing; it cuts off both sides of the screen and/or makes it tiny unless you fiddle around in sub-submenus. There should be a shortcut or something. Also, they biased so far toward low latency that if your network is at all glitchy or jittery, it's almost game over; they don't recover well in situations where the only right answer is, unfortunately, to introduce enough lag to compensate for the jitter.

They are also remarkably bad at taking your money. We finally started paying them a few weeks ago only because we felt bad for them when we started paying for Zoom (see below), which we like less. (As I write this, Tailscale still has no payment system hooked up to our web site at all, so I sympathize, empathize, hypocritize, and shake my head all at once. Sorry, Whereby; we love you, but please try not to be Dumb Like Us.)

As a result of the glitchiness on high-jitter links and their lack of dial-in phone numbers, unfortunately we don't use Whereby for meetings with outsiders; there's too much risk of a connection failure, wasting 10-15 minutes at the start of a meeting. We do use it for internal team meetings (where it's worth investing in one-time setup to have a long-term great experience).

Another weird limitation is that Whereby meeting rooms really act like "rooms"; there isn't a separate meeting id per meeting. That makes it hard to do simple things like a series of half-hour meetings; the next people wander into a meeting with the previous people. Awkward.
Zoom: Ok, I'm gonna be straight with you, Zoom is obviously what you need to use for your meetings - everybody knows it - and it's also quite bad. Some of the badness comes from what used to be their technical advantage: reputedly, they made a special video codec/plugin that would subtly speed up and slow down the audio and video streams to compensate for network jitter. This is kind of a long story, but I think the idea is rather than just freezing the video when packets got lost and retransmitted, they could slow down the stream during the missing segment, then speed it up again once it resumed, and you wouldn't realize there was a glitch. Pretty clever actually. In fact, I'm guessing this is why they're called "Zoom." Clever, right?

Except, well, it doesn't seem to work. I still get plenty of audio/video glitches, albeit non-fatal ones. What's worse, to make their fancy codec work, they have to have a native plugin or app, because webrtc can't do it. That means the first time you call into Zoom, there's this installation process that installs what turned out to be an accidental spy cam app on your Macbook. Whoops. I mean, they've fixed it now, so that's nice.

The latency is also not great (wasn't lower latency the whole point of the fancy algorithm?). The screen layout algorithm is abysmal, giving you the choice between "one giant screen with one person talking and no indicator when new people join the call" or "each person in a tiny little box and most of the screen pixels are wasted."

But here's the thing: invite links work. Their plugin installer is transparent enough that pretty much everyone succeeds at it (quite a feat!), and for everyone else, they have dial-in phone numbers. They have integrations with everything, including Calendly and Google Calendar and Slack. They sell hardware "Zoom rooms" that let you have futuristic videoconferencing rooms that used to cost 10x as much. (Hangouts tried to do this too. It was going pretty well before they lost focus.) Most of all though, Zoom just works. It doesn't work well, which is sad. But it works. I've never had the "10 minutes futzing with the videoconference connection" at the start of a Zoom call. And they absolutely figured out how to extract big money from you as soon as you start liking it, so they have a business model ensuring they'll stick around. So yeah, this is what we use for calls with people outside our team, although the high latency and bad video quality continue to make me sad.
Honourable mention: FaceTime: The audio/video quality and latency in FaceTime is absolutely, positively, the undeniable best of everything we tested, in both good network conditions (where Whereby is comparable) and bad conditions (where everything else is worse). This probably has to do with them hiring some of the best network and codec people in the world to work on it. (eg. I saw a talk by Stuart Cheshire about how ECN contributes to this.) Unfortunately, the usual Apple limitations make it essentially unusable for anything except calling your parents: it only works on Apple devices. There are no meeting URLs to put in a calendar. It only works on Apple devices. You can't pre-schedule multi-way calls. People can phone you whenever they want (I hate that). And oh, did I mention, it only works on Apple devices.

Short answer: we use Whereby for most internal meetings, and Zoom for externally-facing meetings. We would prefer to use Whereby for everthing, if it gets a bit better.

Videoconferencing hardware

As a fully remote company, we don't have "meeting rooms," so Zoom Rooms are not a thing that makes sense for us. Which is fine, because despite what you might guess, the latency is not better with Zoom hardware than with a general purpose computing device.

We tested a few different setups looking for a good combination of latency, video quality, and reliability. It was definitely not as cool as any of Dan Luu's latency tests, but this is the apenwarr blog, not the danluu blog, and you get what you pay for. Sorry.

What we learned was:

PCs of any sort (Linux, macOS, ChromeOS, Windows) all have higher latency than dedicated iPhone or iPad devices. (We didn't bother testing Android video latency because, well, let's be honest, it's not going to be an improvement.)
Most (but not all) front-facing iPad cameras are not great. They're okay, but not great, especially in low light. If you have a lot of meetings, a bit better video quality is nice to have. The very latest 2019 iPad Pro has a pretty great front-facing camera that works in low light (the best kind of light), so that's what I use now.
Older iPhones (like my aging iPhone 6S) go into CPU throttling with some video codecs, notably Whereby's, so the video quality starts off good but degrades after a few minutes when it gets hot. My new iPad Pro does not have this problem. I think my coworker has an iPhone X and also did not report it.
You absolutely should use some sort of "personal microphone" whenever you do a call. The state of echo cancellation is pretty good now (except in Hangouts), but there's nothing a single screen-mounted microphone can do about ambiant noise. The single best favour you can do for your call partners is to use a personal microphone. Airpods include two personal microphones that work great.
Airpods (when connected to iPhones or iPads) have very low latency, not detectable by humans. They're not any worse than a wired microphone, which is a pretty good technical achievement (one of the goals of Bluetooth 4.x I gather... but goals don't always translate into reality). Because of this ultra-low latency, they do occasionally glitch out when there's a 2.4 GHz noise burst, but it's brief and generally worth the tradeoff just to not have your head wired into your computer.
Warning: Airpods (and all bluetooth devices) have higher and highly variable latency depending what you connect them to. macOS is definitely not perfect about latency (and tends to have worse videoconferencing performance overall, for whatever reason, than an iPad). Windows bluetooth varies from great to absymally terrible, depending mostly on the driver but also the phase of the moon. Linux bluetooth is hahahahaha sorry I forgot what I was going to say.
I mounted my iPad above my monitor, at the so-called "selfie angle," using a $40 iPad mount I bought from Amazon. Mounting it this way has two advantages: I look slightly up at the person I'm talking to rather than down, and when I type notes into my computer, it doesn't look like I'm off to the side. This is aside from the separate benefit of using an iPad for calls: I can have the call visible at all times, without obscuring my computer desktop.

We didn't get all fancy with green screens and pro-quality microphones and all that stuff that other people talk about. Maybe it would be better, I don't know, but it definitely sounds like too much work to dump on every employee.

Short answer: iPad + Airpods + Whereby is a really great combination in 2020. And it also works well with Zoom, which is good because you're stuck with it.

git-subtrac: all your git submodules in one place

Sun, 24 Nov 2019 19:04:49 +0000

Long ago, I wrote git-subtree to work around some of my annoyances with git submodules. I've learned a lot since then, and the development ecosystem has improved a lot (shell scripts are no longer the best way to manipulate git repos? Whoa!).

Thus, I bring you: git-subtrac.

It's a bit like git-subtree, except it uses real git submodules. The difference from plain submodules is that, like git-subtree, it encourages you to put all the contents from all your submodules into your superproject repo, rather than scattering it around across multiple repositories (which might be owned by multiple people, randomly disappear or get rebased, etc).

As a result, it's easy to push, pull, fork, merge, and rebase your entire project no matter how many submodules you like to use. When someone does a 'fetch' of your repo, they get all the submodule repos as well.

I wrote a longer git-subtrac README describing how to use it and its internal workings. I think it's pretty cool. Feedback is welcome.

What do executives do, anyway?

Fri, 15 Mar 2024 04:27:36 +0000

An executive with 8,000 indirect reports and 2000 hours of work in a year can afford to spend, at most, 15 minutes per year per person in their reporting hierarchy... even if they work on nothing else. That job seems impossible. How can anyone make any important decision in a company that large? They will always be the least informed person in the room, no matter what the topic.

If you know me, you know I've been asking myself this question for a long time.

Luckily, someone sent me a link to a really great book, High Output Management, by Andy Grove (of Intel fame). Among many other things, it answers this key question! And insultingly, just to rub it in, it answered this question back in the 1980s.

To paraphrase the book, the job of an executive is: to define and enforce culture and values for their whole organization, and to ratify good decisions.

That's all.

Not to decide. Not to break ties. Not to set strategy. Not to be the expert on every, or any topic. Just to sit in the room while the right people make good decisions in alignment with their values. And if they do, to endorse it. And if they don't, to send them back to try again.

There's even an algorithm for this.

It seems too easy to be real. For any disagreement, identify the lead person on each side. Then, identify the lowest executive in the corporate hierarchy that both leads report into (in the extreme case, this is the CEO). Set up a meeting between the three of them. At the meeting, the two leads will present the one, correct decision that they have agreed upon. The executive will sit there, listen, and ratify it.

But... wait. If the decision is already made before the meeting, why do we need the meeting? Because the right decision might not happen without the existence of that meeting. The executive gives formal weight to a major decision. The executive holds the two disagreeing leads responsible: they must figure out not what's best for them, but what's best for the company. They can't pull rank. They can't cheat. They have to present their answer to a person who cares about both of their groups equally. And they want to look good, because that person is their boss! This puts a lot of pressure on people to do the right thing.

(Side note: this has parallels with the weirdly formal structures in eg. Canadian parliament, where theoretically all decisions must be ratified by the seemingly powerless Governor General, who represents The Queen by just always ratifying everything. The theory is that if the decisions were bad, they wouldn't be ratified, so there'd be no point proposing them, and therefore all the decisions proposed are worthy of ratification. Obviously the theory doesn't match the practice here, because bad decisions get ratified, but it's nice to think about.)

Failure modes

What happens when an executive doesn't follow this model? One of several things we've all seen before, depending what the executive does instead.

If the executive makes their own decisions and forces them downstream: the executive doesn't have enough information to make good decisions in detail, so the decision won't be optimal. And there won't be much buy-in from people downstream. This also encourages politics: people whisper in the executive's ear to bend it one way or the other. It encourages "brown-nosing."
If the executive chooses not to be involved in conflicts that are "not important enough; you figure it out": political power games ensue. Whoever can force their way will win, killing morale. Or half the people do one thing and half do the other, and the company loses focus.
If the executive accepts escalations, then tries to make a tie-breaker decision: non-optimal decisions get made, because again the executive is, out of the three people, the least qualified to decide. Offhand, you might think this is fine, if the decision isn't very important anyway. That part is true. But the indirect effects are disastrous: it allows the two leads to abdicate responsibility. They don't have to remind themselves what's good for the company, because you did it for them. It lets them be selfish. It lets disagreement fester. It leaves at least one side not fully bought in.

(I'm wary of "disagree and commit" for this reason. Real people don't commit when they strongly disagree; they only pretend to. In service of a value like "move fast and break things" it can work, because speed overrides wisdom or consistency. That's a legitimate value, like any other, if it serves your strategy.)
If the executive brings in more people to discuss the issue: this is something the two leads should have done already. If they didn't, they are failing at their job, and need to learn how to do it better. Step one is the executive sends them a message: "Go back. Include these additional people/groups in your decision. Come back when you've thought it through properly." If it continues, people have to get fired, because they are bad at making decisions.

Enforcement of culture and values

According to the book, which makes a pretty compelling case, the only other responsibility of an executive is to enforce company values.

What does that mean? It means if someone in the company isn't acting "right" - not acting ethically, not following the conflict resolution algorithm above, playing politics - then they need to be corrected or removed. Every executive is responsible for enforcing the policy all the way down the chain, recursively. And the CEO is responsible for everyone. You have to squash violators of company values, fast, because violators are dangerous. People who don't share your values will hire more people who don't share your values. It's all downhill from there.

Real values aren't what you talk about, they're what you do when times get tough. That means values are most visible during big, controversial decisions. The executive ratifying a decision needs to evaluate that decision against the set of organizational values. Do the two leads both understand our values? Is the decision in line with our values? If not, tell them so, explicitly, and send them back to try again.

What about strategy?

One of the book's claims, which I found shocking at first, was that in a large organization, executives don't set strategy. Not even the CEO sets strategy. Why? Because it's an illusion to believe you can enforce a strategy.

Employees, including executives that report to you, follow company values first and foremost. (This is by ~~definition~~ construction. If they don't, you fired them, see above.) Of course, they're human, so as part of that, they'll be looking out for themselves, their friends, and the people in their organization.

Maybe one of your organizational values is "do what your boss says." That's a thing you can do, and you can enforce. It works. The military works like that supposedly (although I have no experience with the military). But command-and-control is not very efficient for knowledge workers, because of the fundamental problem that for any given situation, the people who know the most about it are the people at the bottom, not the people at the top.

If the people at the bottom can't agree what to do, then great! That's why we have a hierarchy. Use the decision process above until the answer is obvious.

But if the person at the top is trying to "set a strategy" by making operational decisions, those decisions will be based on insufficient facts, because there are simply far too many facts for one person. That means, if your decisions should be based on facts, you will make worse decisions than your subordinates. That's scary.

So what, then? A company just drifts in the void, with no strategy?

Not exactly. It's harder than that. What executives need to do is come up with organizational values that indirectly result in the strategy they want.

That is, if your company makes widgets and one of your values is customer satisfaction, you will probably end up with better widgets of the right sort for your existing customers. If one of your values is to be environmentally friendly, your widget factories will probably pollute less but cost more. If one of your values is to make the tools that run faster and smoother, your employees will probably make less bloatware and you'll probably hire different employees than if your values are to scale fast and capture the most customers in the shortest time.

Why will employees embrace whatever weird organizational values you set? Because in every decision meeting, you enforce your values. And you fire the people who don't line up. Recursively, that means executives lower down the tree will do the same, because that itself is one of the values you enforce.

Unless it's somehow impossible to hire people who agree with your values, you can assemble an organization that aligns with them. It might be a terrible organization that ruins your business, but then... well, those values weren't a good choice.

I can't believe nobody told me this before. It's all so simple, and it's all been documented since the 1980s.

Epilogue: small companies

Almost none of this applies to small companies. They are so small that the founders and the CEO actually do have a chance of fully understanding problems, which means they don't yet need to delegate decisions. Also, in a small company, strategy and values are usually not well defined yet, so a primary goal is to discover them incrementally. You learn from mistakes and refine together until the strategy (and thus the values that will produce the strategy) become clear.

In a small company, it's important to understand how the big company process works, because your values begin to solidify pretty early on, even as you choose co-founders and investors and hire the first employees. It's hard to change your values later, because it usually involves firing people. So you need to be thinking about them from the beginning. Still, the details aren't set in stone on day 1.

Doubilogue: major strategic changes

All this is one reason why if you want a major strategic change, you often replace an executive - maybe even the CEO. Or, conversely, if you replace the CEO, you often get a major strategic change, whether you like it or not. The CEO sets the values, and the values set the strategy.

Company values flow downward. They are very hard to change, and very painful. When you change your company values, you might find that employees who liked the old values don't want to work there anymore, and rightly so. (This happens even if the new values are "better" in your favourite dimensions.)

If your old strategy is failing, you can't fix the company by just declaring a new strategy. You do it by declaring new values. Then you enforce those values. And that's going to make a lot of people very upset. (If you do this too often, you deserve what you get.)

One reason strategy changes are so risky in a big company is that, again, the people at the top really don't know much of what's going on. Although they have a sky-high view of the world, they have a very limited view of the details. Changes of strategy, and therefore changes of values, and therefore changes of executives, usually have wide-ranging unexpected consequences. You do it because you have to, because your old strategy isn't working, not because you want to. You're betting everything.

I wish more executives would be transparent about this. "Our old strategy wasn't working, because our old values weren't working. Here's the new strategy, and the new values. This is gonna hurt."

What you usually get instead is a polite "rewording" or "watering down" of the corporate values, and maybe some whispering about how the old values weren't so good after all, and maybe how the new values were our real values all along. Weak.

Or, worst of all, executives lose their way and stop enforcing any value system at all. Then the value system reverts to the default: politics and backstabbing. It wouldn't bother me so much if it weren't so hopelessly inefficient.

Tripilogue: governments

Governmental politics are bad exactly to the extent that we don't enforce our values by firing the people who don't encompass them.

In a democracy, this is hard because values in the first place are agreed by mass consensus rather than chosen at the top. That's why propaganda is so powerful: it changes our values, which changes who and what we tolerate.

Quadrilogue: Tradeoffs

By the way, useful organizational values come in the form of tradeoffs: giving up one nice thing in order to get some other nice thing. Wishy-washy values like "respect your co-workers" aren't really values, because nobody would ever pick a value like "don't respect your co-workers." Respecting your co-workers is just basic civility. By the time you have to write it down, you've already lost. Put it in your HR policy somewhere, not the top line.

A real value is something like "tell the truth, even when it hurts." Or "deliver the software on schedule, even if there are bugs." In both cases, one can legitimately imagine valuing the opposite.

Absolute scale corrupts absolutely

Wed, 17 Nov 2021 22:49:50 +0000

The Internet has gotten too big.

Growing up, I, like many computery people of my generation, was an idealist. I believed that better, faster communication would be an unmitigated improvement to society. "World peace through better communication," I said to an older co-worker, once, as the millenium was coming to an end. "If people could just understand each others' points of view, there would be no reason for them to fight. Government propaganda will never work if citizens of two warring countries can just talk to each other and realize that the other side is human, just like them, and teach each other what's really true."

[Wired.com has an excellent article about this sort of belief system.]

"You have a lot to learn about the world," he said.

Or maybe he said, "That's the most naive thing I've ever heard in my entire life." I can't remember exactly. Either or both would have been appropriate, as it turns out.

What actually happened

There's a pattern that I don't see talked about much, but which seems to apply in all sorts of systems, big and small, theoretical and practical, mathematical and physical.

The pattern is: the cheaper interactions become, the more intensely a system is corrupted. The faster interactions become, the faster the corruption spreads.

What is "corruption?" I don't know the right technical definition, but you know it when you see it. In a living system, it's a virus, or a cancer, or a predator. In a computer operating system, it's malware. In password authentication, it's phishing attacks. In politics, it's lobbyists and grift. In finance, it's high-frequency trading and credit default swaps. In Twitter, it's propaganda bots. In earth's orbit, it's space debris and Kessler syndrome.

On the Internet, it's botnets and DDoS attacks.

What do all these things have in common? That they didn't happen when the system was small and expensive. The system was just as vulnerable back then - usually even more so - but it wasn't worth it. The corruption didn't corrupt. The attacks didn't evolve.

What do I mean by evolve? Again, I don't know what technical definition to use. The evolutionary process of bacteria isn't the same as the evolutionary process of a spam or phishing attack, or malware, or foreign-sponsored Twitter propaganda, or space junk in low-earth orbit. Biological infections evolve, we think, by random mutation and imperfect natural selection. But malware evolves when people redesign it. And space junk can turn violent because of accidental collisions; not natural, and yet the exact opposite of careful human design.

Whatever the cause, the corruption happens in all those cases. Intelligent human attackers are only one way to create a new corruption, but they're a fast, persistent one. The more humans you connect, the more kinds of corruption they will generate.

Most humans aren't trying to create corruption. But it doesn't matter, if a rare corruption can spread quickly. A larger, faster, standardized network lets the same attack offer bigger benefits to the attacker, without increasing cost.

Diversity

One of the natural defenses against corruption is diversity. There's this problem right now where supposedly the most common strain of bananas is dying out because they are all genetically identical, so the wrong fungus at the right time can kill them all. One way to limit the damage would be to grow, say, 10 kinds of bananas; then when there's a banana plague, it'll only kill, say, 10% of your crop, which you can replace over the next few years.

That might work okay for bananas, but for human diseases, you wouldn't want to be one of the unlucky 10%. For computer viruses, maybe we can have 10 operating systems, but you still don't want to be the unlucky one, and you also don't want to be stuck with the 10th best operating system or the 10th best browser. Diversity is how nature defends against corruption, but not how human engineers do.

In fact, a major goal of modern engineering is to destroy diversity. As Deming would say, reduce variation. Find the "best" solution, then deploy it consistently everywhere, and keep improving it.

When we read about adversarial attacks on computer vision, why are they worse than Magic Eye drawings or other human optical illusions? Because they can be perfectly targeted. An optical illusion street sign would only fool a subset of humans, only some of the time, because each of our neural nets is configured differently from everyone else's. But every neural net in every car of a particular brand and model will be susceptible to exactly the same illusion. You can take a perfect copy of the AI, bring it into your lab, and design a perfect illusion that fools it. Subtracting natural diversity has turned a boring visual attack into a disproportionately effective one.

The same attacks work against a search engine or an email spam filter. If you get a copy of the algorithm, or even query it quickly enough and transparently enough in a black box test, you can design a message to defeat it. That's the SEO industry and email newsletter industry, in a nutshell. It's why companies don't want to tell you which clause of their unevenly-enforced terms of service you violated; because if you knew, you'd fine tune your behaviour to be evil, but not quite evil enough to trip over the line.

It's why human moderators still work better than computer moderators: because humans make unpredictable mistakes. It's harder to optimize an attack against rules that won't stay constant.

...but back to the Internet

I hope you didn't think I was going to tell you how to fix Twitter and Facebook and U.S. politics. The truth is, I have no idea at all. I just see that the patterns of corruption are the same. Nobody bothered to corrupt Twitter and Facebook until they got influential enough to matter, and then everybody bothered to corrupt them, and we have no defense. Start filtering out bots - which of course you must do - and people will build better bots, just like they did with email spam and auto-generated web content and CAPTCHA solvers. You're not fighting against AI, you're fighting against I, and the I is highly incentivized by lots and lots of money and power.

But, ok, wait. I don't know how to fix giant social networks. But I do know a general workaround to this whole class of problem: slow things down. Choose carefully who you interact with. Interact with fewer people. Make sure you are certain which people they are.

If that sounds like some religions' advice about sex, it's probably not a coincidence. It's also why you shouldn't allow foreigners to buy political ads in your country. And why local newspapers are better than national ones. And why "free trade" goes so bad, so often, even though it's also often good. And why doctors need to wash their hands a lot. (Hospital staff are like the Internet of Bacteria.)

Unfortunately, this general workaround translates into "smash Facebook" or "stop letting strangers interact on Twitter," which is not very effective because a) it's not gonna happen, and b) it would destroy lots of useful interactions. So like I said, I've got nothing for you there. Sorry. Big networks are hard.

But Avery, the Internet, you said

Oh right. Me and my cofounders at Tailscale have been thinking about a particular formulation of this problem. Let's forget about Internet Scale problems (like giant social networks) for a moment. The thing is, only very few problems are Internet Scale. That's what makes them newsworthy. I hate to be the bearer of bad news, but chances are, your problems are not Internet Scale.

Why is it so hard to launch a simple web service for, say, just your customers or employees? Why did the Equifax breach happen, when obviously no outsiders at all were supposed to have access to Equifax's data? How did the Capital One + AWS hack happen, when Capital One clearly has decades of experience with not leaking your data all over the place?

I'll claim it again... because the Internet is too big.

Equifax's data was reachable from the Internet even though it should have only been accessible to a few support employees. Capital One's data surely used to be behind layers and layers of misconfigured firewalls, unhelpful proxy servers, and maybe even pre-TCP/IP legacy mainframe protocols, but then they moved it to AWS, eliminating that diversity and those ad-hoc layers of protection. Nobody can say modernizing their systems was the wrong choice, and yet the result was the same result we always get when we lose diversity.

AWS is bananas, and AWS permission bug exploits are banana fungus.

Attackers perfect their attack once, try it everywhere, scale it like crazy.

Back in the 1990s, I worked with super dumb database apps running on LANs. They stored their passwords in plaintext, in files readable by everyone. But there was never a (digital) attack on the scale of 2019 Capital One. Why?

Because... there was no Internet. Well, there was, but we weren't on it. Employees, with a bit of tech skill, could easily attack the database, and surely some got into some white collar crime. And you occasionally heard stories of kids "hacking into" the school's grading system and giving themselves an A. I even made fake accounts on a BBS or two. But random people in foreign countries didn't hack into your database. And the kids didn't give A's to millions of other kids in all the other schools. It wasn't a thing. Each corruption was contained.

Here's what we've lost sight of, in a world where everything is Internet scale: most interactions should not be Internet scale. Most instances of most programs should be restricted to a small set of obviously trusted people. All those people, in all those foreign countries, should not be invited to read Equifax's PII database in Argentina, no matter how stupid the password was. They shouldn't even be able to connect to the database. They shouldn't be able to see that it exists.

It shouldn't, in short, be on the Internet.

On the other hand, properly authorized users, who are on the Internet, would like to be able to reach it from anywhere. Because requiring all the employees to come to an office location to do their jobs ("physical security") seems kinda obsolete.

That leaves us with a conundrum, doesn't it?

Wouldn't it be nice though? If you could have servers, like you did in the 1990s, with the same simple architectures as you used in the 1990s, and the same ~~sloppy security policies~~ developer freedom as you had in the 1990s, but somehow reach them from anywhere? Like... a network, but not the Internet. One that isn't reachable from the Internet, or even addressable on the Internet. One that uses the Internet as a substrate, but not as a banana.

That's what we're working on.

Literary Afterthoughts

I'm certainly not the first to bring up all this. Various sci-fi addresses the problem of system corruption due to excess connectivity. I liked A Fire Upon the Deep by Vernor Vinge, where some parts of the universe have much better connectivity than others and it doesn't go well at all. There's also the Rifters Trilogy by Peter Watts, in which the Internet of their time is nearly unusable because it's cluttered with machine-generated garbage.

Still, I'd be interested in hearing about any "real science" on the general topic of systems corruption at large scales with higher connectivity. Is there math for this? Can we predict the point at which it all falls apart? Does this define an upper limit on the potential for hyperintelligence? Will this prevent the technological singularity?

Logistical note

I'm normally an East Coast person, but I'll be visiting the San Francisco Bay area from August 26-30 to catch up with friends and talk to people about Tailscale, the horrors of IPv6, etc. Feel free to contact me if you're around and would like to meet up.

Programmer migration patterns

Mon, 18 Mar 2019 17:21:18 +0000

I made a little flow chart of mainstream programming languages and how programmers seem to move from one to another.

There's a more common kind of chart, which shows how the languages themselves evolved. I didn't want to show the point of view of language inventors, but rather language users, and see what came out. It looks similar, but not quite the same.

If you started out in language A, this shows which language(s) you most likely jumped to next. According to me. Which is not very scientific, but if you wanted science, you wouldn't be here, right? Perhaps this flow chart says more about me than it says about you.

Disclaimers: Yes, I forgot your favourite language. Yes, people can jump from any language to any other language. Yes, you can learn multiple languages and use the right one for the job. Yes, I have biases everywhere.

With that out of the way, I'll write the rest of this post with fewer disclaimers and state my opinions as if they were facts, for better readability.

I highlighted what are currently the most common "terminal nodes" - where people stop because they can't find anything better, in the dimensions they're looking for. The terminal nodes are: Rust, Java, Go, Python 3, Javascript, and (honourable mention because it's kinda just Javascript) node.js.

A few years ago, I also would have highlighted C as a terminal node. Maybe I still should, because there are plenty of major projects (eg. OS kernels) that still use it and which don't see themselves as having any realistic alternative. But the cracks are definitely showing. My favourite example is Fun with NULL pointers, in which the Linux kernel had a vulnerability caused by the compiler helpfully stripping out NULL pointer checks from a function because NULL values were "impossible." C is a mess, and the spec makes several critical mistakes that newer languages don't make. Maybe they'll fix the spec someday, though.

But let's go back a few steps. If we start at the top, you can see four main branches, corresponding to specializations where people seem to get into programming:

"Low level" programming, including asm and C.
"Business" or "learning" programming, starting with BASIC.
Numerical/scientific programming, such as Fortran, MATLAB, and R.
Scripting/glue programming, like shell (sh) and perl.

(We could maybe also talk about "database query languages" like SQL, except there's really only SQL, to my great dismay. Every attempt to replace it has failed. Database languages are stuck in the 1960s. They even still CAPITALIZE their KEYWORDS because (they THINK) that MAKES it EASIER to UNDERSTAND the CODE.)

(I also left out HTML and CSS. Sorry. They are real languages, but everybody has to learn them now, so there was nowhere to put the arrows. I also omitted the Lisp family, because it never really got popular, although a certain subgroup of people always wished it would. And I would have had to add a fifth category of programmer specialization, "configuring emacs.")

(And I skipped Haskell, because... well, I considered just depicting it as a box floating off to the side, with no arrows into or out of the box, but I figured that would be extraneous. It was a great meta-joke though, because Haskell precludes the concept of I/O unless you involve Monads.)

Anyway, let's go back to the 1990s, and pretend that the world was simple, and (1) low level programmers used C or asm or Turbo Pascal, (2) business programmers used VB, (3) Numerical programmers used Fortran or R or MATLAB, and (4) Glue programmers used sh or perl.

Back then, programming languages were kind of regimented like that. I hadn't really thought about it until I drew the chart. But you pretty obviously didn't write operating system kernels in perl, or glue in MATLAB, or giant matrix multiplications in visual basic.

How things have changed! Because you still don't, but now it's not obvious.

Language migration is mostly about style

Let's look at the section of the tree starting with asm (assembly language). Asm is an incredibly painful way to write programs, although to this day, it is still the best way to write certain things (the first few instructions after your computer boots, for example, or the entry code for an interrupt handler). Every compiled language compiles down to assembly, or machine language, eventually, one way or another, even if that happens somewhere inside the App Store or in a JIT running on your phone.

The first thing that happened, when we abstracted beyond asm, was a fork into two branches: the C-like branch and the Pascal-like branch. (Yes, Algol came before these, but let's skip it. Not many people would identify themselves as Algol programmers. It mostly influenced other languages.)

You can tell the Pascal-like branch because it has "begin...end". You can tell the C-like branch because it uses braces. C, of course, influenced the design of many languages in ways not shown in my chart. Because we're talking about programmers, not language designers.

Let's look at C first. Oddly enough, once people got started in C, they started using it for all kinds of stuff: it was one of the few languages where you could, whether or not it was a good idea, legitimately implement all four categories of programming problem. All of them were a bit painful (except low-level programming, which is what C is actually good at), but it was all possible, and it all ran at a decent speed.

But if you're a C programmer, where do you go next? It depends what you were using it for.

C++ was the obvious choice, but C++, despite its name and syntax, is philosophically not very C-like. Unless you're BeOS, you don't write operating system kernels in C++. The operating systems people stuck with C, at least until Rust arrived, which looks like it has some real potential.

But the business ("large programs") and numerical ("fast programs") people liked C++. Okay, for many, "liked" is not the right word, but they stuck with it, while there was nothing better.

For glue, many people jumped straight from C (or maybe C++) to python 2. I certainly did. Python 2, unlike the weirdness of perl, is a familiar, C-like language, with even simpler syntax. It's easy for a C programmer to understand how python C modules work (and to write a new python module). Calling a C function from python is cheaper than in other languages, such as Java, where you have to fight with a non-refcounting garbage collector. The python "os" module just gives you C system calls, the way C system calls work. You can get access to C's errno and install signal handlers. The only problem is python is, well, slow. But if you treat it as a glue language, you don't care about python's slowness; you write C modules or call C libraries or subprocesses when it's slow.

Separately, when Java came out, many C and C++ "business software" programmers were quick to jump to it. Java ran really slow (although unlike python, it was advertised as "theoretically fast"), but people happily paid the slowness price to get rid of C++'s long compile times, header file madness, portability inconveniences, and use-after-free errors.

I recall reading somewhere that the inventors of Go originally thought that Go would be a competitor for Java or C++, but that didn't really work out. Java is like that famous hotel, also probably from Menlo Park, where once you check in, you never check out. Meanwhile, people who still hadn't jumped from C++ to Java were not likely to jump to another language that a) also ran somewhat slower than C++ and b) also had garbage collection, a religious issue.

Where Go did become popular was with all those glue coders who had previously jumped to python 2. It turns out python's slowness was kind of a pain after all. And as computers get more and more insanely complicated, python glue programs tend to get big, and then the dynamic typing starts to bring more trouble than value, and pre-compiling your binaries starts to pay off. And python 2 uses plenty of memory, so Go gives a RAM improvement, not a detriment like when you move from C++. Go isn't much harder to write than python, but it runs faster and with (usually, somewhat) less RAM.

Nowadays we call Go a "systems" language because "glue" languages remind us too much of perl and ruby, but it's all the same job. (Try telling a kernel developer who uses C that Go is a "systems" language and see what they say.) It's glue. You glue together components to make a system.

The Hejlsberg factor

Let's look next at the Visual Basic and Pascal branches, because there's a weird alternate reality that you either find obviously right ("Why would I ever use something as painful as C or Java?") or obviously wrong ("Visual... Basic? Are you serious?")

Back in the 1980s and 1990s, some people still believed that programming should be approachable to new programmers, so personal computers arrived with a pre-installed programming language for free, almost always BASIC.

In contrast, when universities taught programming, they shunned BASIC ("It is practically impossible to teach good programming to students that have had a prior exposure to BASIC"), but also shunned C. They favoured Pascal, which was considered reasonably easy to learn, looked like all those historical Algol academic papers, and whose syntax could be used to teach a class about parsers without having to fail most of your students. So you had the academic branch and the personal computing branch, but what they had in common is that neither of them liked C.

BASIC on PCs (on DOS) eventually became Visual Basic on Windows, which until javascript came along was probably the most-used and most-loved programming language ever. (It is still the "macro" language used in Excel. There are a lot of Excel programmers, although most of them don't think they're programmers.)

Meanwhile, Pascal managed to migrate to PCs and get popular, mainly thanks to Turbo Pascal, which was probably the fastest compiler ever, by a large margin. They weren't kidding about the Turbo. They even got some C programmers to use it despite preferring C's syntax, just because it was so fast. (Turbo C was okay, but not nearly as Turbo. Faster than everyone else's C compiler, though.)

(Pascal in universities got more and more academic and later evolved into Modula and Ada. That branch would have probably died out if it weren't for the US military adopting Ada for high-reliability systems. Let's ignore Ada for today.)

At that point in history, we had two main branches of "business" developers: the BASIC branch and the Pascal branch. And now Windows was released, and Visual Basic. Turbo Pascal for DOS was looking a bit old. Turbo Pascal for Windows was not super compelling. In order to compete, the inventor of Turbo Pascal, Anders Hejlsberg, created Delphi, a visual environment like Visual Basic, but based on the Turbo Pascal language instead, and with fewer execrable always-missing-or-incompatible-dammit runtime DLLs.

It was really good, but it wasn't Microsoft, so business wise, things got tough. In an unexpected turn of events, eventually Hejlsberg ended up working at Microsoft, where he proceeded to invent the C# language, which launched the Microsoft .NET platform, which also had a Visual Basic .NET variant (which was terrible). This unified the two branches. Supposedly.

Unfortunately, as mentioned, VB.NET was terrible. It was almost nothing like Visual Basic; it was more like a slower version of C++, but with a skin of not-quite-Basic syntax on top, and a much worse UI design tool. C# also wasn't Delphi. But all those things were dead, and Microsoft pushed really hard to make sure they stayed that way. (Except Microsoft Office, which to this day still uses the original Visual Basic syntax, which they call "Visual Basic for Applications," or VBA. It might be more commonly used, and certainly more loved by its users, than all of .NET ever was.)

I actually don't know what became of Visual Basic programmers. Microsoft shoved them pretty hard to get them onto VB.NET, but most of them didn't go along. I wanted to draw the "where they really went" arrow in my diagram, but I honestly don't know. Perhaps they became web developers? Or maybe they write Excel macros.

I think it's interesting that nowadays, if you write software for Windows using Microsoft's preferred .NET-based platforms, you are probably using a language that was heavily influenced by Hejlsberg, whose languages were killed by Microsoft and Visual Basic before he killed them back.

Then he went on to write Typescript, but let's not get ahead of ourselves.

A brief history of glue languages

The original glue language was the Unix shell, famous because it introduced the concept of "pipelines" that interconnect small, simple tools to do something complicated.

Ah, those were the days.

Those days are dead and gone
and the eulogy was delivered by Perl.
-- Rob Pike

It turns out to be hard to design small, simple tools, and mostly we don't have enough time for that. So languages which let you skip the small simple tools and instead write a twisted, gluey mess have become much more popular. (It doesn't help that sh syntax is also very flawed, especially around quoting and wildcard expansion rules.)

First came awk, which was a C-syntax-looking parser language that you could use in a shell pipeline. It was a little weird (at the time) to use a mini-language (awk) inside another language (sh) all in one "line" of text, but we got over it, which is a good thing because that's how the web works all the time now. (Let's skip over csh, which was yet another incompatible C-syntax-looking language, with different fatal flaws, that could be used instead of sh.)

Perl came next, "inspired" by awk, because awk didn't have enough punctuation marks. (Okay, just kidding. Kind of.)

Perl made it all the way to perl 5 with ever-growing popularity, then completely dropped the ball when they decided to stop improving the syntax in order to throw it all away and start from scratch with perl 6. (Perl 6 is not shown in my diagram because nobody ever migrated to it.)

This left room for the job of "glue" to fracture in several directions. If you thought perl syntax was ugly, you probably switched to python. If you thought perl syntax was amazing and powerful and just needed some tweaks, you probably switched to ruby. If you were using perl to run web CGI scripts, well, maybe you kept doing that, or maybe you gave up and switched to this new PHP thing.

It didn't take long for ruby to also grow web server support (and then Ruby on Rails). Python evolved that way too.

It's kind of interesting what happened here: a whole generation of programmers abandoned the command line - the place where glue programs used to run - and wanted to do everything on the web instead. In some ways, it's better, because for example you can hyperlink from one glue program to the next. In other ways it's worse, because all these modern web programs are slow and unscriptable and take 500MB of RAM because you have to install yet another copy of Electron and... well, I guess that brings us to the web.

Web languages

You will probably be unsurprised to see that my chart has pretty much everything in the whole "glue" branch converging on javascript. Javascript was originally considered a frontend-only language, but when node.js appeared, that changed forever. Now you can learn just one language and write frontends and backends and command-line tools. Javascript was designed to be the ultimate glue language, somehow tying together HTML, CSS, object-orientation, functional programming, dynamic languages, JITs, and every other thing you could make it talk to through an HTTP request.

But it's ugly. The emphasis on backward compatibility, which has been essential to the success of the web, also prevents people from fixing its worst flaws. Javascript was famously thrown together in 10 days in 1995. It's really excellent for 10 days of work, but there were also some mistakes, and we can't fix them.

This brings us to the only bi-directional arrow in my chart: from javascript to python 3, and back again. Let's call it the yin-yang of scripting languages.

Most of the other historical glue+web languages are fading away, but not python. At least not yet. I think that's because... it's sane. If you program long enough in javascript, the insanity just starts to get to you after a while. Maybe you need a pressure release valve and you switch to python.

Meanwhile, if you program in python long enough, eventually you're going to need to write a web app, and then it's super annoying that your frontend code is in a completely different language than the backend, with completely different quirks, where in one of them you say ['a','b','c'].join(',') and in the other you say ','.join(['a','b','c']) and you can never quite remember which is which.

One of them has a JIT that makes it run fast once it's started, but one of them starts fast and runs slow.

One of them has a sane namespace system, and the other one... well. Doesn't.

I don't think python 3 can possibly beat javascript in the long run, but it's not obvious it'll lose, either.

Meanwhile, Hejlsberg, never quite satisfied with his alternate reality branch of programming, saw the many problems with javascript and introduced TypeScript. Also meanwhile, Microsoft has suddenly stopped being so pushy about native Windows apps and started endorsing the web and open source in a big way. This means that for the first time, Microsoft is shoving its own developers toward web languages, which means javascript. They have their TypeScript spin on it (which is a very nice language, in my opinion), but that branch, an alternate reality for decades now, is finally converging. It probably won't be long before it ends.

Will TypeScript actually win out over pure javascript? Interesting question. I don't know. It's pretty great. But I've bet on Hejlsberg languages before, and I always lose.

Epilogue: Python 2 vs Python 3

With all that said, now I can finally make a point about python 2 vs 3. They are very similar languages, yet somehow not the same. In my opinion, that's because they occupy totally different spots in this whole programmer migration chart.

Python 2 developers came from a world of C and perl, and wanted to write glue code. Web servers were an afterthought, added later. I mean, the web got popular after python 2 came out, so that's hardly a surprise. And a lot of python 2 developers end up switching to Go, because the kind of "systems glue" code they want to write is something Go is suited for.

Python 3 developers come from a different place. It turns out that python usage has grown a lot since python 3 started, but the new people are different from the old people. A surprisingly large fraction of the new people come from the scientific and numerical processing world, because of modules like SciPy and then Tensorflow. Python is honestly a pretty weird choice for high-throughput numerical processing, but whatever, those libraries exist, so that's where we go. Another triumph of python's easy integration with C modules, I guess. And python 3 is also made with the web in mind, of course.

To understand the difference in audience between python 2 and 3, you only need to look at the different string types. In python 2, strings were a series of bytes, because operating systems deal in bytes. Unix pipelines deal in bytes. Network sockets deal in bytes. It was a glue language for systems programs, and glue languages deal in bytes.

In python 3, strings are a series of unicode characters, because people kept screwing up the unicode conversions... when interacting with the web, where everything is unicode. People doing scientific numerical calculations don't care much about strings, and people doing web programming care a lot about unicode, so it uses unicode. Try to write systems programs in python 3, though, and you'll find yourself constantly screwing up the unicode conversions, even in simple things like filenames. What goes around, comes around.

The log/event processing pipeline you can't have

Wed, 17 Nov 2021 22:49:42 +0000

Let me tell you about the still-not-defunct real-time log processing pipeline we built at my now-defunct last job. It handled logs from a large number of embedded devices that our ISP operated on behalf of residential customers. (I wrote and presented previously about some of the cool wifi diagnostics that were possible with this data set.)

Lately, I've had a surprisingly large number of conversations about logs processing pipelines. I can find probably 10+ already-funded, seemingly successful startups processing logs, and the Big Name Cloud providers all have some kind of logs thingy, but still, people are not satisfied. It's expensive and slow. And if you complain, you mostly get told that you shouldn't be using unstructured logs anyway, you should be using event streams.

That advice is not wrong, but it's incomplete.

Instead of doing a survey of the whole unhappy landscape, let's just ignore what other people suffer with and talk about what does work. You can probably find, somewhere, something similar to each of the components I'm going to talk about, but you probably can't find a single solution that combines it all with good performance and super-low latency for a reasonable price. At least, I haven't found it. I was a little surprised by this, because I didn't think we were doing anything all that innovative. Apparently I was incorrect.

The big picture

Let's get started. Here's a handy diagram of all the parts we're going to talk about:

The ISP where I worked has a bunch of embedded Linux devices (routers, firewalls, wifi access points, and so on) that we wanted to monitor. The number increased rapidly over time, but let's talk about a nice round number, like 100,000 of them. Initially there were zero, then maybe 10 in our development lab, and eventually we hit 100,000, and later there were many more than that. Whatever. Let's work with 100,000. But keep in mind that this architecture works pretty much the same with any number of devices.

(It's a "distributed system" in the sense of scalability, but it's also the simplest thing that really works for any number of devices more than a handful, which makes it different from many "distributed systems" where you could have solved the problem much more simply if you didn't care about scaling. Since our logs are coming from multiple sources, we can't make it non-distributed, but we can try to minimize the number of parts that have to deal with the extra complexity.)

Now, these are devices we were monitoring, not apps or services or containers or whatever. That means two things: we had to deal with lots of weird problems (like compiler/kernel bugs and hardware failures), and most of the software was off-the-shelf OS stuff we couldn't easily control (or didn't want to rewrite).

(Here's the good news: because embedded devices have all the problems from top to bottom, any solution that works for my masses of embedded devices will work for any other log-pipeline problem you might have. If you're lucky, you can leave out some parts.)

That means the debate about "events" vs "logs" was kind of moot. We didn't control all the parts in our system, so telling us to forget logs and use only structured events doesn't help. udhcpd produces messages the way it wants to produce messages, and that's life. Sometimes the kernel panics and prints whatever it wants to print, and that's life. Move on.

Of course, we also had our own apps, which means we could also produce our own structured events when it was relevant to our own apps. Our team had whole never-ending debates about which is better, logs or events, structured or unstructured. In fact, in a move only overfunded megacorporations can afford, we actually implemented both and ran them both for a long time.

Thus, I can now tell you the final true answer, once and for all: you want structured events in your database.

...but you need to be able to produce them from unstructured logs. And once you can do that, exactly how those structured events are produced (either from logs or directly from structured trace output) turns out to be unimportant.

But we're getting ahead of ourselves a bit. Let's take our flow diagram, one part at a time, from left to right.

Userspace and kernel messages, in a single stream

Some people who have been hacking on Linux for a while may know about /proc/kmsg: that's the file good old (pre-systemd) klogd reads kernel messages from, and pumps them to syslogd, which saves them to a file. Nowadays systemd does roughly the same thing but with more d-bus and more corrupted binary log files. Ahem. Anyway. When you run the dmesg command, it reads the same kernel messages (in a slightly different way).

What you might not know is that you can go the other direction. There's a file called /dev/kmsg (note: /dev and not /proc) which, if you write to it, produces messages into the kernel's buffer. Let's do that! For all our messages!

Wait, what? Am I crazy? Why do that?

Because we want strict sequencing of log messages between programs. And we want that even if your kernel panics.

Imagine you have, say, a TV DVR running on an embedded Linux system, and whenever you go to play a particular recorded video, the kernel panics because your chipset vendor hates you. Hypothetically. (The feeling is, hypothetically, mutual.) Ideally, you would like your logs to contain a note that the user requested the video, the video is about to start playing, we've opened the file, we're about to start streaming the file to the proprietary and very buggy (hypothetical) video decoder... boom. Panic.

What now? Well, if you're writing the log messages to disk, the joke's on you, because I bet you didn't fsync() after each one. (Once upon a time, syslogd actually did fsync() after each one. It was insanely disk-grindy and had very low throughput. Those days are gone.) Moreover, a kernel panic kills the disk driver, so you have no chance to fsync() it after the panic, unless you engage one of the more terrifying hacks like, after a panic, booting into a secondary kernel whose only job is to stream the message buffer into a file, hoping desperately that the disk driver isn't the thing that panicked, that the disk itself hasn't fried, and that even if you do manage to write to some disk blocks, they are the right ones because your filesystem data structure is reasonably intact.

(I suddenly feel a lot of pity for myself after reading that paragraph. I think I am more scars than person at this point.)

ANYWAY

The kernel log buffer is in a fixed-size memory buffer in RAM. It defaults to being kinda small (tens or hundreds of kBytes), but you can make it bigger if you want. I suggest you do so.

By itself, this won't solve your kernel panic problems, because RAM is even more volatile than disk, and you have to reboot after a kernel panic. So the RAM is gone, right?

Well, no. Sort of. Not exactly.

Once upon a time, your PC BIOS would go through all your RAM at boot time and run a memory test. I remember my ancient 386DX PC used to do this with my amazingly robust and life-changing 4MB of RAM. It took quite a while. You could press ESC to skip it if you were a valiant risk-taking rebel like myself.

Now, memory is a lot faster than it used to be, but unfortunately it has gotten bigger more quickly than it has gotten faster, especially if you disable memory caching, which you certainly must do at boot time in order to write the very specific patterns needed to see if there are any bit errors.

So... we don't do the boot-time memory test. That ended years ago. If you reboot your system, the memory mostly will contain the stuff it contained before you rebooted. The OS kernel has to know that and zero out pages as they get used. (Sometimes the kernel gets fancy and pre-zeroes some extra pages when it's not busy, so it can hand out zero pages more quickly on demand. But it always has to zero them.)

So, the pages are still around when the system reboots. What we want to happen is:

The system reboots automatically after a kernel panic. You can do this by giving your kernel a boot parameter like "panic=1", which reboots it after one second. (This is not nearly enough time for an end user to read and contemplate the panic message. That's fine, because a) on a desktop PC, X11 will have crashed in graphics mode so you can't see the panic message anyway, and b) on an embedded system there is usually no display to put the message on. End users don't care about panic messages. Our job is to reboot, ASAP, so they don't try to "help" by power cycling the device, which really does lose your memory.) (Advanced users will make it reboot after zero seconds. I think panic=0 disables the reboot feature rather than doing that, so you might have to patch the kernel. I forget. We did it, whatever it was.)
The kernel always initializes the dmesg buffer in the same spot in RAM.
The kernel notices that a previous dmesg buffer is already in that spot in RAM (because of a valid signature or checksum or whatever) and decides to append to that buffer instead of starting fresh.
In userspace, we pick up log processing where we left off. We can capture the log messages starting before (and therefore including) the panic!
And because we redirected userspace logs to the kernel message buffer, we have also preserved the exact sequence of events that led up to the panic.

If you want all this to happen, I have good news and bad news. The good news is we open sourced all our code; the bad news is it didn't get upstreamed anywhere so there are no batteries included and no documentation and it probably doesn't quite work for your use case. Sorry.

Open source code:

logos tool for sending userspace logs to /dev/klogd. (It's logs... for the OS.. and it's logical... and it brings your logs back from the dead after a reboot... get it? No? Oh well.) This includes two per-app token buckets (burst and long-term) so that an out-of-control app won't overfill the limited amount of dmesg space.
PRINTK_PERSIST patch to make Linux reuse the dmesg buffer across reboots.

Even if you don't do any of the rest of this, everybody should use PRINTK_PERSIST on every computer, virtual or physical. Seriously. It's so good.

(Note: room for improvement: it would be better if we could just redirect app stdout/stderr directly to /dev/kmsg, but that doesn't work as well as we want. First, it doesn't auto-prefix incoming messages with the app name. Second, libc functions like printf() actually write a few bytes at a time, not one message per write() call, so they would end up producing more than one dmesg entry per line. Third, /dev/kmsg doesn't support the token bucket rate control that logos does, which turns out to be essential, because sometimes apps go crazy. So we'd have to further extend the kernel API to make it work. It would be worthwhile, though, because the extra userspace process causes an unavoidable delay between when a userspace program prints something and when it actually gets into the kernel log. That delay is enough time for a kernel to panic, and the userspace message gets lost. Writing directly to /dev/kmsg would take less CPU, leave userspace latency unchanged, and ensure the message is safely written before continuing. Someday!)

(In related news, this makes all of syslogd kinda extraneous. Similarly for whatever systemd does. Why do we make everything so complicated? Just write directly to files or the kernel log buffer. It's cheap and easy.)

Uploading the logs

Next, we need to get the messages out of the kernel log buffer and into our log processing server, wherever that might be.

(Note: if we do the above trick - writing userspace messages to the kernel buffer - then we can't also use klogd to read them back into syslogd. That would create an infinite loop, and would end badly. Ask me how I know.)

So, no klogd -> syslogd -> file. Instead, we have something like syslogd -> kmsg -> uploader or app -> kmsg -> uploader.

What is a log uploader? Well, it's a thing that reads messages from the kernel kmsg buffer as they arrive, and uploads them to a server, perhaps over https. It might be almost as simple as "dmesg | curl", like my original prototype, but we can get a bit fancier:

Figure out which messages we've already uploaded (eg. from the persistent buffer before we rebooted) and don't upload those again.
Log the current wall-clock time before uploading, giving us sync points between monotonic time (/dev/kmsg logs "microseconds since boot" by default, which is very useful, but we also want to be able to correlate that with "real" time so we can match messages between related machines).
Compress the file on the way out.
Somehow authenticate with the log server.
Bonus: if the log server is unavailable because of a network partition, try to keep around the last few messages from before the partition, as well as the recent messages once the partition is restored. If the network partition was caused by the client - not too rare if you, like us, were in the business of making routers and wifi access points - you really would like to see the messages from right before the connectivity loss.

Luckily for you, we also open sourced our code for this. It's in C so it's very small and low-overhead. We never quite got the code for the "bonus" feature working quite right, though; we kinda got interrupted at the last minute.

Open source code:

loguploader C client, including an rsyslog plugin for Debian in case you don't want to use the /dev/kmsg trick.
devcert, a tool (and Debian package) which auto-generates a self signed "device certificate" wherever it's installed. The device certificate is used by a device (or VM, container, whatever) to identify itself to the log server, which can then decide how to classify and store (or reject) its logs.

One thing we unfortunately didn't get around to doing was modifying the logupload client to stream logs to the server. This is possible using HTTP POST and Chunked encoding, but our server at the time was unable to accept streaming POST requests due to (I think now fixed) infrastructure limitations.

(Note: if you write load balancing proxy servers or HTTP server frameworks, make sure they can start processing a POST request as soon as all the headers have arrived, rather than waiting for the entire blob to be complete! Then a log upload server can just stream the bytes straight to the next stage even before the whole request has finished.)

Because we lacked streaming in the client, we had to upload chunks of log periodically, which leads to a tradeoff about what makes a good upload period. We eventually settled on about 60 seconds, which ended up accounting for almost all the end-to-end latency from message generation to our monitoring console.

Most people probably think 60 seconds is not too bad. But some of the awesome people on our team managed to squeeze all the other pipeline phases down to tens of milliseconds in total. So the remaining 60 seconds (technically: anywhere from 0 to 60 seconds after a message was produced) was kinda embarrassing. Streaming live from device to server would be better.

The log receiver

So okay, we're uploading the logs from client to some kind of server. What does the server do?

This part is both the easiest and the most reliability-critical. The job is this: receive an HTTP POST request, write the POST data to a file, and return HTTP 200 OK. Anybody who has any server-side experience at all can write this in their preferred language in about 10 minutes.

We intentionally want to make this phase as absolutely simplistic as possible. This is the phase that accepts logs from the limited-size kmsg buffer on the client and puts them somewhere persistent. It's nice to have real-time alerts, but if I have to choose between somewhat delayed alerts or randomly losing log messages when things get ugly, I'll have to accept the delayed alerts. Don't lose log messages! You'll regret it.

The best way to not lose messages is to minimize the work done by your log receiver. So we did. It receives the uploaded log file chunk and appends it to a file, and that's it. The "file" is actually in a cloud storage system that's more-or-less like S3. When I explained this to someone, they asked why we didn't put it in a Bigtable-like thing or some other database, because isn't a filesystem kinda cheesy? No, it's not cheesy, it's simple. Simple things don't break. Our friends on the "let's use structured events to make metrics" team streamed those events straight into a database, and it broke all the time, because databases have configuration options and you inevitably set those options wrong, and it'll fall over under heavy load, and you won't find out until you're right in the middle of an emergency and you really want to see those logs. Or events.

Of course, the file storage service we used was encrypted-at-rest, heavily audited, and auto-deleted files after N days. When you're a megacorporation, you have whole teams of people dedicated to making sure you don't screw this up. They will find you. Best not to annoy them.

We had to add one extra feature, which was authentication. It's not okay for random people on the Internet to be able to impersonate your devices and spam your logs - at least without putting some work into it. For device authentication, we used the rarely-used HTTP client-side certificates option and the devcert program (linked above) so that the client and server could mutually authenticate each other. The server didn't check the certificates against a certification authority (CA), like web clients usually do; instead, it had a database with a whitelist of exactly which certs we're allowing today. So in case someone stole a device cert and started screwing around, we could remove their cert from the whitelist and not worry about CRL bugs and latencies and whatnot.

Unfortunately, because our log receiver was an internal app relying on internal infrastructure, it wasn't open sourced. But there really wasn't much there, honest. The first one was written in maybe 150 lines of python, and the replacement was rewritten in slightly more lines of Go. No problem.

Retries and floods

Of course, things don't always go smoothly. If you're an ISP, the least easy thing is dealing with cases where a whole neighbourhood gets disconnected, either because of a power loss or because someone cut the fiber Internet feed to the neighbourhood.

Now, disconnections are not such a big deal for logs processing - you don't have any. But reconnection is a really big deal. Now you have tens or hundreds of thousands of your devices coming back online at once, and a) they have accumulated a lot more log messages than they usually do, since they couldn't upload them, and b) they all want to talk to your server at the same time. Uh oh.

Luckily, our system was designed carefully (uh... eventually it was), so it could handle these situations pretty smoothly:

The log uploader uses a backoff timer so that if it's been trying to upload for a while, it uploads less often. (However, the backoff timer was limited to no more than the usual inter-upload interval. I don't know why more people don't do this. It's rather silly for your system to wait longer between uploads in a failure situation than it would in a success situation. This is especially true with logs, where when things come back online, you want a status update now. And clearly your servers have enough capacity to handle uploads at the usual rate, because they usually don't crash. Sorry if I sound defensive here, but I had to have this argument a few times with a few SREs. I understand why limiting the backoff period isn't always the right move. It's the right move here.)
Less obviously, even under normal conditions, the log uploader uses a randomized interval between uploads. This avoids traffic spikes where, after the Internet comes back online, everybody uploads again exactly 60 seconds later, and so on.
The log upload client understands the idea that the server can't accept its request right now. It has to, anyway, because if the Internet goes down, there's no server. So it treats server errors exactly like it treats lack of connectivity. And luckily, log uploading is not really an "interactive" priority task, so it's okay to sacrifice latency when things get bad. Users won't notice. And apparently our network is down, so the admins already noticed.
The /dev/kmsg buffer was configured for the longest reasonable outage we could expect, so that it wouldn't overflow during "typical" downtime. Of course, there's a judgement call here. But the truth is, if you're having system-wide downtime, what the individual devices were doing during that downtime is not usually what you care about. So you only need to handle, say, the 90th percentile of downtime. Safely ignore the black swans for once.
The log receiver aggressively rejects requests that come faster than its ability to write files to disk. Since the clients know how to retry with a delay, this allows us to smooth out bursty traffic without needing to either over-provision the servers or lose log messages.

(Pro tip, learned the hard way: if you're writing a log receiver in Go, don't do the obvious thing and fire off a goroutine for every incoming request. You'll run out of memory. Define a maximum number of threads you're willing to handle at once, and limit your request handling to that. It's okay to set this value low, just to be safe: remember, the uploader clients will come back later.)

Okay! Now our (unstructured) logs from all our 100,000 devices are sitting safely in a big distributed filesystem. We have a little load-balanced, multi-homed cluster of log receivers accepting the uploads, and they're so simple that they should pretty much never die, and even if they do because we did something dumb (treacherous, treacherous goroutines!), the clients will try again.

What might not be obvious is this: our reliability, persistence, and scaling problems are solved. Or rather, as long as we have enough log receiver instances to handle all our devices, and enough disk quota to store all our logs, we will never again lose a log message.

That means the rest of our pipeline can be best-effort, complicated, and frequently exploding. And that's a good thing, because we're going to start using more off-the-shelf stuff, we're going to let random developers reconfigure the filtering rules, and we're not going to bother to configure it with any redundancy.

Grinding the logs

The next step is to take our unstructured logs and try to understand them. In other words, we want to add some structure. Basically we want to look for lines that are "interesting" and parse out the "interesting" data and produce a stream of events, each with a set of labels describing what categories they apply to.

Note that, other than this phase, there is little difference between how you'd design a structured event reporting pipeline and a log pipeline. You still need to collect the events. You still (if you're like me) need to persist your events across kernel panics. You still need to retry uploading them if your network gets partitioned. You still need the receivers to handle overloading, burstiness, and retries. You still would like to stream them (if your infrastructure can handle it) rather than uploading every 60 seconds. You still want to be able to handle a high volume of them. You're just uploading a structured blob instead of an unstructured blob.

Okay. Fine. If you want to upload structured blobs, go for it. It's just an HTTP POST that appends to a file. Nobody's stopping you. Just please try to follow my advice when designing the parts of the pipeline before and after this phase, because otherwise I guarantee you'll be sad eventually.

Anyway, if you're staying with me, now we have to parse our unstructured logs. What's really cool - what makes this a killer design compared to starting with structured events in the first place - is that we can, at any time, change our minds about how to parse the logs, without redeploying all the software that produces them.

This turns out to be amazingly handy. It's so amazingly handy that nobody believes me. Even I didn't believe me until I experienced it; I was sure, in the beginning, that the unstructured logs were only temporary and we'd initially use them to figure out what structured events we wanted to record, and then modify the software to send those, then phase out the logs over time. This never happened. We never settled down. Every week, or at least every month, there was some new problem which the existing "structured" events weren't configured to catch, but which, upon investigating, we realized we could diagnose and measure from the existing log message stream. And so we did!

Now, I have to put this in perspective. Someone probably told you that log messages are too slow, or too big, or too hard to read, or too hard to use, or you should use them while debugging and then delete them. All those people were living in the past and they didn't have a fancy log pipeline. Computers are really, really fast now. Storage is really, really cheap.

So we let it all out. Our devices produced an average of 50 MB of (uncompressed) logs per day, each. For the baseline 100,000 devices that we discussed above, that's about 5TB of logs per day. Ignoring compression, how much does it cost to store, say, 60 days of logs in S3 at 5TB per day? "Who cares," that's how much. You're amortizing it over 100,000 devices. Heck, a lot of those devices were DVRs, each with 2TB of storage. With 100,000 DVRs, that's 200,000 TB of storage. Another 300 is literally a rounding error (like, smaller than if I can't remember if it's really 2TB or 2TiB or what).

Our systems barfed up logs vigorously and continuously, like a non-drunken non-sailor with seasickness. And it was beautiful.

(By the way, now would be a good time to mention some things we didn't log: personally identifiable information or information about people's Internet usage habits. These were diagnostic logs for running the network and detecting hardware/software failures. We didn't track what you did with the network. That was an intentional decision from day 1.)

(Also, this is why I think all those log processing services are so badly overpriced. I wanna store 50 MB per device, for lots of devices. I need to pay S3 rates for that, not a million dollars a gigabyte. If I have to overpay for storage, I'll have to start writing fewer logs. I love my logs. I need my logs. I know you're just storing it in S3 anyway. You probably get a volume discount! Let's be realistic.)

But the grinding, though

Oh right. So the big box labeled "Grinder" in my diagram was, in fact, just one single virtual machine, for a long time. It lasted like that for much longer than we expected.

Whoa, how is that possible, you ask?

Well, at 5TB per day per 100,000 devices, that's an average of 57 MBytes per second. And remember, burstiness has already been absorbed by our carefully written log receivers and clients, so we'll just grind these logs as fast as they arrive or as fast as we can, and if there are fluctuations, they'll average out. Admittedly, some parts of the day are busier than others. Let's say 80 MBytes per second at peak.

80 MBytes per second? My laptop can do that on its spinning disk. I don't even need an SSD! 80 MBytes per second is a toy.

And of course, it's not just one spinning disk. The data itself is stored on some fancy heavily-engineered distributed filesystem that I didn't have to design. Assuming there are no, er, colossal, failures in provisioning (no comment), there's no reason we shouldn't be able to read files at a rate that saturates the network interface available to our machine. Surely that's at least 10 Gbps (~1 GByte/sec) nowadays, which is 12.5 of those. 1.25 million devices, all processed by a single grinder.

Of course you'll probably need to use a few CPU cores. And the more work you do per log entry, the slower it'll get. But these estimates aren't too far off what we could handle.

And yeah, sometimes that VM gets randomly killed by the cluster's Star Trek-esque hive mind for no reason. It doesn't matter, because the input data was already persisted by the log receivers. Just start a new grinder and pick up where you left off. You'll have to be able to handle process restarts no matter what. And that's a lot easier than trying to make a distributed system you didn't need.

As for what the grinder actually does? Anything you want. But it's basically the "map" phase in a mapreduce. It reads the data in one side, does some stuff to it, and writes out postprocessed stuff on the other side. Use your imagination. And if you want to write more kinds of mappers, you can run them, either alongside the original Grinder or downstream from it.

Our Grinder mostly just ran regexes and put out structures (technically protobufs) that were basically sets of key-value pairs.

(For some reason, when I search the Internet for "streaming mapreduce," I don't get programs that do this real-time processing of lots of files as they get written. Instead, I seem to get batch-oriented mapreduce clones that happen to read from stdin, which is a stream. I guess. But... well, now you've wasted some perfectly good words that could have meant something. So okay, too bad, it's a Grinder. Sue me.)

Reducers and Indexers

Once you have a bunch of structured events... well, I'm not going to explain that in a lot of detail, because it's been written about a lot.

You probably want to aggregate them a bit - eg. to count up reboots across multiple devices, rather than storing each event for each device separately - and dump them into a time-series database. Perhaps you want to save and postprocess the results in a monitoring system named after Queen Elizabeth or her pet butterfly. Whatever. Plug in your favourite.

What you probably think you want to do, but it turns out you rarely need, is full-text indexing. People just don't grep the logs across 100,000 devices all that often. I mean, it's kinda nice to have. But it doesn't have to be instantaneous. You can plug in your favourite full text indexer if you like. But most of the time, just an occasional big parallel grep (perhaps using your favourite mapreduce clone or something more modern... or possibly just using grep) of a subset of the logs is sufficient.

(If you don't have too many devices, even a serial grep can be fine. Remember, a decent cloud computer should be able to read through ~1 GByte/sec, no problem. How much are you paying for someone to run some bloaty full-text indexer on all your logs, to save a few milliseconds per grep?)

I mean, run a full text indexer if you want. The files are right there. Don't let me stop you.

On the other hand, being able to retrieve the exact series of logs - let's call it the "narrative" - from a particular time period across a subset of devices turns out to be super useful. A mini-indexer that just remembers which logs from which devices ended up in which files at which offsets is nice to have. Someone else on our team built one of those eventually (once we grew so much that our parallel grep started taking minutes instead of seconds), and it was very nice.

And then you can build your dashboards

Once you've reduced, aggregated, and indexed your events into your favourite output files and databases, you can read those databases to build very fast-running dashboards. They're fast because the data has been preprocessed in mostly-real time.

As I mentioned above, we had our pipeline reading the input files as fast as they could come in, so the receive+grind+reduce+index phase only took a few tens of milliseconds. If your pipeline isn't that fast, ask somebody why. I bet their program is written in java and/or has a lot of sleep() statements or batch cron jobs with intervals measured in minutes.

Again here, I'm not going to recommend a dashboard tool. There are millions of articles and blog posts about that. Pick one, or many.

In conclusion

Please, please, steal these ideas. Make your log and event processing as stable as our small team made our log processing. Don't fight over structured vs unstructured; if you can't agree, just log them both.

Don't put up with weird lags and limits in your infrastructure. We made 50MB/day/device work for a lot of devices, and real-time mapreduced them all on a single VM. If we can do that, then you can make it work for a few hundreds, or a few thousands, of container instances. Don't let anyone tell you you can't. Do the math: of course you can.

Epilogue

Eventually our team's log processing system evolved to become the primary monitoring and alerting infrastructure for our ISP. Rather than alerting on behaviour of individual core routers, it turned out that the end-to-end behaviour observed by devices in the field were a better way to detect virtually any problem. Alert on symptoms, not causes, as the SREs like to say. Who has the symptoms? End users.

We had our devices ping different internal servers periodically and log the round trip times; in aggregate, we had an amazing view of overloading, packet loss, bufferbloat, and poor backbone routing decisions, across the entire fleet, across every port of every switch. We could tell which was better, IPv4 or IPv6. (It's always IPv4. Almost everyone spends more time optimizing their IPv4 routes and peering. Sorry, but it's true.)

We detected some weird configuration problems with the DNS servers in one city by comparing the 90th percentile latency of DNS lookups across all the devices in every city.

We diagnosed a manufacturing defect in a particular batch of devices, just based on their CPU temperature curves and fan speeds.

We worked with our CPU vendor to find and work around a bug in their cache coherency, because we spotted a kernel panic that would happen randomly every 10,000 CPU-hours, but for every 100,000 devices, that's still 10 times per hour of potential clues.

...and it sure was good for detecting power failures.

Anyway. Log more stuff. Collect those logs. Let it flow. Trust me.

Update 2019-04-26: Based on a lot of positive feedback from people who read this blog post, I ended up starting a company that might be able to help you with your logs problems. We're building pipelines that are very similar to what's described here.

Update 2020-08-26:

Aha! Okay, for some reason this article is trending again, and I'd better provide an update on my update. We did implement parts of this design for use in our core product, which is now quite distinct from logs processing.

After investigating the "logs" market last year, we decided not to commercialize a logs processing service. The reason is that the characteristics we want our design to have: cheap, lightweight, simple, fast, and reliable - are all things you would expect from the low-cost provider in a market. The "logs processing" space is crowded with a lot of premium products that are fancy, feature-filled, etc, and reliable too, and thus able to charge a lot of money.

Instead, we built a minimalistic version of the above design for our internal use, collecting distributed telemetry about Tailscale connection success rates to help debug the network. Big companies can also use it to feed into their IDS and SIEM systems.

We considered open sourcing the logs services we built (since open source is where attributes like cheap, lightweight, etc tend to flourish) but we can't afford the support overhead right now for a product that is at best tangential to our main focus. Sorry! Hopefully someday.

Quotes from 1992

Fri, 08 Mar 2019 23:19:49 +0000

I was recently recommended to read the book Accidental Empires by Robert X. Cringely, first published in 1992 (or was it 1991?) and apparently no longer in print and also not in e-book format. To my surprise, it turns out archive.org has a solution for this, an epically user-unfriendly "virtual library card" (which is still worth it if you need to read a book) in which they apparently receive one physical copy of a book, scan it, and lend it out digitally, one person at a time, using an aggressively user-hostile DRM abomination called Adobe Digital Editions.

(I'm not kidding about the user hostility. The iOS version, for no reason, actually requires you to double click to open a book from the list. But, like double clicking with a mouse, the second click doesn't count if you're more than a few pixels off. I hope you have really precise fingers! I suspect the people who wrote it literally did not ever run it on real iPhone hardware, and therefore tested it with a mouse.)

ANYWAY

It's a pretty fascinating book, snapshotting the microcomputer industry (and its history) as things sat back in 1991 (or was it 1992?). It surprised me how many things are just like they were back then, 27 years ago (yikes!). The names have changed, but human organizations remain the same.

(He also serialized and updated the book on his blog back in 2012-2013, but it's really long and I didn't have time to read it, and apparently that version also didn't come out in ebook format. I would have happily paid for it to read in a more convenient format. Oh well.)

Here are some quotes I painstakingly retyped. Take that, DRM garbage.

On avoiding adulthood:

They weren't rebels; they resented their parents and society very little. Their only alienation was the usual hassle of the adolescent - a feeling of being prodded into adulthood on somebody else's terms. [...] And turning this culture into a business? That was just a happy accident that allowed these boys to put off forever the horror age - that dividing line to adulthood that they would otherwise have been forced to cross after college.

On big things failing to scale down:

How did we get from big computers that lived in the basement of office buildings to the little computers that live on our desks today? We didn't. personal computers have almost nothing to do with big computers. They never have. [...] Big computers and little computers are completely different beasts created by radically different groups of people. It's logical, I know, to assume that the personal computer came from shrinking a mainframe, but that's not the way it happened.

On amateurs as the carriers of progress, predicting the success of open source:

It takes new ideas a long time to catch on - time that is mainly devoted to evolving the idea into something useful. This fact alone dumps most of the responsibility for early technical innovation in the laps of amateurs, who can afford to take the time. Only those who aren't trying to make money can afford to advance a technology that doesn't pay.

On the surprising existence of limited ambition:

Let's say for a minute that Eubanks was correct, and Gary Kildall didn't give a shit about the business. Who said that he had to? CP/M was his invention; Digital Research was his company. The fact that it succeeded beyond anyone's expectations did not make those earlier expectations invalid. Gary Kildall's ambition was limited, something that is not supposed to be a factor in American business. If you hope for a thousand and get a million, you are still expected to want more, but he didn't.

On the Chief Scientist job title:

In a personal computer hardware or software company, being named chief scientist means that the boss doesn't know what to do with you. Chief scientists don't generally have to do anything; they're just smart people whom the company doesn't want to lose to a competitor.

"Research subjects" at PARC, the original doesn't-count-as-headcount TVCs:

Money wasn't a problem, but manpower was; it was almost impossible to hire additional people at the Computer Science Laboratory because of the arduous hiring gauntlet and Taylor's reluctance to manage extra heads. [...] Simonyi came up with a scam. He proposed a research project to study programmer productivity and how to increase it. In the course of the study, test subjects would be paid to write software under Simonyi's supervision. [...] By calling them research subjects rather than programmers, he was able to bring some worker bees into PARC.

On indoctrination of recent graduates:

Through the architects and program managers, Gates was able to control the work of every programmer at Microsoft, but to do so reliably required cheap and obedient labor. Gates set a policy that consciously avoided hiring experienced programmers, specializing, instead, in recent computer science graduates.

On the dangers of only hiring geniuses:

Charles Simonyi accepts Microsoft mediocrity as an inevitable price paid to create a large organization. "The risk of genius is that the products that result from genius often don't have much to do with each other," he explained.

On the value of limiting yourself to standards:

...which was why the idea of 100 percent IBM compatibility took so long to be accepted. "Why be compatible when you could be better?" the smart guys asked on their way to bankruptcy court.

The social effects of frequent reorgs:

The rest of the company was as confused as its leadership. Somehow, early on, reorganizations - "reorgs" - became part of the Apple culture. they happen every three to six months and come from Apple's basic lack of understanding that people need stability in order to be able to work together. [...] Make a bad decision? Who cares! By the time the bad news arrives, you'll be gone and someone else will have to handle the problems.

On survival in a large organization, which is very different from entrepreneurship:

I learned an important lesson that day: Success in a large organization, whether it's a university or IBM, is generally based on appearance, not reality. It's understanding the system and then working within it that really counts, not bowling scores or body bags.

An interesting organizational structure, where Bill Gates was the chairman but hired a president who would be responsible for everything except software development:

This idea of nurturing the original purpose of the company while expanding the business organization is something that most software and hardware companies lose sight of as they grow. They managed it at Microsoft by having the programmers continue to report to Bill Gates while everyone on the business side reported to Shirley.

On how Flight Simulator got approved:

Then there was Flight Simulator, the only computer game published by Microsoft. There was no business plan that included a role for computer games in Microsoft's future. Bill Gates just liked to play Flight Simulator, so Microsoft published it.

On Novell's very strange history, and why it was so unusually good for its era (it really was, too. That thing was great):

The early versions of most software are so bad that good programmers usually want to throw them away but can't because ship dates have to be met. But Novell wasn't shipping anything in 1982-1983, so early versions of its network software were thrown away and started over again. Novell was able to take the time needed to come up with the correct architecture, a rare luxury for a start-up, and subsequently the company's greatest advantage.

A more specific version of the "Microsoft takes three tries to get anything right" theory:

Microsoft's entry into most new technologies follows this same plan, with the first effort being a preemptive strike, the second effort being market research to see what customers really want in a product, and the third try is the real product.

On industry analysts giving you the numbers you want to hear:

...the question, which was: When will unit sales of OS/2 exceed those of DOS? The assumption (and the flaw) built into this exercise is that OS/2, because it was being pushed by IBM, was destined to overtake DOS, which it hasn't. But given that the paying customers wanted OS/2 to succeed and that the research question itself suggested that OS/2 would succeed, market research companies like Dataquest, InfoCorp, and International Data Corporation dutifully crazy-glued their usual demand curves on a chart and predicted that OS/2 would be a big hit. There were no dissenting voices. Not a single market research report that I read or read about at that time predicted that OS/2 would be a failure.

On Bill Gates's annual reading weeks:

Annual reading weeks, when Gates stays home and reads technical reports for seven days straight and then emerges to reposition the company, are a tradition at Microsoft. Nothing is allowed to get in the way of planned reading for Chairman Bill.

A partially-failed prediction about Steve Jobs:

(When InfoWorld's Peggy Watt asked Gates if Microsoft would develop applications for the NeXT computer, he said, "Develop for it? I'll piss on it.") Alas, I'm not giving very good odds that Steve Jobs will be the leader of the next generation of personal computing.

A little-known partnership between IBM and Apple to try to make a new OS (Pink) that would finally beat DOS:

IBM has 33,000 programmers on its payroll but is so far from leading the software business (and knows it) that it is betting the company on the work of 100 Apple programmers wearing T-shirts in Mountain View, California.

A perception of industry fatigue in 1991-1992, which was around the time almost everyone gave up competing with DOS+Windows. Interestingly, this is also when Linux arrived (produced by "amateurs", see above) and may have rejuvenated things:

But today, everyone who wants to be in the PC business is already in it. Except for a new batch of kids who appear out of school each year, the only new blood in this business is due to immigration. And the old blood is getting tired - tired of failing in some cases or just tired of working so hard and now ready to enjoy life. The business is slowing down, and this loss of energy is the greatest threat to our computing future as a nation. Forget about the Japanese; their threat is nothing compared to this loss of intellectual vigor.

A rather bad-seeming idea for "software studios," which maybe seemed like a good idea at the time, but we've kinda tried it since then and it has a lot of unexpected downsides:

[Comparing to Hollywood studio structure] In the computer business, too, we've held to the idea that every product is going to live forever. We should be like the movies and only do sequels of hits. And you don't have to keep the original team together to do a sequel. All you have to do is make sure that the new version can read all the old product files and that it feels familiar.

An interesting perspective on the (then still in progress) takeover of computer hardware manufacturing by Asian countries. China wasn't yet even on the radar. (His claim was that it didn't matter because software was so valuable and stayed in America. 27 years later, that prediction has held up okay, although Asia seems to be doing fine with that hardware stuff):

The hardware business is dying. Let it. The Japanese and Koreans are so eager to take over the PC hardware business that they are literally trying to buy the future. But they're only buying the past.

(All above quotes by Robert X. Cringely)

Forget privacy: you're terrible at targeting anyway

Sat, 09 Feb 2019 02:09:45 +0000

I don't mind letting your programs see my private data as long as I get something useful in exchange. But that's not what happens.

A former co-worker told me once: "Everyone loves collecting data, but nobody loves analyzing it later." This claim is almost shocking, but people who have been involved in data collection and analysis have all seen it. It starts with a brilliant idea: we'll collect information about every click someone makes on every page in our app! And we'll track how long they hesitate over a particular choice! And how often they use the back button! How many seconds they watch our intro video before they abort! How many times they reshare our social media post!

And then they do track all that. Tracking it all is easy. Add some log events, dump them into a database, off we go.

But then what? Well, after that, we have to analyze it. And as someone who has analyzed a lot of data about various things, let me tell you: being a data analyst is difficult and mostly unrewarding (except financially).

See, the problem is there's almost no way to know if you're right. (It's also not clear what the definition of "right" is, which I'll get to in a bit.) There are almost never any easy conclusions, just hard ones, and the hard ones are error prone. What analysts don't talk about is how many incorrect charts (and therefore conclusions) get made on the way to making correct ones. Or ones we think are correct. A good chart is so incredibly persuasive that it almost doesn't even matter if it's right, as long as what you want is to persuade someone... which is probably why newpapers, magazines, and lobbyists publish so many misleading charts.

But let's leave errors aside for the moment. Let's assume, very unrealistically, that we as a profession are good at analyzing things. What then?

Well, then, let's get rich on targeted ads and personalized recommendation algorithms. It's what everyone else does!

Or do they?

The state of personalized recommendations is surprisingly terrible. At this point, the top recommendation is always a clickbait rage-creating article about movie stars or whatever Trump did or didn't do in the last 6 hours. Or if not an article, then a video or documentary. That's not what I want to read or to watch, but I sometimes get sucked in anyway, and then it's recommendation apocalypse time, because the algorithm now thinks I like reading about Trump, and now everything is Trump. Never give positive feedback to an AI.

This is, by the way, the dirty secret of the machine learning movement: almost everything produced by ML could have been produced, more cheaply, using a very dumb heuristic you coded up by hand, because mostly the ML is trained by feeding it examples of what humans did while following a very dumb heuristic. There's no magic here. If you use ML to teach a computer how to sort through resumes, it will recommend you interview people with male, white-sounding names, because it turns out that's what your HR department already does. If you ask it what video a person like you wants to see next, it will recommend some political propaganda crap, because 50% of the time 90% of the people do watch that next, because they can't help themselves, and that's a pretty good success rate.

(Side note: there really are some excellent uses of ML out there, for things traditional algorithms are bad at, like image processing or winning at strategy games. That's wonderful, but chances are good that your pet ML application is an expensive replacement for a dumb heuristic.)

Someone who works on web search once told me that they already have an algorithm that guarantees the maximum click-through rate for any web search: just return a page full of porn links. (Someone else said you can reverse this to make a porn detector: any link which has a high click-through rate, regardless of which query it's answering, is probably porn.)

Now, the thing is, legitimate-seeming businesses can't just give you porn links all the time, because that's Not Safe For Work, so the job of most modern recommendation algorithms is to return the closest thing to porn that is still Safe For Work. In other words, celebrities (ideally attractive ones, or at least controversial ones), or politics, or both. They walk that line as closely as they can, because that's the local maximum for their profitability. Sometimes they accidentally cross that line, and then have to apologize or pay a token fine, and then go back to what they were doing.

This makes me sad, but okay, it's just math. And maybe human nature. And maybe capitalism. Whatever. I might not like it, but I understand it.

My complaint is that none of the above had anything to do with hoarding my personal information.

The hottest recommendations have nothing to do with me

Let's be clear: the best targeted ads I will ever see are the ones I get from a search engine when it serves an ad for exactly the thing I was searching for. Everybody wins: I find what I wanted, the vendor helps me buy their thing, and the search engine gets paid for connecting us. I don't know anybody who complains about this sort of ad. It's a good ad.

And it, too, had nothing to do with my personal information!

Google was serving targeted search ads decades ago, before it ever occurred to them to ask me to log in. Even today you can still use every search engine web site without logging in. They all still serve ads targeted to your search keyword. It's an excellent business.

There's another kind of ad that works well on me. I play video games sometimes, and I use Steam, and sometimes I browse through games on Steam and star the ones I'm considering buying. Later, when those games go on sale, Steam emails me to tell me they are on sale, and sometimes then I buy them. Again, everybody wins: I got a game I wanted (at a discount!), the game maker gets paid, and Steam gets paid for connecting us. And I can disable the emails if I want, but I don't want, because they are good ads.

But nobody had to profile me to make that happen! Steam has my account, and I told it what games I wanted and then it sold me those games. That's not profiling, that's just remembering a list that I explicitly handed to you.

Amazon shows a box that suggests I might want to re-buy certain kinds of consumable products that I've bought in the past. This is useful too, and requires no profiling other than remembering the transactions we've had with each other in the past, which they kinda have to do anyway. And again, everybody wins.

Now, Amazon also recommends products like the ones I've bought before, or looked at before. That's, say, 20% useful. If I just bought a computer monitor, and you know I did because I bought it from you, then you might as well stop selling them to me. But for a few days after I buy any electronics they also keep offering to sell me USB cables, and they're probably right. So okay, 20% useful targeting is better than 0% useful. I give Amazon some credit for building a useful profile of me, although it's specifically a profile of stuff I did on their site and which they keep to themselves. That doesn't seem too invasive. Nobody is surprised that Amazon remembers what I bought or browsed on their site.

Worse is when (non-Amazon) vendors get the idea that I might want something. (They get this idea because I visited their web site and looked at it.) So their advertising partner chases me around the web trying to sell me the same thing. They do that, even if I already bought it. Ironically, this is because of a half-hearted attempt to protect my privacy. The vendor doesn't give information about me or my transactions to their advertising partner (because there's an excellent chance it would land them in legal trouble eventually), so the advertising partner doesn't know that I bought it. All they know (because of the advertising partner's tracker gadget on the vendor's web site) is that I looked at it, so they keep advertising it to me just in case.

But okay, now we're starting to get somewhere interesting. The advertiser has a tracker that it places on multiple sites and tracks me around. So it doesn't know what I bought, but it does know what I looked at, probably over a long period of time, across many sites.

Using this information, its painstakingly trained AI makes conclusions about which other things I might want to look at, based on...

...well, based on what? People similar to me? Things my Facebook friends like to look at? Some complicated matrix-driven formula humans can't possibly comprehend, but which is 10% better?

Probably not. Probably what it does is infer my gender, age, income level, and marital status. After that, it sells me cars and gadgets if I'm a guy, and fashion if I'm a woman. Not because all guys like cars and gadgets, but because some very uncreative human got into the loop and said "please sell my car mostly to men" and "please sell my fashion items mostly to women." Maybe the AI infers the wrong demographic information (I know Google has mine wrong) but it doesn't really matter, because it's usually mostly right, which is better than 0% right, and advertisers get some mostly demographically targeted ads, which is better than 0% targeted ads.

You know this is how it works, right? It has to be. You can infer it from how bad the ads are. Anyone can, in a few seconds, think of some stuff they really want to buy which The Algorithm has failed to offer them, all while Outbrain makes zillions of dollars sending links about car insurance to non-car-owning Manhattanites. It might as well be a 1990s late-night TV infomercial, where all they knew for sure about my demographic profile is that I was still awake.

You tracked me everywhere I go, logging it forever, begging for someone to steal your database, desperately fearing that some new EU privacy regulation might destroy your business... for this?

Statistical Astrology

Of course, it's not really as simple as that. There is not just one advertising company tracking me across every web site I visit. There are... many advertising companies tracking me across every web site I visit. Some of them don't even do advertising, they just do tracking, and they sell that tracking data to advertisers who supposedly use it to do better targeting.

This whole ecosystem is amazing. Let's look at online news web sites. Why do they load so slowly nowadays? Trackers. No, not ads - trackers. They only have a few ads, which mostly don't take that long to load. But they have a lot of trackers, because each tracker will pay them a tiny bit of money to be allowed to track each page view. If you're a giant publisher teetering on the edge of bankruptcy and you have 25 trackers on your web site already, but tracker company #26 calls you and says they'll pay you $50k a year if you add their tracker too, are you going to say no? Your page runs like sludge already, so making it 1/25th more sludgy won't change anything, but that $50k might.

("Ad blockers" remove annoying ads, but they also speed up the web, mostly because they remove trackers. Embarrassingly, the trackers themselves don't even need to cause a slowdown, but they always do, because their developers are invariably idiots who each need to load thousands of lines of javascript to do what could be done in two. But that's another story.)

Then the ad sellers, and ad networks, buy the tracking data from all the trackers. The more tracking data they have, the better they can target ads, right? I guess.

The brilliant bit here is that each of the trackers has a bit of data about you, but not all of it, because not every tracker is on every web site. But on the other hand, cross-referencing individuals between trackers is kinda hard, because none of them wants to give away their secret sauce. So each ad seller tries their best to cross-reference the data from all the tracker data they buy, but it mostly doesn't work. Let's say there are 25 trackers each tracking a million users, probably with a ton of overlap. In a sane world we'd guess that there are, at most, a few million distinct users. But in an insane world where you can't prove if there's an overlap, it could be as many as 25 million distinct users! The more tracker data your ad network buys, the more information you have! Probably! And that means better targeting! Maybe! And so you should buy ads from our network instead of the other network with less data! I guess!

None of this works. They are still trying to sell me car insurance for my subway ride.

It's not just ads

That's a lot about profiling for ad targeting, which obviously doesn't work, if anyone would just stop and look at it. But there are way too many people incentivized to believe otherwise. Meanwhile, if you care about your privacy, all that matters is they're still collecting your personal information whether it works or not.

What about content recommendation algorithms though? Do those work?

Obviously not. I mean, have you tried them. Seriously.

That's not quite fair. There are a few things that work. Pandora's music recommendations are surprisingly good, but they are doing it in a very non-obvious way. The obvious way is to take the playlist of all the songs your users listen to, blast it all into an ML training dataset, and then use that to produce a new playlist for new users based on... uh... their... profile? Well, they don't have a profile yet because they just joined. Perhaps based on the first few songs they select manually? Maybe, but they probably started with either a really popular song, which tells you nothing, or a really obscure song to test the thoroughness of your library, which tells you less than nothing.

(I'm pretty sure this is how Mixcloud works. After each mix, it tries to find the "most similar" mix to continue with. Usually this is someone else's upload of the exact same mix. Then the "most similar" mix to that one is the first one, so it does that. Great job, machine learning, keep it up.)

That leads us to the "random song followed by thumbs up/down" system that everyone uses. But everyone sucks, except Pandora. Why? Apparently because Pandora spent a lot of time hand-coding a bunch of music characteristics and writing a "real algorithm" (as opposed to ML) that tries to generate playlists based on the right combinations of those characteristics.

In that sense, Pandora isn't pure ML. It often converges on a playlist you'll like within one or two thumbs up/down operations, because you're navigating through a multidimensional interconnected network of songs that people encoded the hard way, not a massive matrix of mediocre playlists scraped from average people who put no effort into generating those playlists in the first place. Pandora is bad at a lot of things (especially "availability in Canada") but their music recommendations are top notch.

Just one catch. If Pandora can figure out a good playlist based on a starter song and one or two thumbs up/down clicks, then... I guess it's not profiling you. They didn't need your personal information either.

Netflix

While we're here, I just want to rant about Netflix, which is an odd case of starting off with a really good recommendation algorithm and then making it worse on purpose.

Once upon a time, there was the Netflix prize, which granted $1 million to the best team that could predict people's movie ratings, based on their past ratings, with better accuracy than Netflix could themselves. (This not-so-shockingly resulted in a privacy fiasco when it turned out you could de-anonymize the data set that they publicly released, oops. Well, that's what you get when you long-term store people's personal information in a database.)

Netflix believed their business depended on a good recommendation algorithm. It was already pretty good: I remember using Netflix around 10 years ago and getting several recommendations for things I would never have discovered, but which I turned out to like. That hasn't happened to me on Netflix in a long, long time.

As the story goes, once upon a time Netflix was a DVD-by-mail service. DVD-by-mail is really slow, so it was absolutely essential that at least one of this week's DVDs was good enough to entertain you for your Friday night movie. Too many Fridays with only bad movies, and you'd surely unsubscribe. A good recommendation system was key. (I guess there was also some interesting math around trying to make sure to rent out as much of the inventory as possible each week, since having a zillion copies of the most recent blockbuster, which would be popular this month and then die out next month, was not really viable.)

Eventually though, Netflix moved online, and the cost of a bad recommendation was much less: just stop watching and switch to a new movie. Moreover, it was perfectly fine if everyone watched the same blockbuster. In fact, it was better, because they could cache it at your ISP and caches always work better if people are boring and average.

Worse, as the story goes, Netflix noticed a pattern: the more hours people watch, the less likely they are to cancel. (This makes sense: the more hours you spend on Netflix, the more you feel like you "need" it.) And with new people trying the service at a fixed or proportional rate, higher retention translates directly to faster growth.

When I heard this was also when I learned the word "satisficing," which essentially means searching through sludge not for the best option, but for a good enough option. Nowadays Netflix isn't about finding the best movie, it's about satisficing. If it has the choice between an award-winning movie that you 80% might like or 20% might hate, and a mainstream movie that's 0% special but you 99% won't hate, it will recommend the second one every time. Outliers are bad for business.

The thing is, you don't need a risky, privacy-invading profile to recommend a mainstream movie. Mainstream movies are specially designed to be inoffensive to just about everyone. My Netflix recommendations screen is no longer "Recommended for you," it's "New Releases," and then "Trending Now," and "Watch it again."

As promised, Netflix paid out their $1 million prize to buy the winning recommendation algorithm, which was even better than their old one. But they didn't use it, they threw it away.

Some very expensive A/B testers determined that this is what makes me watch the most hours of mindless TV. Their revenues keep going up. And they don't even need to invade my privacy to do it.

Who am I to say they're wrong?

Factors in authentication

Fri, 01 Mar 2019 02:16:40 +0000

Multi-factor authentication remains hard-to-use, hard-to-secure, and error-prone. I've been studying authentication lately to see if it might be possible to adapt some security practices, especially phishing prevention, from big companies to small companies and consumers.

Here's what I have so far.

(Sorry: this is another long one. Security is complicated!)

The three factors aren't independent

The traditional definition of multi-factor authentication (although I couldn't find a reference to where it originated) is that you should have two or more of:

Something you know (eg. a password)
Something you have (eg. a card)
Something you are (eg. a fingerprint or retinal scan).

This definition is actually not bad, and has done its job for years. But it tends to lead people into some dead ends that, years later, we now know are wrong.

First of all, let's talk about #3: something you are. The obvious implementation of this is to scan your fingerprint or retina at a door or computer screen. The computer looks it up in a central database somewhere and allows or denies entry. Right?

Well, it turns out, not so much. First of all, there are huge privacy implications to having your biometric data in a central server. I have a Nexus card, which is a programme created jointly by the Canada and U.S. governments to allow theoretically slightly faster border crossing between the two countries. In reality, it is almost certainly a ploy to collect more biometric data. For some reason, Canada insisted on collecting my retinal scans, and the U.S. insisted on collecting my fingerprints, and it seems a bit unlikely to me that the information from one of those will never leak to the other. So now two of my biometrics are in two different databases, someday to be (if not already) merged into one. Whether or not this is legal is probably of only academic interest to the spy agencies doing it.

Now that's for government use, which is bad enough. But I don't need every bank, employer, or apartment building to have my biometrics, for all sorts of reasons. It's a privacy invasion. It can be used to track me well beyond the intended use case. It can be copied, by a sufficiently skilled attacker, and used to masquerade as me. And when that happens, it's not something I can replace like I can replace a stolen password. The dark joke among security engineers is at least I have ten fingers, so I get ten tries at keeping it safe.

Yeah, yeah, this is old news, right? But actually, biometrics have an even bigger problem that people don't talk about enough: false positives. If you have a giant database of fingerprints or retinas or DNA samples or voiceprints or faces, and you have an algorithm for searching the database, and you have one sample you want to look up in the database to find the right match... it turns out to be a hard problem. ML/AI people hide this problem from you all the time. There's a fun toy that tells you which celebrity you look most similar to, and it's eerily accurate, right? Sure. But the toy works because of false positives. It's easy, relatively, to find matches that look similar to you. It's tough to figure out which of those matches, if any, really are you. Which is fine for a toy, but bad for authentication.

(iPhotos and Facebook's face tagging are all susceptible to the same thing. It's pretty easy to see if a given photo matches someone already on your friend list, but very hard, maybe impossible, to match accurately against a single worldwide global database. That's why it only ever suggests tagging a face as one of your friends, even if it's wrong. This whole thing also creates serious obstacles to those dystopian "we'll assign a social score to every citizen and track them all everywhere through public security cameras" plans.)

The fix for false positives is surprisingly easy, and lets you fix a bunch of the other problems with biometrics at the same time: combine it with factor #2, ie. something you have, like a card or a phone.

Instead of storing your fingerprints or retinal scan or facial structure in a big central database, just put it on a card or a phone, in a secure element. Then, when you want to authenticate, the scanner makes a connection to your device, sends the current biometric scan, and the card compares it to its single target (you), and chooses whether or not to unlock itself in response. If it does, it tells the scanner (usually with a cryptographic signature) that all's well.

This is much better: your search algorithm can be much worse. Even a 1 in 1000 false positive rate will be undetectable by random end users, even if it would be utterly hopeless in a million-person database. You don't need to put your biometrics in a central database, which is a big improvement when that database inevitably gets stolen. And best of all, a skilled attacker can't only clone your biometrics, because they won't have the key from your device. No more showing up at the secret facility with some synthetic rubber fingers and waltzing right in. That other factor (something you have... and will notice when it's stolen) goes a long way.

But what exactly is that second factor?

"Something you have" is not quite right

So much for biometrics, factor #3. But what about #2? "Something you have" is pretty straightforward, right? A card, a phone, an OTP fob, a U2F key, whatever. If you have it, that's an authentication factor.

Well, almost. We're going to need to be a little more specific in order to explain exactly why things work the way they do, and why some attacks are possible when you might think they're impossible.

First, let's be clear. "Something you have" is always an encryption key, a long sequence of bits, a large number. It's not a screwdriver, or an 1800s-style physical iron key (physical analogies for two-factor authentication all eventually turn out badly), or an ear infection. It's an encryption key. Let's call it that.

Secondly, it's not just any encryption key. Anybody can generate an encryption key if they have a big enough random number generator. Your encryption key is special. It's a key that you have previously agreed upon ("enrolled") with an authentication provider.

So let's change the definition. Factor #2 isn't something you have; it's just a particular very large number. Maybe you keep it on a device. Maybe you memorize it (if you can memorize 256 digits of pi, I guess you can memorize a 2048-bit RSA key, but I don't envy your key rotation strategy), and then it's something you "know." Maybe you get a QR code tattoo, and then it's something you "are." But it's a big number, and it's valuable only because the other end already knows, by prior agreement, that they will trust someone who has that number.

(Side note: instead of saving each enrolled key in a database, the authentication provider might sign your (public) key using a Certificate Authority (CA). The net result is the same: in order to get a signature from the CA, you had to enroll with it at some point in the past. The difference only affects the format of the backend authentication database on the server, which is opaque to us.)

From now on, factor #2 is "your previously-enrolled private key."

All factors are not created equal

An incorrect conclusion from the "something you know, something you have, something you are" definition is that you can pick any two of these and have a system that's about equally secure as any other combination.

Nope.

In fact, factor #2 - the previously-enrolled private key - is by far the safest of the three. This is why banks give you bank cards but are willing to "authenticate" those cards with (in the U.S.) a useless pen-and-paper signature, or (elsewhere) an easily-stolen-or-guessed 4-digit PIN.

Your factor #2 key can be easily replaced (and re-enrolled). You can have as many as you want, corresponding to as many pseudonymous identities as you want. It's a truly random stream of bits, not a weak one you composed because it would be easy to remember. It's easily and perfectly distinguished from everyone else's key. It can't be used to track you between use cases (unless your device is designed to leak it intentionally, sigh). With software configured correctly, it can't be phished. It can be stolen - but (if the hardware is designed right) your attacker has to steal it in the physical world, which eliminates Internet-based attackers, which is most of them. And if they steal your physical key card, you'll probably notice pretty fast.

Even ancient magstripe credit cards share most of these advantages. Their main weakness is that a physical "attacker" can clone rather than steal your card, which is much harder to detect. That's the only real advantage of "chip" cards: they can't be trivially copied. (The switch from signatures to PINs is mostly a distraction and adds little.) We make fun of old-fashioned banks for "still" accepting magstripe cards and being trapped in the past, but those cards are vastly more secure than almost anything else your non-bank service providers use.

(Let's skip over Internet shopping, in which you just type in the not-so-secret magic number from the back of the credit card and combine it with your not-so-secret postal code. Oh well. Sometimes convenience trumps security, even for a bank. But they still keep issuing those physical cards.)

Your phone is probably doing this right

By the way, this whole post is just repeating what experts already know. Your phone, especially if it's an iPhone, already benefits from people who know all this stuff.

Your iPhone contains a previously-enrolled private key, in a secure element, that can uniquely identify it to Apple. You can generate more of them, like the key used to encrypt your local storage. Actions on the key can be restricted to require a fingerprint first, and even the fingerprint reader uses its own previously-enrolled private key link to the secure enclave, preventing attackers from tearing apart your phone and feeding it digital copies of your fingerprint, bypassing the sensor. Plus, after a power cycle, the iPhone secure element refuses to unlock at all without first seeing your PIN, which an attacker can't lift from your hotel room or jail cell like they can lift your fingerprints. And it only lets an attacker try a few guesses at the PIN before it locks iself permanently.

FaceID is similar. It doesn't really matter if the fingerprint or FaceID algorithms have a fairly high false positive rate, because of all the other protections, especially that previously-enrolled private key. Somehow, an attacker either needs your phone, or needs to hack the software on your phone and wait for you to scan a biometric.

Awesome! Let's use our phone to authenticate everything!

Yeah, hold on, we're getting there.

That pesky "previous" enrollment

So here's the catch. The whole multi-factor authentication thing is almost completely solved at this point. Virtually everybody has a phone already (anyway, more people have phones than computers), and any phone can store a secret key - it's just a number, after all - even if it doesn't have secure element hardware. (The secure element helps against certain kinds of malware attacks, but factor #2 authentication is still a huge benefit even with no secure element.)

The secret key on your phone can be protected with a PIN, or biometric, or both, so even if someone steals your phone, they can't immediately pretend to be you.

And, assuming your phone was not a victim of a supply chain attack, you have a safe and reliable way to tell your phone not to authorize anybody unless they have your PIN or biometric: you just need to be the person who initially configures the phone. Nice! Passwords are obsolete! Your phone is all three authentication factors in one!

All true!

But... how does a random Internet service know your phone's key is the key that identifies you? Who are you, anyway?

The thing about a previously-enrolled private key is you have to... previously... enroll it... of course. Which is a really effective way of triggering Inception memes. Just log into the web site, and tell it to trust... oh, rats.

Authentication is easy. Enrollment is hard.

Which leads us to a surprising conclusion: security research has actually come really far. We now have good technology to prevent phishing while decreasing the chance of forgotten passwords (PINs are easier than passwords!) and avoiding all the privacy problems with biometrics. Cool! Why isn't it deployed everywhere?

Because enrollment remains unsolved.

Here are some ways we do enrollment today:

Pick a password at account creation time and hope you weren't being phished at that time. (Phishable. Incidentally, "password manager" apps are a variant of this.)
Get a security token (OTP, U2F, whatever) issued by your employer in person, after some kind of identity check. (Very secure, but tedious, which is why only corporations bother with it.)
Mail a token to your physical mailbox and hope nobody else gets there first. Banks do this with your credit card. (Subject to (eventually detectable) mail theft by criminals and roommates, but very effective, albeit slow.)
Log in once with a password, then associate a security token for subsequent logins. (Password logins are phishable.)
Your physical device (eg. iPhone) arrives with a key that the vendor (eg. Apple) already trusts, so at least it knows it's talking to a real iPhone. (Great, but it doesn't prove who you are.)
Send a confirmation message to your email address or phone number, thus ensuring you own it. (Inception again: this just verifies that you can log in somewhere else, with an unknown level of security. Email addresses and phone numbers are stolen all the time.)
Ask for personal information, eg. "security questions", your government SIN, your physical address, your credit card number. (These answers aren't as secret as you'd like to think.)

All the above methods have different sorts of flaws or limitations.

That said, we should draw attention to some special kinds of "enrollment" that are actually nearly perfect:

Setting a PIN on my iPhone, assuming the iPhone's supply chain was secure, is reliable and easy because of a physical link between it and me.
Setting fingerprint or face unlock on my iPhone is reliable and easy, again because of a physical (sensor) link between it and me.
My corporate-issued U2F token can issue signatures to my web browser because they are physically linked, and it can assume I plugged it into whatever computer on purpose.
Bluetooth devices make me do an explicit enrollment ("pairing") phase on both devices, which prevents attacks other than by people who are physically nearby at the time of initial pairing. (When available, a pairing PIN adds a small amount of security on top, but is usually more work than it's worth unless the NSA is parked outside.)
Registering a new iCloud account on my iPhone can be safe because the iPhone's built-in certificate can be used to mutually validate between Apple's auth server and the phone. (This is only safe if you create your account from a suitably secure iPhone in the first place. Securing a non-secured account later is subject to the usual enrollment problems.)
Registering additional Apple devices to your existing iCloud account, where Apple pops up a confirmation on your already-registered Apple devices and then makes you type an OTP into the new device. (The OTP in this case might seem extraneous, but it ensures that someone can't just trigger a request from their own device at the same moment as you trigger a request from yours, and have them both go through because you click confirm twice. It's also cool that they show the GPS location of the requester in the confirmation dialog, although that seems forgeable.)

...okay, I didn't expect this article to turn out as an iPhone ad, but you know what? They've really thought through this consumer authentication stuff. (To their credit, although Google is behind on consumer authentication, their work on U2F for corporate athentication is really great. It relies on the tedious-but-highly-effective human enrollment process.)

It seems to me that the above successful enrollment patterns all use one or more of the following techniques:

A human authenticates you and issues you a token (usually in person).
A short-distance, physical link (proximity-based authentication) like a biometric sensor, or USB or bluetooth connection.
Delegation to an existing authenticator (like a popular web service, eg. via oauth2, or by charging a credit card) to confirm that it's you. Once you have even a single chicken, you no longer have a chicken-or-egg problem. After delegating once, you can enroll a new token and avoid future delegations to that service if you want. (That might or might not be desirable.)

That's all I've got. In this article, I'm just writing about what other people do. I don't have any magic solutions to enrollment. So let's leave enrollment for now as an only-partially-solved, rather awkward problem.

(By the way, the reason gpg is so complicated and unusable is that the "web of trust" is all enrollment, all the time.)

But now that we understand about enrollment, let's talk about...

Why is U2F so effective?

Other than enrollment, using U2F (Universal Second Factor) is pretty simple. You get a physical key, plug it into a USB port (or nowadays sometimes a bluetooth link), and you press it when prompted.

When you press it, the device signs a request from any requesting web site, via the web browser, thus proving that you are you. (Whoever "you" are was determined during enrollment.)

The non-obvious best feature is that the signature only applies to a particular web domain that made the request, and the web browser enforces that the domain name is "real" by using HTTPS certificates. (Inception again! How do we know we can trust the web site? Because their HTTPS cert was signed by a global CA. How do we know which CAs to trust? A list of them came with our browser or our OS. How can we trust the browser or OS? Uh... supply chain?)

Anyway, put it all together, and this domain validation is like magic; it completely defeats phishing.

A good mental model for phishing is to imagine some person sitting there with their own web browser, but also operating a web site that proxies your requests straight through to the "real" web site, while capturing the traffic. So you accidentally visit gmail.example.com, which proxies to gmail.com. Somehow you are fooled into thinking you need to log in as if it's gmail. You type your username and password, which are logged by the proxy and sent along to gmail.com. Our hacker friend reads the username and password from their trace, and logs into gmail.com on their web browser. You've been phished!

Now let's add U2F. After you type your password, the site asks you to tap your U2F. You tap it, and generate a valid signature directed to... gmail.example.com, the site your browser confirmed as having made the request. They proxy that valid signature along to gmail.com, but it's useless: gmail.com doesn't trust signed authenticators directed at gmail.example.com. The phishing has been blocked completely.

(By the way, OTP - any device that generates a number that you have to type in - doesn't prevent phishing at all. At least, not if the attacker can do man-in-the-middle attacks, which they probably can. In our example, you'd happily type the OTP into gmail.example.com, for the same reason you typed your password, and it would proxy it to gmail.com, and bam, you lose.)

(I mentioned Apple's authenticator above, which requires you to type an OTP. It doesn't have this weakness, because they are also authenticating using a U2F-like mechanism behind the scenes. Forcing you to also type the OTP is an extra layer of security, part of a very subtle enrollment process that defends against even rarer types of attack.)

Note also that you can unplug your U2F token and plug it into any other computer and web browser instance. Without any special pairing process, it can trust the new web browser and the web browser can trust the U2F. Why? Because of that physical link again. Somebody plugged this U2F key into this browser. That means they intended for them to trust each other. It's axiomatic; that's the design.

For this reason, pure U2F doesn't work well without a second factor, such as a password. Otherwise, if someone steals your U2F key, they could impersonate you by plugging it into their own computer, unless there's something else proving it's you.

The reason these two factors are so effective is because of the threat model. Basically, there are physical attackers (eg. purse snatchers) and there are Internet attackers (eg. spammers). Physical attackers can steal your U2F device, but they also need to see you type your password, and the U2F is traceable when used, and the attack is unscalable (you need a physical human to be near you to steal it). On the other hand, Internet attackers can easily phish your password, but can't steal your U2F.

Skilled, well-funded attackers with physical presence can do both. Sorry. But most of us aren't important enough to worry about that.

Can I use my phone as a U2F?

Learning about the above led me to ask a question many people have asked: why do I need the stupid physical security token, which is only plugged into one computer at a time and which I'm guaranteed to misplace? Can't my phone be the security token?

This is a good question. Aside from enrollment being generally hard, the stupid, expensive physical security token is, almost certainly, the thing preventing widespread adoption of U2F. I doubt it'll ever catch on, other than for employees of corporations (who pay you for the inconvenience and have good ways to enroll a new token when you inevitably lose yours).

It ought to be easy, right? A U2F is really just a secure element containing a previously-enrolled private key. As established earlier, you can put a private key anywhere you want. Heck, your iPhone has one already, protected by a PIN or fingerprint or face. We're almost there!

Almost. The thing is, when I'm browsing on my computer with my phone in my pocket, it's not my phone that needs to authenticate with the web site: it's my computer. So my phone and computer are going to have to talk to each other, just like a USB U2F and a computer talk to each other.

Both my web browser and my phone are connected to the Internet, so that's the most obvious way for them to talk. (If my computer weren't on the Internet, it presumably wouldn't need to authenticate with a web site. And my phone probably has a cellular data plan, so situations where it's on the Internet are "mostly" a superset of situations where my computer is.)

So all I need to do is generate an encrypted push request from my browser, through the Internet, to my phone, saying "please sign an authorization for gmail.com." Then I tap my phone to approve, and it sends the answer back, and the web browser proceeds, exactly as it would with a USB-connected U2F token. Right?

Almost. There's just one catch. How does my phone know the request is coming from a trusted web browser, and not that browser the phishing attacker, up above, was operating, or from some other random browser on the Internet? How can my phone trust my browser?

Inception.

You have to... previously enroll... the browser's secret key... with your phone's U2F app.

Okay, okay, we can do this. Recall that there are three ways to do a safe enrollment: 1) a physical person gives you a key; 2) a direct physical link; 3) delegation from an existing authenticator. Traditional USB U2F uses method #2. When you physically plug a U2F into a new web browser, it starts trusting that web browser. Easy and safe.

But your computer and your phone aren't linked (if they are, then problem solved, but they usually aren't). Enrollment between your browser and your phone can be done in various different ways: you could make a bluetooth link, or generate a key on one and type it into the other, or scan a QR code from one to the other. Let's assume you picked some way to do this, and your browser is enrolled with your phone.

Now what? Okay, now when I want to log in, I must know my password and must physically possess my computer (containing the enrolled browser), and my phone with its U2F app. I type my password, which triggers a U2F request from the web site to the browser, which signs and pushes a signing request to my phone, which pops up an acknowledgement, which I tap, and then it signs and sends an answer back to my browser, which sends it back to the web site, and success!

Except... something's fishy there. That was more complicated than it needs to be. Notice that:

The browser is trusted by the phone U2F app because of a previous enrollment.
The phone U2F app is trusted by the web site because of a (different) previous enrollment.
My screen tap (or fingerprint scan) is trusted by the U2F app because of my physical presence (or another previous enrollment).

Why do I even need my phone in this sequence? I've already enrolled my browser; it's trusted. During enrollment, we could have given it a key that's just as useful as a U2F device, and we could have done that precisely as safely as enrolling a new U2F device. (It's nice if my computer has a secure element in which to store the key, but as established above, that only matters for security if my computer gets hit by malware.)

We could just do this instead: I type my password. Web site generates U2F request. Web browser signs the request using its previously-enrolled private key, enrolled earlier using my phone. Login completes, and phishing is prevented! Plus we don't rely on my phone or its flakey Internet connection.

Great, right?

(Note that while this does indeed prevent phishing - the biggest threat for most organizations - it doesn't prevent your roommate, spouse, or 6-year old child from watching you type your password, then sometime later unlocking your computer and using its already-enrolled web browser to pretend to be you. By leaving your 30-pound desktop computer just lying around, you've let your second factor get stolen. In that way, phones are better second factors than computers, because we're so addicted to them that we rarely let them out of our sight. That doesn't stop your children from borrowing your fingerprints while you nap, however.)

There is, of course, still a catch. If you do this, you have to enroll every browser you might use, separately, using your phone, with one of these tedious methods (bluetooth, typing a key, or QR code scan). That's kind of a hassle. With a USB U2F key, you can just carry it around and plug it into any computer. Even better, when you stop trusting that computer, you just unplug the token, and it cancels the enrollment. With a physical device, it's super easy to understand exactly what you've enrolled and what you haven't. (With a software key, the server needs a good UI to let you un-enroll computers you don't need anymore, and users will surely screw that up.)

What people tend to miss, though, is that this enrollment is necessary whether or not you send a push notification to the phone during login. The push notification is only secure if this specific browser instance is enrolled; but if this browser instance is enrolled, and my computer is not easier to steal than my phone, then the push notification adds no extra security. The enrollment was the security.

And that's where I'm stuck. I've more or less convinced myself that phone-based OTP (prone to phishing) or phone-push-based U2F (not useful after initial enrollment) add no interesting security but do make things harder for end users. I guess they call that "security theatre." Meanwhile, physical U2F tokens are unlikely to become popular with consumers because they're inconvenient.

At least that conclusion is consistent with the state of the world as it exists today, and what the market leaders are currently doing. Which means maybe I've at least explained it correctly.

(Special thanks to Tavis Ormandy, Reilly Grant, and Perry Lorier for answering some of my questions about this on Twitter. But any mistakes in this article are my fault, not theirs.)

Misadventures in process containment

Mon, 14 Jan 2019 04:49:40 +0000

I've been working on a series of tutorials using redo for various use cases. (One of the most common user requests is more examples of how to solve real-world problems with redo. The problem with extremely flexible tools is that it can be hard for people to figure out how to start.)

The most recent in the series is a a tutorial on building docker and kvm containers from scratch using redo. I think it turned out pretty well. Maybe the tutorial deserves a disclaimer, though: is this really something you should do?

Good question. I don't know.

You see, there are a few "standard" ways to build containers nowadays, most commonly using Dockerfiles and the docker build command. It's not at all obvious that we need another way to do it. Then again, it's not obvious that we don't.

Wait, let's back up a bit. Where are we and how did we get here?

The idea of isolated chroot-based containers has been around for a very long time. In my first startup, we even had a commercial version of the concept as early as 2005, first implemented by Patrick Patterson, called Nitix Virtual Server (NVS). The original idea was to take our very-locked-down Linux-based server appliance and let you install apps on it, without removing the (very useful for security and reliability) locked-down-ness of the operating system. Nitix was a very stripped down, minimal Linux install, but NVS was a full install of a CentOS-based system in userspace, so you could do basically anything that Linux could do. (You might recognize the same concepts in ChromeOS's Crostini.) We could start, stop, and snapshot the "inner" NVS operating system and make backups. And most importantly, if you were an app developer, you could take one of these pre-installed and customized snapshots, package it up, and sell it to your customers. Hopefully with one of our appliances!

Eventually one of these packaged apps became appealing enough that the app maker, who was much larger than us, decided to acquire our company, then (as often happens) accidentally killed it with love, and that was sadly the end of that branch of the container evolutionary tree.

Our containers were installed on physical appliance hardware - another branch of evolution that seems to have died. Nobody believes anymore that you can have a zero-maintenance, self configuring, virus free appliance server that runs on your office network without needing a professional sysadmin. Come to think of it, most people didn't believe us back then either, at least not until after seeing a demo. But whatever, the product doesn't exist anymore, so nowadays they're right.

In any case, the modern solution to this is for everybody to host everything in the Cloud. The Cloud has its own problems, but at least those problems are fairly well understood, and most importantly, you can pay by the minute for a crack team of experts, the ones who own the servers, to fix problems for you. For most people, this works pretty well.

But back to containers. The way we made them, long ago, was a bit ad-hoc: a person installed a fresh NVS, then installed the app, then wrote a few scripts to take system configuration data (user accounts, remember when we used those? and hostnames, IP addresses, and so on) and put them in the right app-specific config files. Then they'd grab a snapshot of the whole NVS and distribute it. Making new versions of an app container involved either making additional tweaks to the live image (a little risky) and re-snapshotting, or having a human start over from scratch and re-run all the manual steps.

Those were simpler times.

Nowadays, people care a lot more about automated builds and automated testing than they did back in 2005, and this is a big improvement. They also collaborate a lot more. Docker containers share almost the same basic concepts: take a base filesystem, do some stuff to it, take another snapshot, share the snapshot. But it's more automated, and there are better ways to automate the "do some stuff" part. And each step is a "layer", and you can share your layers, so that one person can publish a base OS install, another person can install the Go compiler, another person can build and install their app, and another person can customize the app configuration, and all those people can work at different companies or live in different countries.

As a sign of how out of touch I am with the young'uns, I would never have thought you could trust a random unauthenticated person on the Internet to provide a big binary image for the OS platform your company uses to distribute its app. And maybe you can't. But people do, and surprisingly it almost never results in a horrible, widespread security exploit. I guess most people are surprisingly non-evil.

Anyway, one thing that bothered me a lot, both in the old NVS days and with today's Dockerfiles, was all the extra crud that ends up in your image when you install it this way. It's a whole operating system! For example, I looked at what might be the official Dockerfile for building a MySQL server image (although I'm not sure how one defines "official") and it involves installing a whole C++ compiler toolchain, then copying in the source code and building and installing the binary. The resulting end-user container still has all that stuff in it, soaking up disk space and download time and potentially adding security holes and definitely adding a complete lack of auditability.

I realize nobody cares. I care, though, because I'm weird and I care about boring things, and then I write about them.

Anyway, there are at least two other ways to do it. One way endorsed by Dockerfiles is to "skip" intermediate layers: after building your package, uninstall the compiler and extra crap, and then don't package the "install compiler" and "install source code" layers at all. Just package one layer for the basic operating system, and one more for all the diffs between the operating system and your final product. I call this the "blacklist" approach: you're explicitly excluding things you don't want from your image. Dockerfiles make this approach relatively easy, once you get the hang of it.

A more obsessive approach is a "whitelist": only include the exact files you want to include. The trick here is to first construct the things you want in your final container, and then at the end, to copy only the interesting things into a new, fresh, empty container. Docker doesn't really make this harder than anything else, but Docker doesn't really help here, either. The problem is that Docker fundamentally runs on the concept of "dive into the container, execute some commands, make a snapshot" and we don't even have a container to start with.

So that's the direction I went with my redo tutorial; I built some scripts that actually construct a complete, multi-layered container image without using Docker at all. (To Docker's credit, this is pretty easy, because their container format is simple and pretty well-defined.) Using those scripts, it's easy to just copy some files into a subdirectory, poke around to add in the right supporting files (like libc), and then package it up into a file that can be loaded into docker and executed. As bonus, we can do all this without being the 'root' user, having any docker permissions, or worrying about cluttering the local docker container cache.

I like it, because I'm that kind of person. And it was a fun exercise. But I'm probably living in the past; nobody cares anymore if it takes a few gigabytes to distribute an app that should be a few megabytes at most. Gigabytes are cheap.

Side note: incremental image downloads

While we're here, I would like to complain about how people distribute incremental changes to containers. Basically, the idea is to build your containers in layers, so that most of the time, you're only replacing the topmost layers (ie. your app binaries) and not the bottommost layers (ie. the OS). And you can share, say, the OS layer across multiple containers, so that if you're deploying many containers to a single machine, it only has to download the OS once.

This is generally okay, but I'm a bit offended¹ that if I rebuild the OS with only a few changes - eg. a couple of Debian packages updated to fix a security hole - then it has to re-download the whole container. First of all, appropriate use of rsync could make this go a lot more smoothly.

But secondly, I already invented a solution, eight years ago, and open sourced it, and then promptly failed to document or advertise it so that (of course) nobody knew it exists. Oops.

The solution is something I call bupdate (a mix of "bup" and "update"), a little-known branch of my bup incremental backup software, which I've written about previously.

Unlike bup, the bupdate client works with any dumb (static files only) http server. bupdate takes any group of files - in this case, tarballs, .iso images, or VM disk images - runs the bupsplit algorithm to divide them into chunks, and writes their file offsets to files ending in .fidx (file index, similar to git's .idx packfile indexes and bup's .midx multi-pack indexes), which you then publish along with the original files. The client downloads the .fidx files, generates its own index of all the local files it already has lying around (eg. old containers, in this case), and constructs exact replicas of the new files out of the old chunks and any necessary newly-downloaded chunks. It requests the new chunks using a series of simple HTTP byterange requests from the image files sitting on the server.

It's pretty neat. There's even an NSIS plugin so that you can have NSIS do the download and reassembly for you when installing a big blob on Windows (which I implemented for one of our clients at one point), like for updating big video game WAD files.

(By the way, this same technique would help a lot with, say, apt-get update's process for retrieving its Packages files. All we'd need to do is upload a Packages.fidx alongside the Packages file itself, and a new client which understood bupdate could use that to retrieve only the parts of the Packages file that has changed since last time. This could reduce incremental Packages downloads from several megabytes to tens of kilobytes. Old clients would ignore the .fidx and just download the whole Packages file as before.)

bupdate is pretty old (8 years now!), and relies on an even older C++ library that probably doesn't work with modern compilers, but it wouldn't be too hard to rejuvenate. Somebody really ought to start using it for updating container layers or frequently-updated large lists. Contact me if you think this might be useful to you, and maybe I'll find time to bring bupdate back to life.

gzip --rsyncable

On that note, if you haven't heard of it already, you really should know about gzip --rsyncable.

It's widely known that gzip'd files don't work well with rsync, because if even one byte changes near the beginning of the file, that'll change the compression for the entire rest of the file, so you have to re-download the whole thing. And bupdate, which is, at its core, really just a one-sided rsync, suffers from the same problem.

But gzip with --rsyncable is different. It carefully changes the compression rules so the dictionary is flushed periodically, such that if you change a few bytes early on, it'll only disrupt the next few kilobytes rather than the entire rest of the file. If you compress your files with --rsyncable, then bupdate will work a lot better.

Alternatively, if you're using a web server that supports on-the-fly compression, you can serve an uncompressed file and let the web server compress the blocks you're requesting. This will be more byte-efficient than gzip --rsyncable (since you don't have to download the entire block, up to the next resync point), but costs more CPU time on the server. Nowadays, CPU time is pretty cheap and gzip is pretty fast, so that might be a good tradeoff.

Footnote

¹ When I say I'm offended by the process used to update containers, it's not so much that I'm offended by people failing to adopt my idea - which, to be fair, I neglected to tell anyone about. Mostly I'm offended that nobody else managed to invent a better idea than bupdate, or even a comparably good idea^2,3, in the intervening 8 years. Truly, there is no point worrying about people stealing my ideas. Rather the opposite.

² Edit 2019-01-13: Eric Anderson pointed me to casync, which is something like bup and bupdate, and references bup as one of its influences. So I guess someone did invent at least a "comparably good idea." I think the bupdate file format is slightly cuter, since it sits alongside and reuses the original static files, which allows for cheap backward compatibility with plain downloads or rsyncs. But I'm biased.

³ Edit 2019-01-13: JNRowe points out a program called zsync, which sounds very similar to bupdate and shares the same goal of not disturbing your original file set. In fact, it's even more clever, because you can publish a zsync index on one web site that refers to chunks on another web site, allowing you to encourage zsync use even if the upstream maintainer doesn't play along. And it can look inside .gz files even if you don't use gzip --rsyncable! Maybe use that instead of bupdate. (Disclaimer: I haven't tried it yet.)

Honourable Governance

Sun, 09 Dec 2018 12:46:00 +0000

From a book I've been reading:

Power is certainly important, particularly in dictatorships, in places where constitutions, laws, unwritten rules, traditions and understandings don't count. But in a healthy democracy, power is a surprisingly limited element. And the unwritten conventions, understandings, forms of respect for how things are done, for how citizens relate to government and to each other, are surprisingly important. Why? Because if democracy is only power, then what we are left with is a system of deep distrust. Why? Because if only power matters - even if it is the result of an election - then the government feels that it has a mandate to do whatever it wants, that the law is there principally to serve power. If democracy is only about winning power and using it, then it has been deformed into a denial of society and of the idea of responsible citizenship.
And that is the increasingly common characteristic of government, even in democracies. Only power matters. This is partly the outcome of government being de-intellectualized. Elections are now thought to be unsuitable moments for real debates over ideas. In between these elections the focus is on administrative problems - legalistic, managerial undertakings. In this case, real debates over ideas are unnecessary because the decision about power was taken on election night. There is nothing, therefore, to debate. Worse still, the efficient putting in place of programs to be administered can only be made inefficient by debate.
And so we are witnessing a growth in the Napoleonic or Mussolinian corporatist idea that when citizens vote in an election, it is actually an all-purpose referendum or plebiscite. It is then the winner's job to get on with running things. First win an election, then administer as you wish. Omnibus bills are one of the ways you can speed things up; they are a great way to convert the deeply inefficient process of democracy, with all its thinking, debating and complex differences of opinion, into a sort of shop-floor system of utilitarian efficiency. Therefore, once elected, a government has broad, unlimited permission.
[...]

In any case, today's plebiscitary approach is both populist and anti-democratic. What it amounts to is this: We won the election. We have power. It is now a matter of administrative efficiency. We can do what we want.

- John Ralson Saul, The Comeback, 2014

This finally put into words a concept I've been struggling with for a long time: the idea that ethics and responsibility come with (should come with) political power, and how they seem to be on the decline lately.

For example, I recently learned, to my horror, about the concept of a Frankenstein Veto, where in some U.S. states, the governor can just delete words (eg. "not") from a bill passed by the legislature, producing and then approving a bill with a completely different meaning than was intended.

The exact legal machinations of this are beside the point. The point is, politicians increasingly feel that they "first win an election, then administer as you wish." That almost sounds logical, so why does it feel so wrong?

Because it is wrong. The job of an elected official isn't to do whatever they want. It's to figure out what the people want (or need), and to deliver that, in accordance with principles and ethics. This is a surprisingly selfless expectation: sometimes the right thing to do is the opposite of what you want to do. And it can be hard to figure out what's right, which is why we have debates, and why we listen to our opponents in those debates, even when we have a majority and they're "merely" the opposition.

Even direct polling on issues doesn't always work, because sometimes issues are too subtle for the population to simply vote yes or no; that's why we elect representatives who will presumably do their homework, consult the right people, and figure out the right, complex answer. At least one thing is clear: if the legislature vigorously debates an issue and decides one thing, a governor passing a law that does the opposite - or even rejecting the one that was so carefully produced - is just not right.

I've been struck many times by the way the U.S. system of political "checks and balances" treats politicians like children: none of them can be trusted, so we must restrict them at every opportunity. The underlying assumption is that they will all operate without honour. As most of us eventually learn in our human relationships, if you treat people as children - or worse, as actively malicious adults - then they will tend to meet your expectations. The great irony is that in countries with less of a focus on checking "power" (such as Canada), politicians seem to abuse their power less often and less dramatically.

The above quote made me think that maybe the secret recipe is some combination of traditions, conventions, understanding, and respect. Maybe those are more effective balances to power than any system designed to simply neutralize it. And maybe this was obvious to the people who designed it, and we've forgotten, long ago.

Don't delegate the most important thing

Wed, 05 Dec 2018 02:49:00 +0000

A while ago I was discussing hiring plans with a co-founder of a recently-funded startup. Their story was something like this:

"The company is growing very fast and we now have about a dozen people. We have tight deadlines, and none of the founders has much experience in project management or people management. So, we're trying to hire an experienced project manager and an experienced engineering lead / people manager."

This course of action is pretty common. It also always makes me nervous, but I haven't thought much about it until now. This discussion finally forced me to clarify my thoughts on the subject.

First of all, I asked the co-founder what their role is. The answer was a good one, albeit vague, as startup roles always are: "I don't really know. It's frustrating that I never know. But I do know my job is whatever it takes to make the company successful."

Well, what are the most important problems that the other co-founders aren't solving? The answer came quickly: "We don't have a clear idea of our project schedule. And nobody is managing interpersonal, cultural, social issues. There are lots of other problems, but people are handling those." So those were the top two. Nobody was managing those, and it was getting serious.

This reminded me of some advice I must have heard somewhere, but no longer remember where: Don't delegate the most important thing. Or maybe it was a 1980's management book: Don't outsource your core competency.¹

This advice is counterintuitive. At first glance it sounds like it should be intuitive, but once you think about it, it's not. The most important thing - whatever it is! - needs to be done well. Shouldn't we hire the best person possible to handle it? And isn't it vanishingly unlikely that the person already works here?

You're right! In an ideal world, you would get the best, most motivated, most passionate project manager in the world to manage your project. And the best, most motivated, most empathetic, most wise people manager to manage your people. But the real world is rarely ideal. You are unlikely to find the best person in the world on short notice. You probably can't afford to pay them what they deserve. In a wide world where they have their choice of exciting projects, they (statistically) probably don't care enough about your project anyway. And because they're so good, they're probably accustomed to leading huge teams with huge problems that need huge solutions.

In short, you aren't likely to hire the ideal person. You can probably hire a pretty good person. Or you might get unlucky and hire a terribly mismatched person.

That variability is a huge problem. Even if the average person you could hire might be better than you, the standard deviation is very high. It's essentially a random chance, a lottery ticket. Do you want to buy a lottery ticket for the (self-selected) most important thing in your entire domain, whatever it is?

The alternative is to find a person with less specific expertise, but who can understand the company vision, connect with everyone on the team personally, be highly invested in the shared outcome (not just personal gain), and be so passionate about the project that they'd be willing to step aside if they can someday find someone who is a better fit. As a bonus, this person already works at your company: it's you.

Relatively speaking, it's actually pretty easy to learn project management and scheduling or people management. It takes time and effort, but not much more, to become passably decent at either one. Whereas you can't buy shared vision, personal knowledge, shared motivation, or humility, for any amount of money.

Once the schedule or the management of small teams is no longer your most critical problem, then delegate those. Better still, by then you will know how to do the jobs, so you will be very good at interviewing your replacement.

Footnotes

¹ "Core competencies" is one of those once-useful terms that has been badly diluted through overuse. It mostly just means "things that uniquely make you special compared to your competitors." If you're a software company, that's probably your software.² So buy some commodity package (or service) to do your accounting, which doesn't differentiate you, and hire people to write your software, which does.

² Even at a software company, sometimes your software is crappy and your customer service or sales organizations are your real advantage. Confusion about this has led to horrible strategic blunders. IBM's core competency, for example, clearly is not software.

Xnest, Xephyr, ChromeOS, synergy, and syncing some clipboards

Sat, 24 Nov 2018 13:56:30 +0000

I recently decided to switch my laptop from a Macbook to a Chromebook, partly because Apple's keyboards are so terrible lately, and partly because ChromeOS is suddenly useful now that they invented Crostini.

(Some people ask why I, a person who actually knows how to use Linux and has debugged wifi drivers and XF86Config files, would want to use a locked-down desktop Linux variant instead of just installing Debian or something. And I do install Debian, on desktop hardware. But on a laptop, hardware support is paramount: external monitors (eg. for presentations), bluetooth audio (for music while travelling), long battery life, and rapid, non-crashy suspend/resume, are all really important to me. ChromeOS actually does all that stuff reliably nowadays, because they design the OS and the hardware at the same time. Debian can't compete with that.)

One showstopper for me when I'm trying to do software development, however, is having a proper window manager, which is to say, one that I can run without resorting to a mouse or touchpad. Because I am old and crusty and unreasonably opinionated, the one I want to run is ion1. I had it working on MacOS, but I wanted it on ChromeOS.

Now, modern ChromeOS uses Wayland as its display manager, not X11, which is of some concern becuse ion1 stopped evolving more than a decade ago (which is how I like it) and therefore only understands X11. Also, ChromeOS provides a Wayland compositor and doesn't let you replace it from Crostini, even though they do let you securely launch Wayland and X11 windows from Crostini (which is pretty cool).

Do we give up? No! The "obvious" "solution" is Xnest, an "X proxy" from before the dawn of time, which puts all your windows inside one big window. So I created one big full-screen Xnest window (managed by Wayland), and then ran ion1 and a bunch of rxvt terminals inside.

This actually worked almost right, except: ion1 doesn't support these fancypants client-rendered fonts. No. It's old and crusty, like me. It expects the X server to render its fonts. And unfortunately, ChromeOS contains only about four fonts in its X server, all of which are hopelessly microscopic on the 200dpi screen in my Chromebook. Oops.

Luckily, about 11 years ago, slightly after the dawn of time, someone else didn't like some Xnest limitations and made Xephyr, which apparently is more of a framebuffer and less of a proxy, the upshot of which is that it renders its own "server side" fonts (which from Wayland's point of view are on the client side, but from ion1's point of view are definitely on the server side). As a bonus, thanks to xrandr, it understands the idea of having its window resized, so I can use the handy ChromeOS "full screen" key and have it do something nice. Or drag it to an external monitor and get decent results.

I didn't try to do anything with 3D, but nominally Xephyr can do that too. But Crostini supposedly can't. I don't know and I don't really care, I'm just trying to run some terminals here.

Xephyr and its DPI calculation

One weird problem I've had with Xephyr is that whatever it's doing to calculate "dots per inch" (as reported by xdpyinfo) is completely insane. It starts off with a value that is definitely not the same as its host display, and then if you resize the window, it just changes the value and gets more and more confused. This is bad for people who want to specify their font size in points so that fonts will be roughly the same size no matter what size the display is.

As far as I can tell, this is just a bug in Xephyr. But if anybody knows what's going on or (especially) how to fix it, I'd love to know. Meanwhile, I learned to specify my font sizes in pixels instead of points.

Cut and paste

An even more annoying problem, which deserves its own section, is the question of how to deal with cut-and-paste between my Xephyr+ion1+rxvt session and the toplevel Wayland session. This is needed for two reasons:

I want to copy URLs and text between my web browser and my terminals.
I want to be able to run multiple Xephyr sessions and share text between them.

By default, Xephyr appears to have no clipboard sync at all between its internal clipboard and its host server's clipboard. That's no fun. (To their credit, ChromeOS does seem to manage to sync the clipboard between Wayland and its toplevel XWayland session, which is essential if we want anything to work. It's just Xnest and Xephyr that break the chain.)

Now, there are a few things you should know about X11 clipboards. The canonical explanation is jwz's X Selections, Cut Buffers, and Kill Rings, which is quite excellent and gives some background on how it's not the X11 clipboard that's crazy, it's all the apps using it.

So anyway, with that background in mind, all we need to do is magically keep the clipboard in sync between :0 (the toplevel XWayland server, which is synced with Wayland and Chrome), and :1 (inside my Xephyr server), and ideally :2 .. :n (inside other nested Xephyr servers). How hard can it be?

Well, apparently it can be hard. The best answers I could find on the Internet (which I won't link to, because they suck) are:

Run a script that uses xclip to periodically grab the clipboard content from each server. If one server has different clipboard content than the currently-expected content, then copy it to the other server. This method has a few problems: first, you have to choose a periodicity for the sync process, which is inevitably either annoyingly long or battery-killingly short. Second, the "content based" sync decision is rather error prone and results in potentially unstable race conditions, especially with multi-way sync. And third, typical implementations are a little too pushy about copying the clipboard data to all screens: jwz's article talks about this problem in refence to the "X cut buffer" support before the new-style support was added. If you highlight/copy text frequently or in large volumes, it's pretty wasteful to copy it to other screens before it's needed for pasting.
Run synergy, a tool that lets you seamlessly extend your mouse/keyboard/clipboard across multiple displays on multiple computers. This was very tempting, despite being severe overkill (I don't want to extend my mouse and keyboard, just my clipboard). Unfortunately, it didn't work. It almost worked. But it didn't.

Luckily(?) for you, I spent quite some time diagnosing why synergy didn't work for me in my use case. The symptom was that it would sync the clipboard in only one direction (say A->B), and only the first time I copied something. If I copied another thing on A, the clipboard on B would not be updated. To make it update, I had to copy something on B (which always fails to sync to A), and then copy something on A (which would work).

How hard can it be? I thought to myself, again, foolishly, and decided to read the source code.

Now, the synergy source code is actually pretty good. It has a nice abstraction layer for the various clipboard types in X, MacOS, and Windows. It's pretty easy to follow. It has a bit too few debug trace messages, but okay, those are easy enough to add as we go.

Unfortunately, synergy's clipboard support has two fatal design flaws:

Like the periodic xclip case above, it grabs a copy of clipboard data right away when the clipboard ownership changes. It's better than a naive xclip script, because it actually gets a notification when the clipboard ownership changes, rather than polling periodically. Unfortunately, those notifications are also its downfall. See, in X11, there is only a clipboard notification when the clipboard owner changes, not when the content changes. If I copy text from rxvt, it will grab the clipboard. Synergy will notice this and read the clipboard. But if I then copy different text in rxvt, the owner doesn't change, so there is no notification, so Synergy doesn't re-copy it. That explains why it only worked the first time. (It also explains why copying on B and then on A causes it to work again exactly once: the clipboard ownership changes.) (This bug may not be visible on all terminals. If rxvt would give up the clipboard, then take it back, every time I made a copy, it would work around this problem.) (I think VNC's clipboard sync has/had the same problem.)
Synergy, because it's mostly a keyboard/mouse sharing app, maintains the concept of a "current screen." That is, it watches which screen currently has the mouse pointer, and only replicates the clipboard to that screen. This is a performance optimization: since it (like the poorly designed "x cut buffer" mentioned by jwz) takes a copy every time the clipboard changes, it doesn't want to replicate this to screens where you're not using it. Unfortunately, since my screens are nested, I had to disable the keyboard/mouse sharing feature, which also leaves the "current screen" incorrect exactly half the time, which is why the clipboard fails to replicate from B->A and only works from A->B.

I was willing to try to fix some minor clipboard bugs in synergy, but I gave up when I realized this design (grab and replicate the content as soon as clipboard owner changes) was never going to work well with rxvt. That's when I gave up and decided to write my own trivial clipboard syncing tool, based on all the otherwise-useless trivia I had acquired while investigating the above.

The result is xclipsync, and, other than omitting non-text clipboard formats, I think I did it right.

It starts up by taking ownership of the clipboard on display A.
When it loses clipboard ownership on A, it takes ownership of the clipboard on display B.
When it loses clipboard ownership on B, it goes back to step 1.
When it receives a request for clipboard contents (which should be someone requesting a paste), it then reads the clipboard content from the display that it doesn't currently own, and forwards it along.

And that's it!

This avoids the problem of a single owner changing their clipboard content (since it grabs content only on demand). It doesn't do extra work if you copy content without pasting. It actually does nothing at all if you do a lot of work on one display: it loses the clipboard content on that display, which means it does nothing at all until you use the clipboard on the other display. It doesn't ever poll anything, so there are no arbitrary delays or race conditions.

And best of all, this algorithm works even for multi-way sync. You can run parallel instances of xclipsync between any two displays, and as long as you don't create any bridging loops, it will do the right thing across all of them. That is, exactly one display will "own" the clipboard, and all the other ones will copy from it. This works because every time you copy something from a new display, exactly one xclipsync instance will lose ownership, which causes it to assert ownership on exactly one display. If another xclipsync is syncing with that display, it will then lose ownership, and assert ownership on exactly one other display, and so on. As long as there are no loops, this process will terminate, and it'll do so very efficiently.

Things that could be better

There's no particular reason xclipsync can't support non-plaintext clip formats. I just didn't implement it, because I didn't need anything but text, since my Xephyr session is just terminals anyway.

xclipsync currently uses tcl/tk to take clipboard ownership (yes!). This would have been unnecessary if xclip had just one more feature: the ability to run a command at paste time, rather than always reading clipboard content from stdin at startup time. Then xclipsync would have been just a couple of (foregrounded) alternating xclip calls in a loop.

Xephyr probably should just implement this exact clipboard syncing protocol internally.

Note that it appears ChromeOS+Wayland is actually implementing some other kind of clipboard sync between Wayland and the toplevel XWayland server. When xclipsync tries to take ownership of the XWayland clipboard, it immediately experiences one "paste" operation and then loses ownership. This might be related to WAyland's inter-process security isolation features. In any case, xclipsync reacts as usual (giving clipboard ownership to XWayland and proxying requests to XWayland from other displays that want to paste) and all is well.

I really wish Alt-Tab would work even when the Xephyr instance is fullscreened in ChromeOS. I understand why they want to let me capture Alt-Tab in my full-screen X apps, but also... I don't want to.

The marketing-driven "Assistant key" on the Pixelbook is a user-hostile disaster in its current form. On the other hand, if they would let me remap it to, say, Meta, it would instantly redeem itself.

The good feeling from bad feedback

Sat, 17 Nov 2018 04:36:05 +0000

I'm working on (statistically speaking) my least interesting work in years.

Previously I spent time doing wifi drivers, boot scripts, logfile processors, payment systems, and project estimation, so one might reasonably have assumed I can't get much more boring. But hah! We have standards to exceed.

Here's what's weird though: I'm enjoying it.

There's this Feeling I get, very rarely, when I'm sure I'm on the right track. Over the last several years, I've almost had the Feeling occasionally, but not very often. It's been so long since I felt it that I actually forgot what it feels like. Talking to software people, I strongly suspect some have never felt it at all.

A common question, when recent graduates ask me for advice after landing their dream job, is of the form, "Is it... always like this? When I started to learn coding, I really liked it. I kind of assumed work would be... better." What they mean isn't that coding sucks, but that their project is unfulfilling. They suspect the Feeling exists. Maybe they remember having it as a kid. They thought they were almost there again. Life is all lined up: great school, great grades, great employer at a great salary, great co-workers. And then... nothing. Is it always like... this?

No, it's not. Not always.

But a lot.

...

Yesterday, someone on a mailing list told me (politely)¹ that my documentation sucked. A few days before, someone called my code spaghetti. And the thing is... YES. Yes, I am a pretty good writer, but that documentation is not my best work. That code is some of the best I've written, but it's also the most incomprehensible, and I've known for years how to make it better, but I haven't gotten around to it. It's refreshing to have a project with no stakes, with volunteers who care about elegance, where I can hear stuff like that. I live on that. It's what makes me go, and when I finally do get it right, it means the positive feedback is real.

It's been years since someone told me my work sucked. And that was one thing, I can live with that, it's West Coast People, but in the last year or so, the compliments have been egregious. People's acceptance of my work and my opinions had a lot more to do with my reputation (among a certain small group) than about my quality. It's easy to get addicted to that, to let it take the place of the Feeling, but in the end it's just candy. The sugar high fades, and too soon, you need more sugar.

That's why I had to change gears. The withdrawal process has been a bit painful, but it's worth it. Maybe my project won't amount to anything. But if it dies, at least it'll die beautiful.

Footnote

¹ Just to be clear, constructive negative feedback is important, but being a jerk is not. There's no need to go on a swearing/ranting angry rampage when things go wrong. It doesn't help anyway. But pretending bad things are good things doesn't help either.

mtime comparison considered harmful

Sat, 17 Nov 2018 06:08:42 +0000

tl;dr: Rebuilding a target because its mtime is older than the mtimes of its dependencies, like make does, is very error prone. redo does it better, and so can you.

A recent twitter discussion (pro tip: never do those) led me to realize that I have studied the problem of mtime comparison in considerably more depth than most people. I don't know whether to be proud of this or very concerned, but nevertheless, here we are. Soon, you'll know everything I do about the topic. I expect you will regret that as much as I have.

What is an mtime, anyway?

mtime is the "modified time" for the content associated with a given file. Generally, if anyone writes bytes anywhere in a file, the mtime will be updated. If a file has more than one name (ie. it's hardlinked to more than one place), all the names share the same inode and content, and thus all share the same mtime.

Annoyingly, when you update the content of a file, the mtime of its containing directory is not changed. All sorts of very convenient tree traversals would be possible if the directory mtime were updated (recursively to the root) when contained files changed, but no. This is probably because of hardlinks: since the kernel doesn't, in general, know all the filenames of an open file, it literally cannot update all the containing directories because it doesn't know what they are either. And anyway, purists might argue that the "content" of a directory doesn't change when the files it points to change; the content is merely a list of filenames and inode numbers, after all, and those stay the same, no matter what happens inside those inodes. Purists make me sad.

(Random side note: on MacOS, the kernel does know all the filenames of a hardlink, because hardlinks are secretly implemented as fancy symlink-like data structures. You normally don't see any symptoms of this except that hardlinks are suspiciously slow on MacOS. But in exchange for the slowness, the kernel actually can look up all filenames of a hardlink if it wants. I think this has something to do with Aliases and finding .app files even if they move around, or something.)

Related to mtime is the ctime, which most people would guess means "create time," but it absolutely does not. It means "attribute change time," which is different from the "modified time" because it updates whenever various inode fields change, not just the file contents. mtime is one of the inode fields, so whenever mtime changes, ctime also changes, but not vice versa. Among other things, ctime changes when file ownership, size, or link count change.

Link count is especially interesting: if you create or delete a hardlink to a given file, its ctime changes. Renaming is defined as creating a new hardlink and then removing another one, which means it updates the ctime (not the mtime), even though when it finishes, the link count is back to normal so the inode looks unchanged (other than the ctime). (Whether rename's create and unlink are supposed to be a single atomic transaction is a subject of much debate.)

So anyway, ctime changes much more sensitively than mtime. It turns out that mostly you don't care about the changes ctime measures, so it causes false positives, especially because of that pesky link count, but if you're paranoid, this can be helpful. Let's mostly talk about mtimes for now.

For completeness, there is also the atime, which means "access time." Originally, this would update whenever anyone "accessed" a file, usually defined as reading bytes from it. But this is unhelpful for two reasons: first, it means reading a filesystem causes writes to that filesystem, which greatly increases disk load (some people estimate by ~30%). Secondly, the definition of "access time" does not match what end users mean, which means various programs (especially backup software and search engines) try to avoid updating it. This workaround is so common that Linux added an O_NOATIME flag to open(2) to prevent updating atime. The default atime performance hit is so bad that many filesystems now have a relatime mount flag which decreases the precision of atime, thus reducing disk load. (Trivia: the Debian popularity-contest, which I started long ago, uses atime to figure out which installed packages you actually use.) (More trivia: if you mount your filesystem readonly, it is technically not POSIX compliant anymore because the atimes won't be updated.)

Popular misconceptions about mtime

How precise is it? It depends on the OS and filesystem. Originally, mtime had a precision of one second, which is all you can safely rely on. Nowadays most OSes have a stat(2) syscall that returns a struct timespec, which contains nanoseconds, but almost no filesystems provide that level of precision, and it depends on your kernel and disk format. For example, on my system (Debian Linux 4.9.0-7 with ext4), I get about 0.01s granularity. Stackoverflow has an explanation.
Is mtime monotonically increasing? No, it can go backwards. For example, the utimes(2) syscall, used by the touch command, can set the mtime to any value at all. (And tar might do this when extracting a tarball, for example.) If your system clock jumps from one time to another, it will set subsequent mtimes to match the new clock, even if the jump was backwards. And so on.
Does mtime get set to >= the current time? No, this depends on clock granularity. For example, gettimeofday() can return times in microseconds on my system, but ext4 rounds timestamps down to the previous ~10ms (but not exactly 10ms) increment, with the surprising result that a newly-created file is almost always created in the past:
```
  $ python -c "
  import os, time
  t0 = time.time()
  open('testfile', 'w').close()
  print os.stat('testfile').st_mtime - t0
  "

  -0.00234484672546
```
Does mtime get set to <= the current time? No, it might be set to a future time. For example, imagine you have an NFS server whose clock is set 5 seconds in the future relative to your client. The mtime is assigned by the server, so when you create the file, its mtime will be 5 seconds in the future. (Changing the standard so that mtime is set by the client doesn't really help: then programs running on the server will see a file 5 seconds in the past. And relying on ntpd isn't perfect either: it can only reduce clock skew between machines, not eliminate it.) For extra inconsistency, if a client uses utimes(2) to force the time to a particular value, this gets passed through to the server unchanged.
Is mtime always nonzero? No. Various cheaply-written virtual filesystems, like many fuse-based ones, don't bother setting mtime.
Does a changed mtime guarantee that a file has different content? No. Perhaps you wrote a block that happened to be identical to the block that already existed at that point in the file; the mtime changes anyway. Perhaps you wrote a block and then changed it back; the mtime changes twice.
Does changed content guarantee a changed mtime? No. Clock skew, low precision, or utimes(2) can cause an mtime to be the same as last time you checked. (This is also true for ctime, etc.)
Do version control systems like git save the mtime? No, not really. The tree and blob objects stored by git contain no timestamp information at all. (This is very good for deduplication.) commit objects contain various timestamps (commit time, author time, etc), and you could use that to reverse-engineer a guess for the mtime of a given file: the commit time of the most recent commit that changed that file's content, for example. But that's not what people do, mostly because it creates problems with make, which we'll get to shortly. (Git doesn't have the dangerous mtime-setting feature built in, but it does seem to exist in svn. You probably still shouldn't do it.)

(This all creates interesting philosophical questions. Is the "last modified" time of a file the time when the new content itself was created, or when this particular instance of it was written to disk? If you had a sci-fi device that could make a perfect scan of my physical being and run me in a simulation, what would be the mtime of the input file? And so on.)

(The bup project I started, which uses a git-formatted repo to back up your filesystem, does need to save mtime and other metadata. It stores metadata in separate hidden files in the git tree and reapplies it at restore time.)
Does switching branches in git screw up the mtime? No, not more than anything else. git just rewrites the changed files and lets the kernel update the mtime, so they look as if someone edited them with a text editor.
Does writing to a file via mmap() update the mtime? Hah. Well, maybe. See, POSIX guarantees that the mtime "will be marked for update at some point in the interval between a write reference to the mapped region and the next call to msync() ... If there is no such call, these fields may be marked for update at any time after a write reference." This definition actually leaves a lot of leeway for weirdness.

I wrote a little test program (mmap_test.c) to check how this works nowadays, and, of course, it varies between OSes. On Linux (4.9.0, ext4), the mtime is updated at the first page dirty after an mmap() or msync(). On FreeBSD (11.2, ufs), it's updated at msync() or munmap() time. On MacOS (10.11.6), it updates only at msync() time, not at munmap() time. I even tried with the "WSL personality" (4.4.0-17134-Microsoft) on Windows 10, which had especially terrible results: mmaped writes never updated the mtime at all.

I think the MacOS behaviour is allowed because the spec says "may" instead of "will" in that second sentence, but it's a stretch. The Linux behaviour may be illegal depending how you define "a write reference"; Linux seems to interpret it as "the first" or "a randomly selected" write reference, while I would expect to interpret it as "each" write reference (with the result that mtime must be updated at least once between the last reference and the msync(), which would be fine).

Of all these, the only useful behaviour seems to be FreeBSD's; at minimum, we surely want mtime to be updated at least once after all changes to a file have been done. MacOS and Linux don't always do so, and WSL never does so. This lends credence to the claim that the .git/index file, which uses mmap, is synced incorrectly by file sync tools relying on mtime. Ironically, the faster and better the file sync tool, the more likely it is to hit the race condition. An easy fix would be to have git always write() a useless byte before closing the index file. But I'd prefer if the kernel were less dumb.

Okay! That's the introduction. Now let's move on to application.

mtimes and make

I've kinda ruined the surprise by listing the caveats above. But let's look at what all that means when we try to use mtime for something.

make dependencies work in a very simple way. Now that we, as an industry, have decades of experience learning all the above caveats, we might describe it as "naive" in the sense that, when make was first invented, nobody had heard of all these problems, so it would be unfair to expect the author to design around them. In the world where make was first written:

there was no NFS;
there was no mmap;
there was no version control;
there were no fuse filesystems;
computers and compilers were so slow that a one-second timestamp granularity was never a problem.

In that world, they made the seemingly obvious decision to rebuild any target if the mtime of any of its dependencies was > the mtime of the target. (If you want to be extra safe in the presence of granularity problems, rebuild if >= rather than >.) This was an exciting innovation at the time.

Unfortunately, we now know that this can lead to numerous mistakes:

with NFS and clock skew, if a source file is edited on one machine and you run make on another, the input file might have mtime < target mtime, so nothing will happen. Or, you might rebuild the target and its mtime will still be < source mtime, so it'll be rebuilt again later.
If you accidentally set your system clock ahead by a day and build some stuff, then set your clock back to the present, all the stuff you built during that time will show up as "in the future" and therefore newer than source files you edit today, preventing all rebuilds. (Eventually GNU make started detecting future-dated files and printing a warning.)
If you have files modified through mmap, the mtime might not be up to date. (Luckily mmap is a rarity when editing source files or building software. Usually you aren't directly using live databases as your source files.)
If you replace one source file with another, eg.
```
  mv foo.c foo.c.bak
  mv foo.c.new foo.c
```
then the mtime is not updated, and make will see the old mtime of foo.c.new. That might be older than your foo binary, even though the binary does not yet contain the new foo.c. It won't be rebuilt.
If you have a dependency like
```
  foo.a: $(patsubst %.c,%.o,$(wildcard *.c))
```
(ie. produce foo.a from all the .o files built from all the C source files), then if one of the source files is deleted, it will no longer be one of the dependencies at all. But all the remaining dependencies are still older than foo.a, so foo.a will not be rebuilt.
If you put automake/autoconf-generated files (like ./configure and Makefile) in version control, you can get surprising results. Let's say automake has a Makefile rule to regenerate Makefile whenever the automake input files (eg. Makefile.am) change. In a tarball, which preserves mtimes, this will work, because Makefile will be newer than Makefile.am. But in a version control system, which uses the default kernel-assigned mtime when writing the files, it's undefined whether Makefile or Makefile.am is written first. If your timestamps are high precision (or they're low precision and you get unlucky), then Makefile could be "older" than Makefile.am, and automake will try to run anyway. Or if not, then it won't. So different people checking out the same source code will get different results based on random luck.
Computers are now so fast that you can save foo.c in your editor, and then produce foo.o, and then compile foo, all in the same one-second time period. If you do this and, say, save foo.c twice in the same second (and you have one-second granularity mtimes), then make can't tell if foo.o and foo are up to date or not. (As above, make can work around this by assuming if source mtime == target mtime, the target still needs to be rebuilt. This could cause spurious rebuilds, but is less dangerous than missing rebuilds.)

(This often happens if you're using one of those fancy new inotify-based tools that fires off a compile immediately, every time you hit save in your editor. Typescript does something like this, for example, as do auto-reloaders for various modern web languages. Symptom: needing to save your source file twice before the autocompiler catches it. And it happens more on MacOS, which has 1-second mtime granularity, than on Linux, which has 0.01-second mtimes.)
If your source files are in a virtual filesystem where mtime is always 0, then make will always think your source files have not changed and the target will never rebuild.

While we're here, there are some other common problems that aren't really the fault of mtime, but are common dependency problems with make:

If you upgrade your toolchain (eg. your C compiler), make doesn't know to rebuild your source files, unless you declare an explicit dependency on the toolchain files, which nobody does because it's hard to write that system-dependent stuff as a Makefile dependency rule. (This is one reason autoconf needs to be a ./configure script that generates a Makefile, instead of just a dependency executed by your Makefile.)

For that matter, when you update your toolchain, it's often from a distro-provided package (basically a tarball) with timestamps helpfully in the past, which are probably older than all your output files. So make won't see it as updated anyway!
If you pass variables on the make command line, like CFLAGS=-O2, they will usually not be part of a dependency and so won't cause a rebuild, and you'll end up with programs built halfway with the old flags, and halfway with the new ones. You can fix this by writing CFLAGS to a file, atomically replacing it only if the content differs, and depending on that file. But nobody does.
If you modify the Makefile, make will not by default rebuild any targets. You can fix this by adding an explicit dependency on Makefile, but this is a giant pain during development, because Makefile contains all your build rules; you don't want to recompile every source file just because you changed the linker command line, for example. (Some nowadays-rare versions of make actually tried to track Makefile changes, per rule, and cause rebuilds for these cases.)

make is not the only program that is affected by naive use of mtime. It's fairly common. For example, Go had so much trouble that they recently changed the Go compiler to just read and hash all the input files every time it runs. (Thanks to bradfitz for this link.)

redo: mtime dependencies done right

I happened to be aware of all these problems (well, not the mmap() madness; bleah!) when I set out to write redo so many years ago. I was also influenced by djb's design for redo, in which he writes, "When redo is asked to create a file that it hasn't heard of before, it presumes that the file is a source file if it exists, or a target file otherwise. In the second case (new target), redo immediately saves this decision to disk."

In other words, redo's design fundamentally depends on keeping a database of targets, if only to remember which files were produced by redo and which were not. From there, it's easy enough to extend that database to include mtime information about sources. And from there, we can add a bit more metadata to make the timestamp even more reliable.

My implementation of redo remembers the following information about each source and target:

mtime
size
inode number
file mode
owner uid and gid
(targets only) the sequence number of the last time it was built

redo considers a dependency dirty if any of those attributes changed since the last time a target was built. Notice how this dodges the various problems of mtime skew:

NFS client/server time skew doesn't matter; as long as the mtime changes in any direction, it's fine.
mmap() weirdness is reduced, because we notice changes in file size, as well as source mtimes that changed but are still older than the target.
If you mv a file to replace another, it will have a different inode number, which we notice. It also probably has a different size and (even if not newer than the target) mtime, any of which are sufficient.
Because redo has a database of all the dependencies used to produce a given target, if one of those inputs disappears, the target needs to be rebuilt. make doesn't remember the dependencies used last time, it only remembers the dependencies declared this time, so it can miss important changes in the list of dependencies.

(More generally, it's an interesting mathematical phenomenon that to correctly build software, we need to know not only the dependencies as they are now, but as they were before. Those two lists are used very differently. I don't think most build systems are designed with this realization, and it leads to subtle failures.)
If you put autoconf/automake generated files in your source repo, redo will "presume that the file is a source file," make a note of that, and not rebuild it. (It's still probably not a great idea to check those into version control. But at least now your build system won't go crazy.) If you then delete them, redo will consider them targets to be built.
redo has special treatment of source files whose mtime == the target mtime, so it can correct for overlaps even when your filesystem has very coarse timestamp granularity. Also, if you continue editing a source file, it will usually end up with a changed size, which also marks it as changed.
If your source files are in a braindead fuse filesystem, redo can use inode number and size to detect changes (although it still sucks and you should fix your fuse filesystem).

We can also fix the non-mtime-related missing dependencies:

It's easy to declare dependencies on your toolchain, because the rule for each target can track which parts of the toolchain were used while building, then retroactively declare a dependency on those. And we still notice a change if the new mtimes are in the past.
redo doesn't allow you to set variables on the command line; you have to write them to a file instead. This lets you easily declare dependencies on the file.
Since rules are written in separate .do files instead of one big Makefile, it's reasonable for redo to auto-declare a dependency on the .do file it used for a given target. When you edit a rule, the affected targets are automatically rebuilt.

I mentioned above that the Go compiler had problems with naive mtime-based dependency checking. I don't expect Go to switch to redo, but they could solve their problems in a similar way: generate a "database" (which might just be a text file) at build time. In the database, list the source files and their stamp information (mtime, inode, etc). Also list the toolchain version and relevant command line flags. Next time, read the database and compare against the new list of source files, the new stamps, and the new flags. If any are different, run the build. (Of course, all this is just a performance optimization that allows the compiler to avoid opening and reading files unnecessarily. The Go developers might reasonably continue to opt for the slower choice with fewer edge cases.)

Why not use checksums instead of mtimes?

Inevitably when the discussion of build dependencies comes up, someone who has heard part of the above story (usually some of the make problems caused by mtime comparisons) suggests throwing away mtimes entirely and always doing dependencies based on file checksums.

This can work, sometimes. And wow, I love checksums a lot (I wrote bup after all). But it isn't perfect for every situation.

As a clue to how complicated this can get: most people talking about this option suggest checksums as a way to avoid false negatives, ie., failing to rebuild when a source file has changed. But inode attributes change, in theory, at least as often as the content hash changes. Checksums are more useful for reducing false positives (ie. to avoid rebuilding in situations where we know the output will be identical). If someone is talking to you about rebuilding based on checksums, ask if they have thought about that difference.

Anyway, here are some specific problems with checksum-based dependencies:

Sometimes building a target has side effects. For example, imagine you have a redo rule for deploying a container to AWS. This does not really produce a "file" locally that you can checksum; it usually produces just log messages, or blank output, and the checksum of that will usually not change. Now, imagine you have a second container that you want to deploy only if the first container gets deployed correctly. If the checksum of the first container deployment is unchanged, the second one will think all its dependencies are unchanged, and not run, which might be incorrect. There are numerous other examples of side effects where this always-use-checksums behaviour is undesirable.

(On the other hand, some systems out there, like blaze/bazel, specialize in build systems without side effects. In that case a pure-checksum system is more appropriate. But then you have to escape from such systems if you want to do fun stuff like deploying containers. You end up punting the dependency problem elsewhere.)
Checksumming every output after building it is somewhat slow. This requires the build system to read the whole content of the file and do some math on it. Mostly this is not too serious: the file is probably already in disk cache (since you just wrote it a moment ago!) and calculating a checksum is almost always much faster than generating the file in the first place. And it only happens when a build was needed, which is expensive anyway. But it does add time to every build step.
Checksumming every input file before building is very slow. If you're considering whether to rebuild foo.a, and foo.a depends on *.o, and each.o depends on each.c, then you have to checksum the full content of every .c file every time you consider making an incremental build of foo. In large projects, this could be thousands, or tens of thousands of files, each of which we have to open(), read(), checksum, and close(), possibly over a network filesystem. For small projects this is fine, but for large projects, this sucks a lot.

blaze/bazel come from a world where source files are stored in a virtual filesystem, which happens to have the ability to tell you a precalculated checksum for every source file (except the ones you've changed locally). If you only have to checksum your locally-changed files on each build, that'll be very fast. But you need filesystem support to make this possible, and we can't assume that everywhere.

redo does support checksum-based dependencies, but it avoids the above problems as much as possible:

If you do nothing, redo uses database-mtime-based dependency checking, which is extremely fast on all operating systems. It's even reasonably fast on NFS.
redo-stamp lets you provide, after building a target, the data used to calculate that target's checksum (which might differ from the target itself, if you want).
redo-stamp records the checksum in its database after building a target. Any downstream target remembers that checksum in its list of dependencies; if it changes later, then the downstream target needs to be rebuilt. There is no need to actually recalculate any checksums when checking dependencies in the future. No special filesystem support is needed.

So you can use redo-stamp, in appropriate places, to reduce false positives in a way that causes overhead only at build time (not for checking dependencies later), and only for targets that need it.

That mmap() behaviour though. Seriously.

redo, buildroot, and serializing parallel logs

Tue, 06 Nov 2018 04:45:13 +0000

Lately I've been getting back to hacking on my djb redo implementation. We've fixed some problems with file attribute handling on NFS, obscure locking, and MacOS/FreeBSD portability. If you haven't tried redo in a while, you might want to give it another shot.

In case you haven't heard of redo before, here's the overview: it's like make, but with no special syntax (just sh scripts). The first time you "do" a build, it runs a set of recursive sh scripts, once per target. Those scripts can run a command called redo-ifchange, which declares dependencies on the given targets, checks if they are up to date, and if not, recurses into more scripts in order to build them. And that's it!

redo combines the best parts of imperative systems with the best parts of functional systems. The build scripts are all imperative - it just runs commands, and declaring dependencies happens as a side effect of some of those commands (redo-ifchange). When it's done, you have a purely-functional data structure that you can use for extremely fast dependency calculation. (Theoretically as fast as ninja, but my code isn't as optimized.)

[Credit: redo was invented by Daniel J. Bernstein. I merely implemented it.]

Parallelism

Things get a little more complex on modern multicore computers, where you almost always want parallel builds, which means producing different parts of the tree all at once, so in principle, a sequential-imperative tree of sh scripts is no longer the perfect model. Luckily, redo can handle it: if your script does redo-ifchange on more than one target at a time, it'll try to build all those in parallel. Then, if more than one parallel target tries to build a given dependency, it uses inter-process file locking to make sure the dependency only builds once.

But what's this about serializing logs?

I'm sure you already have your favourite build system and it builds things, and it almost certainly handles parallelism for whatever your use case. Even make does parallelism.

Where things tend to fall down is in rendering the output of a parallel build. When we're running a lot of jobs all at once, and blasting them all to stdout/stderr, and one step deep in the tree gets an error, then you might get pages and pages of successful output from other tasks interspersed with your error, making it very hard to figure out what went wrong and why.

There are various approaches to solving that. Some people would argue that the Unix Way is for programs that didn't fail to just print nothing at all; that's how the Go compiler works, for example. If you like that philosophy but you're using tools that don't agree (such as make itself, which prints all kinds of stuff while it works), you could wrap every command in a script that withholds its output, printing it only if the command returns a nonzero exit code.

That's all nice until you have to debug what went wrong. It's not a coincidence that make, which is made by Unix people, does not follow the Unix Way. Makefiles are just too complicated and hard to debug if you can't see what they're doing; and if step 10 goes wrong, you might be very curious about step 9, even though (nominally) it worked. It's not okay to throw away the successful log messages from step 9.

Fine. Parallel make output is flawed and gross. But everyone knows make is flawed and gross, so they switch to other systems. Most other popular build systems are tool-specific. Someone did a lot of work in cmake, for example, to make it print pretty messages during parallel builds of C/C++ programs. That works well. But if you're not building C/C++ programs, it can't help.

redo is a general purpose dependency system like make, so by definition it's going to run scripts which produce a lot of clutter, possibly including instances of make itself, and someone is going to have to debug them. What can we do to sanitize the logs?

Digression from 2012: loglinear

I've actually been thinking about this problem for more than six years already. Back in 2012, I added a log sanitizer script called loglinear (please pause for a moment to admire the pun) to our project's buildroot. (buildroot is a really handy all-in-one tool for building embedded Linux systems from scratch.)

loglinear worked like this: every time we ran a sub-make command like make path/to/project, we'd instead replace it with loglinear make path/to/project. loglinear then prefixes each line of the output with a job name, say path/to/project, and buffers it. When one of the loglinear processes exits, the top-level loglinear process then takes the buffer from that instance and dumps it to the top-level stdout.

Let's do an example. Imagine we're parallel building A, which depends on J, which depends on all of X, Y, and Z. We launch loglinear make A, which starts loglinear make J, which itself starts (all in parallel) loglinear make X, loglinear make Y, and loglinear make Z. J cannot continue until X, Y, and Z are done, but those three might finish in any order, and loglinear will print the output of each one as soon as it's done. So, the output will look something like this:

  Z: ...some stuff...
  Z: exited with code 0

  X: ...
  X: exited with code 0

  Y: ...
  Y: exited with code 0

  J: ...stuff...
  J: make X
  J: make Y
  J: make Z
  J: ...
  J: exited with code 0

  A: make J
  A: exited with code 0

loglinear also had some magic in case one of the processes returned nonzero: in that case, we'd print the successful processes first and the unsuccessful processes last, in the hope that the "most interesting" messages would end up at the bottom.

This made debugging a lot easier, because build messages from entire packages (like, say, the Linux kernel and busybox) separated out instead of interspersed, but it had some flaws. Most importantly, the output was very bursty: it waited until a given job was completely done before it printed anything. When busybox finished, you saw all the busybox logs; when the kernel finished, you saw all the kernel logs. Although useful, this is, frankly, not as fun as watching your 16-core workstation live-blast several screenfuls of compiler log messages per second. It feels slow.

There was also a secondary problem, which is that the messages, although linearized, were in the wrong order. Notice that, in the above, 'make J' (in A) happens after all the messages from J. This is because we print jobs in the order that they finish, and by definition, J must finish before the job that started J can finish. If we tried to print it in a more reasonable order (topmost job first, dependencies next, etc), then we couldn't print any incremental logs at all: A is guaranteed to finish last, but we want to print it first. This is all very logical when you think deeply about it, but trust me, it gets tedious explaining it to every new developer on your team, who just wants to know why the time is flowing backwards.

So we used loglinear, and it was a necessary evil, especially for viewing autobuilder logs, but nobody liked it. I dreamed of a better way.

Back to 2018: redo-log

I've had many years to contemplate my 2012 sins, and I have good news: I finally figured out how to do it right. Not only that, but instead of introducing a weird tool that you have to hack into your makefiles (and hack I did, oh boy did I ever, to make buildroot parallelize things the way I wanted), I've helpfully integrated the magic directly into redo. And not only that, but I've updated buildroot to use redo so that not only can you get linearized logs, but you can get faster buildroot startup time and faster/better buildroot dependencies too.

(Dear buildroot team: If you're reading this, I was going to send this patch to your mailing list, but it's not ready for prime time, or even code review, yet. I'd love to hear your feedback if you have any.)

redo-log takes a totally different approach from loglinear:

It saves the log for each target persistently to its own file, so you can look at it again later.
Rather than a flat list of log files, it tracks their tree order.
It prints log messages "depth first" instead of "breadth first," for less burstiness.
It prints output in the order dependencies were launched, instead of the order in which they were finished.
It can helpfully indent log messages based on their recursion level.
Since we persist logs anyway, we reserve the right to simply not print messages from some irrelevant targets when an error happens. You can always pull up the logs later if you care.

In other words, the logs from our earlier build now look like this:

  A: redo J
  J:   ...J stuff...
  J:   redo X
  X:     ...X stuff...
  X:     exit 0
  J:   redo Y
  Y:     ...Y stuff...
  Y:     exit 0
  J:   redo Z
  Z:     ...Z stuff...
  Z:     exit 0
  J:   ...more J stuff...
  J:   exit 0
  A: exit 0

The important realization - which is maybe obvious to you, but it wasn't obvious to me - is that, if you decide to do a depth-first traversal of log messages, the "deepest" one that is still running will continue producing incremental messages until it finishes. There's no need to buffer them!

During that time, other parallel branches of the tree will also be producing messages, which we do buffer until later. So Z might finish before X, but we just print the messages from X as they come out, until X is done. Then we go back to J, which sends us to Y, which we follow until it's done. When we get to Z, which is done already, we just print all its enqueued messages in one big blast, then move on.

An interesting invariant here is that it doesn't matter whether X, Y, or Z finishes first. If they each print their own messages (including launching their own subtasks) in a reproducible order, then no matter how the CPU schedules them, the total output will be in a reproducible order. This has the almost-impossible-sounding property that a set of "reproducible build" steps will produce a byte-for-byte reproducible log, even in the presence of unlimited parallelism.

The tricks go a little deeper. Let's say X, Y, and Z all depend on Q. Because of how .do scripts work, they will each run redo-ifchange Q at some undefined time in their respective build scripts. We only need to build Q once, but we don't know which of X, Y, or Z will be the one to do it. This is where the persistent logs come in; we don't actually care! Effectively the log is a DAG (directed acyclic graph, the same kind of structure used in git) with multiple links to Q. Its structure is like this:

  A: redo J
  J:   ...J stuff...
  J:   redo X
  X:     redo Q
  Q:       ...build Q...
  X:     ...X stuff...
  J:   redo Y
  Y:     redo Q
  Q:       ...build Q...
  Y:     ...Y stuff...
  J:   redo Z
  Z:     redo Q
  Q:       ...build Q...
  Z:     ...Z stuff...
  J:   ...more J stuff...
  A: exit 0

Of course we only ran Q once, so it's silly to print its output more than once. Let's trim it:

  A: redo J
  J:   ...stuff...
  J:   redo X
  X:     redo Q
  Q:       ...build Q...
  X:     ...X stuff...
  J:   redo Y
  Y:     redo Q
  Y:     ...Y stuff...
  J:   redo Z
  Z:     redo Q
  Z:     ...Z stuff...
  J:   ...stuff...
  A: exit 0

Because of our depth-first traversal rule, the log will always look exactly like that - even if job Q was "actually" launched by job Y and not X. redo-log prints logs in dependency order.

After the build finishes, though, you might want to investigate exactly how Z got built. To do that, you run redo-log Z, which prints this:

  Z: redo Q
  Q:   ...build Q...
  Z: ...Z stuff...
  Z: exit 0

In this case, we can show the steps for job Q as a subtree of Z, even though Q was actually built by Y, because it's not redundant when we're not printing Y.

One more complication arises if one of Z's dependencies changes and we need to rebuild Z, but Q has not changed. If we do that, then the "honest" redo log for the incremental rebuild of Z looks like this:

  Z: redo-ifchange Q  [nothing happens]
  Z: ...Z stuff...

But depending what you're doing - for example, if you want to see if the "reproducible log" for an incremental build of your whole reproducible build project matches a from-scratch build - it might make sense to show where Q came from. This is redo-log's -u option ("recurse into unchanged targets"), which then prints this:

  Z: redo Q
  Q:   ...build Q...
  Z: ...Z stuff...
  Z: exit 0

...in other words, the exact same log as you got when you built Z the first time.

Conclusion

I'm sure almost everyone reading this thinks I'm hopelessly pedantic to care so much about the sequence of lines of output in my build logs. You're right! But you're getting off easy, because you didn't have to live through my obsessing over LED blink synchronization across a lab full of wifi routers. (Useless trivia: I found at least three bugs in openntpd by noticing the LEDs in our lab were not all blinking uniformly.)

And that, my friends, is why tree traversal algorithms are fair game in job interviews.

...

Uh, also, you should try redo. You may also want to see how I redo-ized buildroot. If you're interested, you can join the discussions on the redo-list mailing list.

I simulated California housing and learned... about simulators

Mon, 17 Sep 2018 21:44:13 +0000

Here's a post I've been thinking about for a while. I shared it privately with a few people last year, but wanted to find a way to present it that wouldn't be wildly misconstrued. No luck so far.

Summary: Inspired by some conversations at work, I made a variant of my previously popular SimSWE (software engineer simulator) that has our wily engineers trying to buy houses and commute to work. The real estate marketplace is modeled based on Silicon Valley, a region where I don't live but my friends do, and which I define as the low-density Northern Californian suburban area including Sunnyvale, Mountain View, Cupertino, etc, but excluding San Francisco. (San Francisco is obviously relevant, since lots of engineers live in SF and commute to Silicon Valley, or vice versa, but it's a bit different so I left it out for simplicity.)

Even more than with the earlier versions of SimSWE (where I knew the mechanics in advance and just wanted to visualize them in a cool way), I learned a lot by making this simulation. As with all software projects, the tangible output was what I expected, because I kept debugging it until I got what I expected, and then I stopped. But there were more than the usual surprises along the way.

Is simulation "real science"?

Let's be clear: maybe simulation is real science sometimes, but... not when I do it.

Some of my friends were really into Zen and the Art of Motorcycle Maintenance back in high school. The book gets less profound as I get older, but one of my favourite parts is their commentary on the scientific method:

A man conducting a gee-whiz science show with fifty thousand dollars’ worth of Frankenstein equipment is not doing anything scientific if he knows beforehand what the results of his efforts are going to be. A motorcycle mechanic, on the other hand, who honks the horn to see if the battery works is informally conducting a true scientific experiment. He is testing a hypothesis by putting the question to nature. [...]

The formation of hypotheses is the most mysterious of all the categories of scientific method. Where they come from, no one knows. A person is sitting somewhere, minding his own business, and suddenly - flash! - he understands something he didn’t understand before. Until it’s tested the hypothesis isn’t truth. For the tests aren’t its source. Its source is somewhere else.

Here's the process I followed. I started by observing that prices in Silicon Valley are unexpectedly high, considering how much it sucks to live there, and rising quickly. (Maybe you like the weather in California and are willing to pay a premium; but if so, that premium has been rising surprisingly quickly over the last 15 years or so, even as the weather stays mostly the same.)

Then I said, I have a hypothesis about those high prices: I think they're caused by price inelasticity. Specifically, I think software engineers can make so much more money living in California, compared to anywhere else, that it would be rational to move there and dramatically overpay for housing. The increase in revenue will exceed the increase in costs.

I also hypothesized that there's a discontinuity in the market: unlike, say, New York City, where prices are high but tend to gently fluctuate, prices in Silicon Valley historically seem to have two states: spiking (eg. dotcom bubble and today's strong market) or collapsing (eg. dotcom crash).

Then I tried to generate a simulator that would demonstrate those effects.

This is cheating: I didn't make a simulator from first principles to see what would happen. What I did is I made a series of buggy simulators, and discarded all the ones that didn't show the behaviour I was looking for. That's not science. It looks similar. It probably has a lot in common with p-hacking. But I do think it's useful, if you use the results wisely.

If it's not science, then what is it?

It's part of science. This approach is a method for improved hypothesis formulation - the "most mysterious" process described in the quote above.

I started with "I think there's a discontinuity," which is too vague. Now that I made a simulator, my hypothesis is "there's a discontinuity at the point where demand exceeds supply, and the market pricing patterns should look something like this..." which is much more appropriate for real-life testing. Maybe this is something like theoretical physics versus experimental physics, where you spend some time trying to fit a formula to data you have, and some time trying to design experiments to get specific new data to see if you guessed right. Except worse, because I didn't use real data or do experiments.

Real science in this area, by the way, does get done. Here's a paper that simulated a particular 2008 housing market (not California) and compared it to the actual market data. Cool! But it doesn't help us explain what's going on in Silicon Valley.

The simulator

Okay, with all those disclaimers out of the way, let's talk about what I did. You can find the source code here, if you're into that sort of thing, but I don't really recommend it, because you'll probably find bugs. Since it's impossible for this simulation to be correct in the first place, finding bugs is rather pointless.

Anyway. Imagine a 2-dimensional region with a set of SWEs (software engineers), corporate employers, and various homes, all scattered around randomly.

Luckily, we're simulating suburban Northern California, so there's no public transit to speak of, traffic congestion is uniformly bad, and because of zoning restrictions, essentially no new housing ever gets built. Even the 2-dimensional assumption is accurate, because all the buildings are short and flat. So we can just set all those elements at boot time and leave them static.

What does change is the number of people working, the amount companies are willing to pay them, the relative sizes of different companies, and exactly which company employs a given SWE at a given time. Over a period of years, this causes gravity to shift around in the region; if an engineer buys a home to be near Silicon Graphics (RIP), their commute might get worse when they jump over to Facebook, and they may or may not decide it's time to move homes.

So we have an array of autonomous agents, their income, their employer (which has a location), their commute cost, their property value (and accumulated net worth), and their property tax.

(I also simulated the idiotic California "property tax payments don't change until property changes owners" behaviour. That has some effect, mainly to discourage people from exchanging equally-priced homes to move a bit closer to work, because they don't want to pay higher taxes. As a result, the market-distorting law ironically serves to increase commute times, thus also congestion, and make citizens less happy. Nice work, California.)

The hardest part of the simulator was producing a working real estate bidding system that acted even halfway believably. My simulated SWEs are real jerks; they repeatedly exploited every flaw in my market clearing mechanics, leading to all kinds of completely unnatural looking results.

Perversely, the fact that the results in this version finally seem sensible gives me confidence that the current iteration of my bidding system is not totally wrong. A trained logician could likely prove that my increased confidence is precisely wrong, but I'm not a logician, I'm a human, and here we are today.

The results

Let's see that plot again, repeated from up above.

The x axis is time, let's say months since start. The top chart shows one dot for every home that gets sold on the open market during the month. The red line corresponds to the 1.0 crossover point of the Demand to Supply Ratio (DSR) - the number of people wanting a home vs the number of homes available.

The second plot shows DSR directly. That is, when DSR transitions from <1.0 to >1.0, we draw a vertical red line on all three plots. For clarity there's also a horizontal line at 1.0 on the second plot.

The third plot, liquidity, shows the number of simulated homes on the market (but not yet sold) at any given moment. "On the market" means someone has decided they're willing to sell, but the price is still being bid up, or nobody has made a good enough offer yet. (Like I said, this part of the simulator was really hard to get right. In the source it just looks like a few lines of code, but you should see how many lines of code had to die to produce those few. Pricing wise, it turns out to be quite essential that you (mostly) can't buy a house which isn't on the market, and that bidding doesn't always complete instantaneously.)

So, what's the deal with that transition at DSR=1.0?

To answer that question, we have to talk about the rational price to pay for a house. One flaw in this simulation is that our simulated agents are indeed rational: they will pay whatever it takes as long as they can still make net profit. Real people aren't like that. If a house sold for $500k last month, and you're asking $1.2 million today, they will often refuse to pay that price, just out of spite, even though the whole market has moved and there are no more $500k houses. (You could argue that it's rational to wait and see if the market drops back down. Okay, fine. I had enough trouble simulating the present. Simulating my agents' unrealistic opinions of what my simulator was going to do next seemed kinda unwieldy.)

Another convenient aspect of Silicon Valley is that almost all our agents are engineers, who are a) so numerous and b) so rich that they outnumber and overwhelm almost all other participants in the market. You can find lots of news articles about how service industry workers have insane commutes because they're completely priced out of our region of interest.

(Actually there are also a lot of long-term residents in the area who simply refuse to move out and, while complaining about the obnoxious techie infestation, now see their home as an amazing investment vehicle that keeps going up each year by economy-beating percentages. In our simulator, we can ignore these people because they're effectively not participating in the market.)

To make a long story short, our agents assume that if they can increase their income by X dollars by moving to Silicon Valley vs living elsewhere, then it is okay to pay mortgage costs up to R*X (where R is between 0 and 100%) in order to land that high-paying job. We then subtract some amount for the pain and suffering and lost work hours of the daily commute, proportionally to the length of the commute.

As a result of all this, housing near big employers is more expensive than housing farther away. Good.

But the bidding process depends on whether DSR is less than one (fewer SWEs than houses) or more than one (more SWEs than houses). When it's less than one, people bid based on, for lack of a better word, the "value" of the land and the home. People won't overpay for a home if they can buy another one down the street for less. So prices move, slowly and smoothly, as demand changes slowly and smoothly. There's also some random variation based on luck, like occasional employer-related events (layoffs, etc). Market liquidity is pretty high: there are homes on the market that are ready to buy, if someone will pay the right price. It's a buyer's market.

Now let's look at DSR > 1.0, when (inelastic) demand exceeds supply. Under those conditions, there are a lot of people who need to move in, as soon as possible, to start profiting from their huge wages. But they can't: there aren't enough homes. So they get desperate. Every month they don't have a house, they forfeit at least (1-R)*X in net worth, and that makes them very angry, so they move fast. Liquidity goes essentially to zero. People pay more than the asking price. Bidding wars. Don't stop and think before you make an offer: someone else will buy it first, at a premium. It's a seller's market.

When this happens, prices settle at, basically, R*X. (Okay, R*X is the mortgage payment, so convert the annuity back to a selling price. The simulator also throws in some variably sized down payments depending on the net worth you've acquired through employment and previous real estate flipping. SWEs gonna SWE.)

Why R*X? Because in our simulator - which isn't too unlike reality - most of our engineers make roughly the same amount of income. I mean, we all know there's some variation, but it's not that much; certainly less than an order of magnitude, right? And while there are a few very overpaid and very underpaid people, the majority will be closer to the median income. (Note that this is quite different from other housing markets, where there are many kinds of jobs, the income distribution is much wider, and most people's price sensitivity is much greater.)

So as a simplification, we can assume R and X are the same for "all" our engineers. That means they simply cannot, no matter how much they try, pay more than R*X for a home. On the other hand, it is completely rational to pay all the way up to R*X. And demand exceeds supply. So they if they don't pay R*X, someone else will, and prices peak at that level.

When DSR dips back below 1.0: liquidity goes up and prices go back down. Interestingly, the simulated prices drop a lot slower than they shot up in the first place. One reason is that most people are not as desperate to sell as they were to buy. On the other hand, the people who do decide to sell might have a popular location, so people who were forced to buy before - any home at any price - might still bid up that property to improve their commute. The result is increasing price variability as people sell off not-so-great locations in exchange for still-rare great locations.

What does all this mean?

First of all, unlike healthier markets (say, New York City) where an increase in demand translates to higher prices, and demand can increase or decrease smoothly, and you can improve a property to increase its resale price, Silicon Valley is special. It has these three unusual characteristics:

Demand is strictly greater than supply
Most buyers share a similar upper limit on how much they can pay
Other than that limit, buyers are highly price insensitive

That means, for example, that improving your home is unlikely to increase its resale value. People are already paying as much as they can. Hence the phenomenon of run-down homes worth $1.5 million in ugly neighbourhoods with no services, no culture, and no public transit, where that money could buy you a huge mansion elsewhere, or a nice condo in an interesting neighbourhood of a big city.

It means raising engineer salary to match the higher cost of living ("cost of living adjustment") is pointless: it translates directly to higher housing prices (X goes up for everyone, so R*X goes up proportionally), which eats the benefit.

Of course, salaries do continue to rise in Silicon Valley, mostly due to continually increasing competition for employees - after all, there's no more housing so it's hard to import more of them - which is why we continue to see a rise in property values at all. But we should expect it to be proportional to wages and stock grants, not housing value or the demand/supply ratio.

In turn, this means that a slight increase in housing supply should have effectively no impact on housing prices. (This is unusual.) As long as demand exceeds supply, engineers will continue to max out the prices.

As a result though, the market price provides little indication of how much more supply is needed. If DSR > 1.0, this simulation suggests that prices will remain about flat (ignoring wage increases), regardless of changes in the housing supply. This makes it hard to decide how much housing to build. Where the market is more healthy, you can see prices drop a bit (or rise slower) when new housing comes on the market, and you can extrapolate to see how much more housing is appropriate.

At this point we can assume "much more" housing is needed. But how much? Are we at DSR=2.5 or DSR=1.001? If the latter, a small amount of added housing could drop us down to DSR=0.999, and then the market dynamics would change discontinuously. According to the simulation - which, recall, we can't necessarily trust - the prices would drop slowly, but they would still drop, by a lot. It would pop the bubble. And unlike my simulation, where all the engineers are rational, popping the bubble could cause all kinds of market panic and adjacent effects, way beyond my area of expertise.

In turn, what this means is that the NIMBYs are not all crazy. If you try to improve your home, the neighbourhood, or the region, you will not improve the property values, so don't waste your money (or municipal funds); the property values are already at maximum. But if you build more housing, you run the risk of putting DSR below 1.0 and sending property values into free fall, as they return to "normal" "healthy" market conditions.

Of course, it would be best globally if we could get the market back to normal. Big tech companies could hire more people. Service industry workers could live closer to work, enjoy better lives, and be less grumpy. With more market liquidity, engineers could buy a home they want, closer to work, instead of just whatever was available. That means they could switch employers more easily. People would spend money to improve their property and their neighbourhood, thus improving the resale value and making life more enjoyable for themselves and the next buyer.

But global optimization isn't what individuals do. They do local optimization. And for NIMBYs, that popping bubble could be a legitimate personal financial disaster. Moreover, the NIMBYs are the people who get to vote on zoning, construction rules, and improvement projects. What do you think they'll vote for? As little housing as possible, obviously. It's just common sense.

I would love to be able to give advice on what to do. It's most certainly a housing bubble. All bubbles pop eventually. Ideally you want to pop the bubble gently. But what does that mean? I don't know; an asset that deteriorates to 30% of its current price, slowly, leaves the owner just as poor as if it happened fast. And I don't know if it's possible to hold prices up to, say, 70% instead of 30%, because of that pesky discontinuity at DSR=1.0. The prices are either hyperinflated, or they aren't, and there seems to be no middle.

Uh, assuming my simulator isn't broken.

That's my hypothesis.

XML, blockchains, and the strange shapes of progress

Thu, 31 Jan 2019 23:09:04 +0000

Back in the early 2000s, XML was all the rage. An unusual evolution from HTML, which itself was an evolution (devolution?) from SGML, XML was supposed to be a backlash against complexity.

SGML originally grew from the publishing industry (for example, the original DocBook was an SGML language) and had flexible parser features so not-too-technical writers could use it without really understanding how tags worked. It provided some interesting shortcuts: for example, there's no reason to close the last <chapter> when opening a new <chapter>, because obviously you can't have a chapter inside a chapter, and so on. SGML was an organically-evolved mess, but it was a mess intended for humans. You can see that legacy in HTML, which was arguably just a variant of SGML for online publishing, minus a few features.

All that supposedly-human-friendly implicit behaviour became a problem, especially for making interoperable implementations (like web browsers). Now, don't get me wrong, the whole parsability issue was pretty overblown. Is browser compatibility really about what I mean when I write some overlapping tags like <b>hello <u>cruel</b> world</u>? I mean, yes. But more important are semantics, like which methods of javascript DOM objects take which sorts of parameters, or which exist at all, and what CSS even means.

But we didn't know that then. Let's say all our compatibility problems were caused by how hard it is to parse HTML.

Given that, some brave souls set out to solve the problem Once and For All. That was XML: a simplification of HTML/SGML with parsing inconsistencies removed, so that given any XML document, if nothing else, you always knew exactly what the parse tree should be. That made it a bit less human friendly (now you always had to close your tags), but most humans can figure out how to close tags, eventually, right?

Because strictness was the goal, Postel's Law didn't apply, and there was a profusion of XML validators, each stricter than the last, including fun features like silently downloading DTDs from the Internet on every run, and fun bugs like arbitrary code execution on your local machine or data leakage if that remote DTD got hacked.

(Side note about DTDs: those existed in SGML too. Interestingly, because of the implicit tag closing, it was impossible to parse SGML without knowing the DTD, because only then could you know which tags to nest and which to auto-close. In XML, since all tags need to be closed explicitly, you can happily parse any document without even having the DTD: a welcome simplification. So DTDs are vestigial, syntactically, and could have been omitted. (You can still ignore them whenever you use XML.) DTDs still mean something - they prevent syntactically legal parse trees from being accepted if they contain certain semantic errors - but that turns out to be less important. Oh well.)

Unfortunately, XML was invented by a [series of] standards committees with very little self control, so after simplifying it they couldn't stop themselves from complexifying it again. But you could mostly ignore the added bits, except for the resulting security holes, and people mostly did, and they were mostly happy.

There was a short-lived attempt to convince every person on the Internet to switch from easy-to-write HTML to easy-to-parse XHTML (HTML-over-XML), but that predictably failed, because HTML gets written a few billion times a day and HTML parsers get written once or twice a decade, so writability beats parsability every time. But that's an inconsequential historical footnote, best forgotten.

What actually matters is this:

XML is the solution to every problem

Why do we still hear about XML today? Because despite failing at its primary goal - a less hacky basis for HTML - it was massively successful at the related job of encoding other structured data. You could grab an XML parser, write a DTD, and auto-generate code for parsing pretty much anything. Using XSL, you could also auto-generate output files from your auto-parsed XML input files. If you wanted, your output could even be more XML, and the cycle could continue forever!

What all this meant is that, if you adopted XML, you never needed to write another parser or another output generator. You never needed to learn any new syntax (except, ironically, XSL and DTD) because all syntax was XML. It was the LISP of the 2000s, only with angle brackets instead of round ones, and not turing complete, and we didn't call it programming.

Most importantly, you never needed to argue with your vendor about whether their data file was valid, because XML's standards compliant validator tools would tell you. And never mind, since your vendor would obviously run the validator before sending you the file, you'd never get the invalid file in the first place. Life would be perfect.

Now we're getting to the real story. XML was created to solve the interoperability problem. In enterprises, interoperability is huge: maybe the biggest problem of all. Heck, even humans at big companies have trouble cooperating, long before they have to exchange any data files. Companies will spend virtually any amount of money to fix interoperability, if they believe it'll work.

Money attracts consultants, and consultants attract methodologies, and metholologies attract megacorporations with methodology-driven products. XML was the catalyst. Money got invested, deployments got deployed, and business has never been the same since.

Right?

Okay, from your vantage point, situated comfortably with me here in the future, you might observe that it didn't all work out exactly as we'd hoped. JSON came along and wiped out XML for web apps (but did you ever wonder why we fetch JSON using an XMLHttpRequest?). SOAP and XML-RPC were pretty unbearable. XML didn't turn out to be a great language for defining your build system configs, and "XML databases" were discovered to be an astonishingly abysmal idea. Nowadays you mostly see XML in aging industries that haven't quite gotten with the programme and switched to JSON and REST and whatever.

But what's interesting is, if you ask the enterprisey executive types whether they feel like they got their money's worth from the giant deployments they did while going Full XML, the feedback will be largely positive. XML didn't live up to expectations, but spending a lot of money on interoperability kinda did. Supply chains are a lot more integrated than they used to be. Financial systems actually do send financial data back and forth. RPCs really do get Remotely Called. All that stuff got built during the XML craze.

XML, the data format, didn't have much to do with it. We could have just as easily exchanged data with JSON (if it had existed) or CSV or protobufs or whatever. But XML, the dream, was a fad everyone could get behind. Nobody ever got fired for choosing XML. That dream moved the industry forward, fitfully, chaotically, but forward.

Blockchains

So here we are back in the present. Interoperability remains a problem, because it always will. Aging financial systems are even more aged now than they were 15 or 20 years ago, and they exchange data only a little better than before. Many of us still write cheques and make "wire" transfers, so named because they were invented for the telegraph, and before that we used horses instead of wires. Manufacturing supply chains are a lot better, but much of that improvement came from everybody just running the same one or two software megapackages. Legal contracts are really time consuming and essentially non-automated. Big companies are a little aggravated at having to clear their transactions through central authorities, not because they have anything against centralization and paying a few fees, but because those central authorities (whether banks, exchanges, or the court system) are really slow and inefficient.

We need a new generation of investment. And we need everyone to care about it all at once, because interoperability doesn't get fixed unless everybody fixes it.

That brings us to blockchains. Like XML, they are kinda fundamentally misguided; they don't solve a problem that is actually important. XML solved syntax, which turned out not to be the problem. Blockchains [purport to] solve centralization, which will turn out not to be the problem. But they do create the incentive to slash and burn and invest a lot of money hiring consultants. They give us an excuse to forget everything we thought we knew about contracts and interoperability and payment clearing, much of which was already irrelevant.

It's the forgetting that will allow progress.

Disclaimers

Bitcoin is like the XHTML of blockchains.
No, I don't think cryptocurrency investing is a good idea.
Blockchain math is actually rather useful, to the extent that it is a (digitally signed) "chain of blocks," which was revolutionary long ago, when it was first conceived. As one example, git is a chain of blocks and many of its magical properties come directly from that. Chains of blocks are great.

But the other parts are all rather dumb. We can do consensus in many (much cheaper) ways. Most people don't want their transactions or legal agreements published to the world. Consumers actually like transactions to be reversible, within reason; markets work better that way. Companies even like to be able to safely unwind legal agreements sometimes when it turns out those contracts weren't the best idea. And they rarely want the public to know about their contracts, let alone their inventory details.

I predict that in 20 years, we're going to have a lot of "blockchain" stuff in production, but it won't be like how people imagine it today. It'll have vestigial bits that we wonder about, and it'll all be faintly embarrassing, like when someone sends you their old XML-RPC API doc and tells you to use that.

"Yeah, I know," they'll say. "But it was state of the art back then."

Updates

2018-09-15: Some people are saying that json is "schemaless." It isn't, not any more than XML, but schema enforcement is optional, like in XML, and there's more than one way to do it, like in XML. A really elegant json schema mechanism is Go's reflection-based one, where you declare a struct and the standard library knows how to convert it to/from json. This is harder to do with XML, because generic XML doesn't map directly onto typical language data structures. (The downside of declaring your schemas in Go is that the enforcement doesn't work in any other language, of course.)

A little bump in the wire that makes your Internet faster

Sat, 15 Sep 2018 18:34:09 +0000

My parents live in a rural area, where the usual monopolist Internet service provider provides the usual monopolist Internet service: DSL, really far from the exchange point, very very asymmetric, and with insanely oversized buffers (ie. bufferbloat), especially in the upstream direction. The result is that, basically, if you tried to browse the web while uploading anything, it pretty much didn't work at all.

I wrote about the causes of these problems (software, of course) in my bufferbloat rant from 2011. For some reason, there's been a recent resurgence of interest in that article. Upon rereading it, I (re-)discovered that it's very... uh... stream-of-consciousness. I find it interesting that some people like it so much. Even I barely understand what I wrote anymore. Also, it's now obsolete, because there are much better solutions to the problems than there used to be, so even people who understand it are not going to get the best possible results. Time for an update!

The Challenge

I don't live in the same city as my parents, and I won't be back for a few months, but I did find myself with some spare time and a desire to pre-emptively make their Internet connection more usable for next time I visited. So, I wanted to build a device (a "bump in the wire") that:

Needs zero configuration at install time
Does not interfere with the existing network (no DHCP, firewall, double NAT, etc)
Doesn't reduce security (no new admin ports in the data path)
Doesn't need periodic reboots
Actually solves their bufferbloat problem

Let me ruin the surprise: it works. Although we'll have to clarify "works" a bit.

If you don't care about all that, skip down to the actual setup down below.

This is an improvement, I promise!

Here's the fast.com test result before we installed the Bump.

(Side note: there are a lot of speedtests out there. I like fast.com for two reasons. First, they have an easy-to-understand bufferbloat test. Second, their owner has strong incentives to test actual Internet speeds including peering, and to bypass various monopolistic ISPs' various speedtest-cheating traffic shaping techniques.)

And here's what it looked like after we added the Bump:

...okay, so you're probably thinking, hey, that big number is lower now! It got worse! Yes. In a very narrow sense, it did get worse. But in most senses (including all the numbers in smaller print), it got better. And even the big number is not as much worse as it appears at first.

It would take a really long time and a lot of words to try to explain how these numbers interact and why it matters. But unluckily for you, I'm on vacation!

Download speed is the wrong measurement

In my wifi data presentation from 2016, I spent a lot of time exploring what makes an Internet connection feel "fast." In particular, I showed a slide from an FCC report from 2015 (back when the FCC was temporarily anti-monopolist):

What's that slide saying? Basically, that beyond 20 Mbps or so, typical web page load times stop improving.¹ Sure, if you're downloading large files, a faster connection will make it finish sooner.² But most people spend most of their time just browsing, not downloading.

Web page load times are limited by things other than bandwidth, including javascript parsing time, rendering time, and (most relevant to us here) round trip times to the server. (Most people use "lag", "latency", and "round trip time" to mean about the same thing, so we'll do that here too.) Loading a typical web page requires several round trips: to one or more DNS servers, then the TCP three-way handshake, then SSL negotiation, then grabbing the HTML, then grabbing the javascript it points to, then grabbing whatever other files are requested by the HTML and javascript. If that's, say, 10 round trips, at 100ms each, you can see how a typical page would take at least a second to load, even with no bandwidth constraints. (Maybe there are fewer round trips needed, each with lower latencies; same idea.)

So that's the first secret: if your page load times are limited by round trip time, and round trip time goes from 80ms (or 190ms) to 31ms (or 42ms), then you could see a 2x (or 4.5x) improvement in page load speed, just from cutting latency. Our Bump achieved that - which I'll explain in a moment.

It also managed to improve the measured uplink speed in this test. How is that possible? Well, probably several interconnected reasons, but a major one is: TCP takes longer to get up to speed when the round trip time is longer. (One algorithm for this is called TCP slow start.) And it has even more trouble converging if the round trip time is variable, like it was in the first test above. The Bump makes round trip time lower, but also more consistent, so it improves TCP performance in both ways.

But how does it work?

Alert readers will have noticed that by adding a Bump in the wire, that is, by adding an extra box and thus extra overhead, I have managed to make latency less. Alert readers will hate this, as they should, because it's called "negative latency," and alert readers know that there is no such thing. (I tried to find a good explanation of it on the web, but all the pages I could find sucked. I guess that's fair, for a concept that does not exist. Shameless self-plug then: I did write a fun article involving this topic back in 2009 about work we did back in 2003. Apparently I've been obsessing over this for a long time.)

So, right, the impossible. As usual, the impossible is a magic trick. Our Bump doesn't subtract latency; it just tricks another device - in this case the misconfigured DSL router provided by the monopolistic ISP - into adding less latency, by precisely adding a bit of its own. The net result is less than the DSL router on its own.

Bufferbloat (and chocolate)

Stop me if you've heard this one before. Most DSL routers and cable modems have buffers that were sized to achieve the maximum steady-state throughput on a very fast connection - the one that the monopolistic ISP benchmarks on, for its highest priced plan. To max out the speed in such a case, you need a buffer some multiple of the "bandwidth delay product," (BDP) which is an easier concept than it sounds like: just multiply the bandwidth by the round trip time (delay). So if you have 100ms round trip time and your upstream is about 25 Mbps = ~2.5 MBytes/sec, then your BDP is 2.5 Mbytes/sec * 0.1sec = 2.5 MBytes. If you think about it, the BDP is "how much data fits in the wire," the same way a pipe's capacity is how much water fits in the pipe. For example, if a pipe spits out 1L of water per second, and it takes 10 seconds for water to traverse the pipe, then the pipe contains 1L x 10 seconds = 10L.

Anyway, the pipe is the Internet³, and we can't control the bandwidth-delay product of the Internet from our end. People spend a lot of time trying to optimize that, but they get paid a lot, and I'm on vacation, and they don't let me fiddle with their million-dollar equipment, so too bad. What I can control is the equipment that feeds into the pipe: my router, or, in our plumbing analogy, the drain.

Duck in a bathtub drain vortex,
via pinterest

You know how when you drain the bathtub, near the end it starts making that sqlrshplshhh sucking noise? That's the sound of a pipe that's not completely full. Now, after a nice bath that sound is a key part of the experience and honestly makes me feel disproportionately gleeful, but it also means your drain is underutilized. Err, which I guess is a good thing for the environment. Uh.

Okay, new analogy: oil pipelines! Wait, those are unfashionable now too. Uh... beer taps... no, apparently beer is bad for diversity or something... chocolate fountains!

Chocolate fountain via indiamart

Okay! Let's say you rented one of those super fun chocolate fountain machines for a party: the ones where a pool of delicious liquid chocolate goes down a drain at the bottom, and then gets pumped back up to the top, only to trickle gloriously down a chocolate waterfall (under which you can bathe various fruits or whatever) and back into the pool, forever, until the party is over and the lucky, lucky party consultants get to take it home every night to their now-very-diabetic children.

Mmmm, tasty, tasty chocolate. What were we talking about again?

Oh right. The drain+pump is the Internet. The pool at the bottom is the buffer in your DSL router. And the party consultant is, uh, me, increasingly sure that I've ended up on the wrong side of this analogy, because you can't eat bits, and now I'm hungry.

Aaaaanyway, a little known fact about these chocolate fountain machines is that they stop dripping chocolate before the pool completely empties. In order to keep the pump running at capacity, there needs to be enough chocolate in the pool to keep it fully fed. In an ideal world, the chocolate would drip into the pool and then the pump at a perfectly constant rate, so you could adjust the total amount of chocolate in the system to keep the pool+pump content at the absolute minimimum, which is the bandwidth-delay product (FINALLY HE IS BACK ON TOPIC). But that would require your chocolate to be far too watery; thicker chocolate is more delicious (SIGH), but has the annoying habit of dripping in clumps (as shown in the picture) and not running smoothly into the drain unless the pool has extra chocolate to push it along. So what we do is to make the chocolate thicker and clumpier (not negotiable) and so, to keep the machine running smoothly, we have to add extra chocolate so that the pool stays filled, and our children thus become more diabetic than would otherwise be necessary.

Getting back to the math of the situation, if you could guarantee perfectly smooth chocolate (packet) flow, the capacity of the system could be the bandwidth-delay product, which is the minimum you need in order to keep the chocolate (bits) dripping at the maximum speed. If you make a taller chocolate tower (increase the delay), you need more chocolate, because the BDP increases. If you supersize your choco-pump (get a faster link), it moves the chocolate faster, so you need more chocolate, because the BDP increases. And if your chocolate is more gloppy (bursty traffic), you need more chocolate (bits of buffer) to make sure the pump is always running smoothly.

Moving back into pure networking (FINALLY), we have very little control over the burstiness of traffic. We generally assume it follows some statistical distribution, but in any case, while there's an average flow rate, the flow rate will always fluctuate, and sometimes it fluctuates by a lot. That means you might receive very little traffic for a while (draining your buffer aka chocolate pool) or you might get a big burst of traffic all at once (flooding your buffer aka chocolate pool). Because of a phenomenon called self-similarity, you will often get the big bursts near the droughts, which means your pool will tend to fill up and empty out, or vice versa.

(Another common analogy for network traffic is road traffic. When a road is really busy, car traffic naturally arranges itself into bursts, just like network traffic does.)

Okay! So your router is going to receive bursts of traffic, and the amount of data in transit will fluctuate. To keep your uplink fully utilized, there must always be 1 BDP of traffic in the Internet link (the round trip from your router to whatever server and back). To fill the Internet uplink, you need to have a transmit queue in the router with packets. Because the packets arrive in bursts, you need to keep that transmit queue nonempty: there's an ideal fill level so that it (almost) never empties out, but so our children don't get unnecessarily diabetic, um, I mean, so that our traffic is not unnecessarily delayed.

An empty queue isn't our only concern: the router has limited memory. If the queue memory fills up because of a really large incoming burst, then the only thing we can do is throw away packets, either the newly-arrived ones ("tail drop") or some of the older ones ("head drop" or more generally, "active queue management").

When we throw away packets, TCP slows down. When TCP slows down, you get slower speedtest results. When you get slower speedtest results, and you're a DSL modem salesperson, you sell fewer DSL modems. So what do we do? We add more RAM to DSL modems so hopefully the queue never fills up.⁴ The DSL vendors who don't do this, get a few percent slower speeds in the benchmarks, so nobody buys their DSL modem. Survival of the fittest!

...except, as we established earlier, that's the wrong benchmark. If customers would time page load times instead of raw download speeds, shorter buffers would be better. But they don't, unless they're the FCC in 2015, and we pay the price. (By the way, if you're an ISP, use better benchmarks! Seriously.)

So okay, that's the (very long) story of what went wrong. That's "bufferbloat." How do we fix it?

"Active" queue management

Imagine for a moment that we're making DSL routers, and we want the best of both worlds: an "unlimited" queue so it never gets so full we have to drop packets, and the shortest possible latency. (Now we're in the realm of pure fiction, because real DSL router makers clearly don't care about the latter, but bear with me for now. We'll get back to reality later.)

What we want is to have lots of space in the queue - so that when a really big burst happens, we don't have to drop packets - but for the steady state length of the queue to be really short.

But that raises a question. Where does the steady state length of the queue come from? We know why a queue can't be mainly empty - because we wouldn't have enough packets to keep the pipe full - and we know that the ideal queue utilization has something to do with the BDP and the burstiness. But who controls the rate of incoming traffic into the router?

The answer: nobody, directly. The Internet uses a very weird distributed algorithm (or family of algorithms) called "TCP congestion control." The most common TCP congestion controls (Reno and CUBIC) will basically just keep sending faster and faster until packets start getting dropped. Dropped packets, the thinking goes, mean that there isn't enough capacity so we'd better slow down. (This is why, as I mentioned above, TCP slows down when packets get dropped. It's designed that way.)

Unfortunately, a side effect of this behaviour is that the obvious dumb queue implementation - FIFO - will always be full. That's because the obvious dumb router doesn't drop packets until the queue is full. TCP doesn't slow down until packets are dropped,⁵ so it doesn't slow down until the queue is full. If the queue is not full, TCP will speed up until packets get dropped.⁶

So, all these TCP streams are sending as fast as they can until packets get dropped, and that means our queue fills up. What can we do? Well, perversely... we can drop packets before our queue fills up. As far as I know, the first proposal of this idea was Random Early Detection (RED), by Sally Floyd and Van Jacobson. The idea here is that we calculate the ideal queue utilization (based on throughput and burstiness), then drop more packets if we exceed that length, and fewer packets if we're below that length.

The only catch is that it's super hard to calculate the ideal queue utilization. RED works great if you know that value, but nobody ever does. I think I heard that Van Jacobson later proved that it's impossible to know that value, which explains a lot. Anyway, this led to the development of Controlled Delay (CoDel), by Kathleen Nichols and Van Jacobson. Instead of trying to figure out the ideal queue size in packets, CoDel just sees how long it takes for packets to traverse the queue. If it consistently takes "too long," then it starts dropping packets, which signals TCP to slow down, which shortens the average queue length, which means a shorter delay. The cool thing about this design is it's nearly configuration-free: "too long," in milliseconds, is pretty well defined no matter how fast your link is. (Note: CoDel has a lot of details I'm skipping over here. Read the research paper if you care.)

Anyway, sure enough, CoDel really works, and you don't need to configure it. It produces the best of both worlds: typically short queues that can absorb bursts. Which is why it's so annoying that DSL routers still don't use it. Jerks. Seriously.

Flow queueing (FQ)

A discussion on queue management wouldn't be complete without a discussion about flow queueing (FQ), the second half of the now very popular (except among DSL router vendors) fq_codel magic combination.

CoDel is a very exciting invention that should be in approximately every router, because it can be implemented in hardware, requires almost no extra memory, and is very fast. But it does have some limitations: it takes a while to converge, and it's not really "fair"⁷. Burstiness in one stream (or ten streams) can increase latency for another, which kinda sucks.

Imagine, for example, that I have an ssh session running. It uses almost no bandwidth: most of the time it just goes as fast as I can type, and no faster. But I'm also running some big file transfers, both upload and download, and that results in an upload queue that has something to do with the BDP and burstiness of the traffic, which could build up to hundreds of extra milliseconds. If the big file transfers weren't happening, my queue would be completely empty, which means my ssh traffic would get through right away, which would be optimal (just the round trip time, with no queue delay).

A naive way to work around this is prioritization: whenever an ssh packet arrives, put it at the front of the queue, and whenever a "bulk data" packet arrives, put it at the end of the queue. That way, ssh never has to wait. There are a few problems with that method though. For example, if I use scp to copy a large file over my ssh session, then that file transfer takes precedence over everything else. Oops. If I use ssh on a different port, there's no way to tag it. And so on. It's very brittle.

FQ tries to give you (nearly) the same low latency, even on a busy link, with no special configuration. To make a long story short, it keeps a separate queue for every active flow (or stream), then alternates "fairly"⁷ between them. Simple round-robin would work pretty well, but they take it one step further, detecting so-called "fat" flows (which send as fast as they can) and "thin" flows (which send slower than they can) and giving higher priority to the thin ones. An interactive ssh session is a thin flow; an scp-over-ssh file transfer is a fat flow.

And then you put CoDel on each of the separate FQ queues, and you get Linux's fq_codel, which works really well.

Incidentally, it turns out that FQ alone - forget about CoDel or any other active queue management - gets you most of the benefits of CoDel, plus more. You have really long queues for your fat flows, but the thin flows don't care. The CoDel part still helps (for example, if you're doing a videoconference, you really want the latency inside that one video stream to be as low as possible; and TCP always works better with lower latency), and it's cheap, so we include it. But FQ has very straightforward benefits that are hard to resist, as long as you and FQ agree on what "fairness"⁷ means.

FQ is a lot more expensive than CoDel: it requires you to maintain more queues - which costs more memory and CPU time and thrashes the cache more - and you have to screw around with hash table algorithms, and so on. As far as I know, nobody knows how to implement FQ in hardware, so it's not really appropriate for routers running at the limit of their hardware capacity. This includes super-cheap home routers running gigabit ports, or backbone routers pushing terabits. On the other hand, if you're limited mainly by wifi (typically much less than a gigabit) or a super slow DSL link, the benefits of FQ outweigh its costs.⁸

Back to the Bump

Ok, after all that discussion about CoDel and FQ and fq_codel, you might have forgotten that this whole exercise hinged on the idea that we were making DSL routers, which we aren't, but if we were, we could really cut down that latency. Yay! Except that's not us, it's some hypothetical competent DSL router manufacturer.

I bet you're starting to guess what the Bump is, though, right? You insert it between your DSL modem and your LAN, and it runs fq_codel, and it fixes all the queuing, and life is grand, right?

Well, almost. The problem is, the Bump has two ethernet ports, the LAN side and the WAN side, and they're both really fast (in my case, 100 Mbps ethernet, but they could be gigabit ethernet, or whatever). So the data comes in at 100 Mbps, gets enqueued, then gets dequeued at 100 Mbps. If you think about it for a while, you'll see this means the queue length is always 0 or 1, which is... really short. No bufferbloat there, which means CoDel won't work, and no queue at all, which means there's nothing for FQ to prioritize either.

What went wrong? Well, we're missing one trick. We have to release the packets out the WAN port (toward the DSL modem) more slowly. Ideally, we want to let them out perfectly smoothly at exactly the rate that the DSL modem can transmit them over the DSL link. This will allow the packets to enqueue in the Bump instead, where we can fq_codel them, and will leave the DSL modem's dumb queue nearly empty. (Why can that queue be empty without sacrificing DSL link utilization? Because the burstiness going into the DSL modem is near zero, thanks to our smooth release of packets from the Bump. Remember our chocolate fountain: if the chocolate were perfectly smooth, we wouldn't need a pool of chocolate at the bottom. There would always be exactly the right amount of chocolate to keep the pump going.)

Slowing down the packet outflow from the Bump is pretty easy using something called a token bucket filter (tbf). But it turns out that nowadays there's a new thing called "cake" which is basically fq_codel+tbf combined. Combining them has some advantages that I don't really understand, but one of them is that it's really easy to set up. You just load the cake qdisc, tell it the upload and download speeds, and it does the magic. Apparently it's also less bursty and takes less CPU. So use that.

The only catch is... what upload/download speeds should we give to cake? Okay, I cheated for that one. I just asked my dad what speed his DSL link goes in real life, and plugged those in. (Someday I want to build a system that can calculate this automatically, but... it's tricky.)

But what about the downstream?

Oh, you caught me! All that stuff was talking about the upstream direction. Admittedly, on DSL links, the upstream direction is usually the worst, because it's typically about 10x slower than the downstream, which means upstream bufferbloat problems are about 10x worse than downstream. But of course, not to be left out, the people making the big heavy multi-port DSL equipment at the ISP added plenty of bufferbloat too. Can we fix that?

Kind of. I mean, ideally they'd get a Bump over on their end, between the ISP and their DSL megarouter, which would manage the uplink's queue. Or, if we're dreaming anyway, the surprisingly competent vendor of the DSL megarouter would just include fq_codel, or at least CoDel, and they wouldn't need an extra Bump. Fat chance.

It turns out, though, that if you're crazy enough, you can almost make it work in the downstream direction. There are two catches: first, FQ is pretty much impossible (the downstream queue is just one queue, not multiple queues, so tough). And second, it's a pretty blunt instrument. What you can do is throw away packets after they've traversed the downstream queue, a process called "policing" (as in, we punish your stream for breaking the rules, rather than just "shaping" all streams so that they follow the rules). With policing, the best you can do is detect that data is coming in too fast, and start dropping packets to slow it down. Unfortunately, the CoDel trick - dropping traffic only if the queue is persistently too long - doesn't work, because on the receiving side, you don't know how big the queue is. When you get a packet from the WAN side, you just send it to the LAN side, and there's no bottleneck, so your queue is always empty. You have to resort to just throwing away packets whenever the incoming rate is even close to the maximum. That is, you have to police to a rate somewhat slower than the DSL modem's downlink speed.

Whereas in the upload direction, you could use, say, 99.9% of the upload rate and still have an empty queue on the DSL router, you don't have the precise measurements needed for that in the download direction. In my experience you have to use 80-90%.

That's why the download speed in the second fast.com test at the top of this article was reduced from the first test: I set the shaping rate pretty low. (I think I set it too low, because I wanted to ensure it would cut the latency. I had to pick some guaranteed-to-work number before shipping the Bump cross-country to my parents, and I only got one chance. More tuning would help.)

Phew!

I know, right? But, assuming you read all that, now you know how the Bump works. All that's left is learning how to build one.

BYOB (Build Your Own Bump)

Modern Linux already contains cake, which is almost all you need. So any Linux box will do, but the obvious choice is a router where you install openwrt. I used a D-Link DIR-825 because I didn't need it to go more than 100 Mbps (that's a lot faster than a 5 Mbps DSL link) and I liked the idea of a device with 0% proprietary firmware. But basically any openwrt hardware will work, as long as it has at least two ethernet ports.

You need a sufficiently new version of openwrt. I used 18.06.0. From there, install the SQM packages, as described in the openwrt wiki.

Setting up the cake queue manager

This part is really easy: once the SQM packages are installed in openwrt, you just activate them in the web console. First, enable SQM like this:

In the Queue Discipline tab, make sure you're using cake instead of whatever the overcomplicated and mostly-obsolete default is:

(You could mess with the Link Layer Adaptation tab, but that's mostly for benchmark twiddlers. You're unlikely to notice if you just set your download speed to about 80%, and upload speed to about 90%, of the available bandwidth. You should probably also avoid the "advanced" checkboxes. I tried them and consistently made things worse.)

If you're boring, you now have a perfectly good wifi/ethernet/NAT router that happens to have awesome queue management. Who needs a Bump? Just throw away your old wifi/router/firewall and use this instead, attached to your DSL modem.

Fancy bridge mode

...On the other hand, if, like me, you're not boring, you'll want to configure it as a bridge, so that nothing else about the destination network needs to be reconfigured when you install it. This approach just feels more magical, because you'll have a physical box that produces negative latency. It's not as cool if the negative and positive latencies are added together all in one box; that's just latency.

What I did was to configure the port marked "4" on the DIR-825 to talk to its internal network (with a DHCP server), and configure the port marked "1" to bridge directly to the WAN port. I disabled ports 2 and 3 to prevent bridging loops during installation.

To do this, I needed two VLANs, like this:

(Note: the DIR-825 labels have the ports in the opposite order from openwrt. In this screenshot, port LAN4 is on VLAN1, but that's labelled "1" on the physical hardware. I wanted to be able to say "use ports 1 and WAN" when installing, and reserve port 4 only for configuration purposes, so I chose to go by the hardware labels.)

Next, make sure VLAN2 (aka eth0.2) is not bridged to the wan port (it's the management network, only for configuring openwrt):

And finally, bridge VLAN1 (aka eth0.1) with the wan port:

You may need to reboot to activate the new settings.

Footnotes

¹ Before and since that paper in 2015, many many people have been working on cutting the number of round trips, not just the time per round trip. Some of the recent improvements include TCP fast open, TLS session resumption, and QUIC (which opens encrypted connections in "zero" round trips). And of course, javascript and rendering engines have both gotten faster, cutting the other major sources of page load times. (Meanwhile, pages have continued getting larger, sigh.) It would be interesting to see an updated version of the FCC's 2015 paper to see if the curve has changed.

² Also, if you're watching videos, a faster connection will improve video quality (peaking at about 5 Mbps/stream for an 1080p stream or 25 Mbps/stream for 4K, in Netflix's case). But even a 20 Mbps Internet connection will let you stream four HD videos at once, which is more than most people usually need to do.

³ We like to make fun of politicians, but it's actually very accurate to describe the Internet as a "series of tubes," albeit virtual ones.

⁴ A more generous interpretation is that DSL modems end up with a queue size calculated using a reasonable formula, but for one particular use case, and fixed to a number of bytes. For example, a 100ms x 100 Mbps link might need 0.1s x 100 Mbit/sec x ~0.1 bytes/bit = 1 Mbyte of buffer. But on a 5 Mbit/sec link, that same 1 Mbyte would take 10 Mbits / 5 Mbit/sec = 2 seconds to empty out, which is way too long. Unfortunately, until a few years ago, nobody understood that too-large buffers could be just as destructive as too-small ones. They just figured that maxing out the buffer would max out the benchmark, and that was that.

⁵ Various TCP implementations try to avoid this situation. My favourite is the rather new TCP BBR, which does an almost magically good job of using all available bandwidth without filling queues. If everyone used something like BBR, we mostly wouldn't need any of the stuff in this article.

⁶ To be more precise, in a chain of routers, only the "bottleneck" router's queue will be full. The others all have excess capacity because the link attached to the bottleneck is overloaded. For a home Internet connection, the bottleneck is almost always the home router, so this technicality doesn't matter to our analysis.

⁷ Some people say that "fair" is a stupid goal in a queue. They probably say this because fairness is so hard to define: there is no queue that can be fair by all possible definitions, and no definition of fair will be the "best" thing to do in all situations. For example, let's say I'm doing a videoconference call that takes 95% of my bandwidth and my roommate wants to visit a web site. Should we now each get 50% of the bandwidth? Probably not: video calls are much more sensitive to bandwidth fluctuations, whereas when loading a web page, it mostly doesn't matter if it takes 3 seconds instead of 1 second right now, as long as it loads. I'm not going to try to take sides in this debate, except to point out that if you use FQ, the latency for most streams is much lower than if you don't, and I really like low latency.

⁸ Random side note: FQ is also really annoying because it makes your pings look fast even when you're building up big queues. That's because pings are "thin" and so they end up prioritized in front of your fat flows. Weirdly, this means that most benchmarks of FQ vs fq_codel show exactly the same latencies; FQ hides the CoDel improvements unless you very carefully code your benchmarks.

Books that explain (parts of) how the world really works

Wed, 08 Aug 2018 10:04:12 +0000

I spend a lot of time answering various people's tech-business questions. Occasionally, I will say something that sounds brilliant, and an unlucky listener will exclaim, "Avery, how do you know so much stuff?"

Answer: I read it in a book. Now, I don't actually read very many books. But I've been lucky enough to receive some great recommendations. Here are mine. Now you, too, can sound like a genius.

To make it more interesting, I'll start with the questions, then tell you the book that answers them.

Q: Why is your multi-year epic project still not profitable?

Because you don't understand how the "technology adoption life cycle" (the one with visionaries, early adopters, mainstream, laggards, etc) really works. The ultimate book on this topic is Crossing the Chasm, by Geoffrey Moore in 1991.

(You're going to notice a pattern to my book suggestions: they're old. It turns out really good advice stays relevant.)

Crossing the Chasm is very dear to my heart: it was recommended to me by one of the VCs that invested in my first startup - unfortunately a bit late. Chapter by chapter, it was like reading the post-mortem for my startup, except it was all written before we ever started.

Don't do this, because that will happen. Oh, crap, that's what we did, and that's what happened. Whatever you do, avoid this common mistake, because it always results in xyz. Yup, xyz everywhere. And so on. When we finally turned things around, I gave all the credit directly to this book. It is that good.

My favourite part is their definition of a "market segment." Most people don't know what they're talking about when they talk about a market segment. You think you do, but you don't. I thought I did, but I didn't. You can't run a startup without understanding what a market segment is. You will fail. Read the book instead.

One of their bits of advice: don't try to capture 10% of a big market. Capture 100% of a small market. Anyone who says "if we can just get 1% of this 10 billion dollar market..." is admitting defeat. Nobody will buy your product if it resolves only 90% of their obstacles, and for every market segment, it's a different 90%. You need to find a market segment, even a small one, for which you can solve 100%.

Q: Why are all these Cloud service providers losing money and offering terrible products?

Because the "Cloud" market has already Crossed the Chasm and has moved into the mainstream ("hypergrowth") phase, where the rules are nothing like what you're used to.

After the chasm has been crossed and you've made a product that serves several adjacent market niches, it gets easier to win even more market segments, partly because your company is actually making money and growing. Each new segment makes you more profitable, so you grow more and expand more, in a positive feedback loop.

The only problem is that, once you start taking off, you attract competition. Crossing the Chasm is largely about staying orthogonal to your competitors, but as you enter the mainstream, that stops working.

That's Cloud services right now. And ride sharing.

Inside the Tornado, also by Geoffrey Moore, a few years later in 1995, talks about this situation and how to deal with it.

To be honest, I read this book because I liked Crossing the Chasm so much, but I recommend it a lot less often. Almost nobody is lucky enough to get into the middle of a hypergrowth market, so the book is mostly irrelevant to almost everyone. (One exception: by reading the book and learning what a hypergrowth market is really like, you might realize you aren't in one after all, and save yourself a lot of mistakes. Or you might realize you don't have the capital needed to compete, and bail out.)

An interesting observation in the book is that hypergrowth phases are temporary (eventually the entire mainstream is saturated), and the market share of various competitors, after hypergrowth ends, stays mostly fixed. Customers sign up with a particular product, and are reluctant to change. That's why, say, desktop PCs have run about the same set of OSes for a long time, the same word processors and spreadsheets have been popular in the same proportion for a long time, and the relative market shares between Coke and Pepsi or Colgate and Crest rarely change.

(After saturation, customers still switch from one supplier to another, but there's a dynamic equilibrium: everybody keeps spending money on sales and features, but none of them can do it any better than the others, so you might lose one customer and gain another.)

So, to oversimplify, the book's recommended strategy during hypergrowth is to invest like crazy to acquire market share, because the big customers you acquire now might stick around for decades. It's worth losing money in the short term. You have to move impossibly fast and land those customers at any cost, sacrificing short-term profitability, product elegance, and so on.

The tornado is ugly. It results in ugly products and weird business models (at least temporarily). There'll be time to fix it all later, when the market is saturated.

Q: Won't Big Company X just clone your product and steal all your customers?

Maybe! But maybe not. It depends on some surprising factors.

Nowadays in the tech industry we use the words "disruption" and "innovation" interchangeably. We like to talk about "disruptive innovation," but by now we've forgotten why we insert the adjective instead of just saying "innovation." Is it just because disruption just sounds cooler?

We can trace the term "disruptive innovation" back to the book that invented it in the first place: The Innovator's Dilemma, by Clayton M. Christensen, in 1997. That book is about the difference between two kinds of innovation: sustaining innovation, and disruptive innovation.

As techies, what we've forgotten is that sustaining innovation is the common one. It's so common we forget to name it. When Intel makes a newer, faster chipset (sadly less often lately) or there's a new model of laptop, that's sustaining innovation. It happens all the time. And notably, big companies are really great at it. If you're a big company and you make laptops, then if you make an even better laptop, you'll probably make customers happier, sell more of them, and make more money. Everybody wins. Companies pour money into that.

If your startup is making a product that's the same thing only a bit better, then yeah, Big Company X is going to clone it and eat your lunch. You're doing sustaining innovation, and all that takes is money. They have more money than you. The end.

Very different is disruptive innovation. A disruptive innovation generally has some fatal flaw that makes it laughable to incumbents. A smart phone... without a keyboard? Come on. A taxi service... where the drivers are untrained, unprofessional, and unlicensed? An ad algorithm that tries to show you fewer ads? A tiny little 1" hard disk that's way more expensive per gigabyte? Who wants that?

The Innovator's Dilemma explores what actually makes innovation "disruptive," across several different product areas, from digging machines, to printers, to hard disks, and shows how the same trend repeats over and over. As the incumbent, you keep making your products better (sustaining innovation). Customer requirements increase, but not as fast as your products improve. Meanwhile, some competitors are working on some obviously inferior technology selling to your lowest-value cheapskate customers you're happy to have off your books anyway, while you sell to ever-higher-end customers and make ever-larger profits.

The competing technology is far from meeting your customers' demands, and its trajectory clearly shows it will never catch up to your product line. Then, one day, the inferior technology - still much worse than yours - finally gets good enough to meet your biggest customers' needs. Boom. You're dead. It's still not as good as your product, but that doesn't matter. It's good enough.

What makes this a "dilemma" is that, right up until that crossover point, your company would lose money by investing in the new thing. The old thing is far more profitable, and your customers don't even want the new thing; the new thing can't yet do what they want! Any profitable company has been highly optimized to deliver what customers want, and reject what customers don't want, so of course you won't invest in the inferior product. And then, one day, all your customers change their minds all at once, and you're too late.

You can plot the whole thing on a graph. You can even plot it before it happens, knowing it will happen, and still not be able to stop it. Math is beautiful. (The book suggests a few tricks to try to save yourself, but the tricks are very non-obvious.)

If you want to know why IBM missed the PC revolution, and why Microsoft was caught off guard by the Internet, eventually exited the smartphone market, still runs apps with tiny mouse-optimized toolbars on touchscreen tablets, and still can't run your Access databases on the web, this book is the one.

It'll also help you answer that very important question: what's really stopping Big Company X from cloning my product?

Q: I have lots of money. How do I clone a competitor's product, only better?

Don't do that.

Perhaps it's too obvious when I phrase it that way, but let's be honest, people try this all the time. Imagine, say, I don't know, instant messaging apps. There's an existing product getting popular, it looks pretty trivial, I could clone that in a week! Then I'll just spend more on marketing, or bundle some product integrations, or my better brand name, and rake in the customers.

Maybe. I mean, you're not automatically doomed if you try this approach. It's worked before. Like the U.S. invasion of Afghanistan... oh wait. Well, okay, the U.S. didn't lose, right? They did spend like 10,000x as much money as their opponent though. Perhaps there was a better way.

This is where I recommend The Art of War, by the famous Sun Tzu, in the 5th century BC.

I'll admit it, the book is a bit of a cliché at this point. The advice all sounds obvious, in its obtusely-phrased way, perhaps because people have been studying it for millenia and then quoting it in Hollywood movies. Problem is, most people still aren't getting it.

The book starts with "The supreme art of war is to subdue the enemy without fighting" and moves on from there. Why are you fighting? Why are you fighting on the opponent's turf? Why is your strategy so transparent? Do you even have a strategy?

Strategy is a thing that can be learned. You won't learn it all from reading this book, which is pretty short and a bit outdated (although it does have some parts about supply chain management). But it's a start.

If you want to know how Microsoft really did get "a computer on every desk and in every home, running Microsoft software," in a world where they had an inferior product and lots of competitors, this is a good place to start.

Q: I have the Best Architecture and the Fanciest Developers. Why do people hate my product?

Lots of reasons, of course! But an interesting reason is that designers and architects have a tendency to look at their products from a bird's eye, high level, super abstract view. They want the abstraction to look beautiful, and they spend not enough time on the ground, in the dirt, fine tuning things to make them work right.

It's virtually impossible to explain this, in terms of programming, to the programmers who are doing it. And we do it, all of us. But sometimes we can learn from analogy. That's one reason I really liked The Death and Life of Great American Cities, by Jane Jacobs, in 1961. (It was recommended to me long ago by a one-time roommate of mine, and it really changed how I think about cities and about design.)

It has absolutely nothing to do with programing, but like The Innovator's Dilemma talking about bankrupt digging machine companies, we can learn from Jane Jacobs's excellent rant about terrible-brilliant urban planners, with huge budgets, following Best Practices and optimizing literal aerial views, making most of the cities of North America into awful messes, and also severely screwing up large parts of even the cities that are better.

Like the people who said Dependency Injection was a bad idea, Jacobs was ridiculed for a long time, but history has proven her right.

It's fascinating to read why flattening neighbourhoods and rebuilding them from scratch doesn't work; results are vastly worse than simply refactoring (although the term "refactoring" hadn't been invented in 1961). It turns out that a lot of subtle stuff was built up in that neighbourhood over time, and when you start from scratch, you lose it all. You might have heard this somewhere before.

I love this book because not only can you learn how Car Culture happened (while assuming people were doing their best, rather than resorting to a conspiracy theory about car manufacturers and public transit), but also what causes crime, what motivates people to care (or not care) for their neighbourhood buildings, and the mental traps that confound anyone in search of elegance.

Q: Why is software development so unpredictable?

Actually it isn't, you're just predicting it wrong. Sorry. I already wrote way too extensively about that so I'll resist going over it all again.

Out-of-control software development processes turn out to be a special case of general out-of-control processes, and who's the expert on process control? W. Edwards Deming, of course, in his several books, including The New Economics from the year 2000. I wrote about some of his work about a year and a half ago.

Since then, I've had more time to contemplate what I read, and I'm now pretty sure I know the very most important thing I learned: the difference between random variation and outliers.

When we're modeling any kind of statistics, we have a tendency to assume our data fits some kind of standard statistical distribution - for example, Gaussian.

What Deming was teaching was that real-life human processes often don't cleanly fit a continuous model. The reality, in manufacturing and in software, is only mostly Gaussian. Every now and then something wacky will happen that is not just part of the distribution. Imagine the difference between drilling a hole slightly off center, vs breaking the drill bit. How many standard deviations out is a broken drill bit? Wrong question.

Imagine doing a least-square-error curve fit while including those outliers. Because errors are squared (hence the name), a few big errors will move the distribution as much as a large number of small errors. But the outliers are by definition not predictable or reproducible, so including them in the curve fit only stops you from predicting the remaining predictable parts.

Deming's greatest lesson, in my opinion, is that you have to treat outliers and common errors completely differently. When you overdrive a software development process, then things don't just stretch, they break. Teammates getting tired and needing recovery (worse standard deviation) is one thing; teammates quitting (outlier) is another thing entirely. You want to reduce standard deviation, but you usually do that by adjusting a continuous variable (eg. to make recovery time more predictable, work people less hard). To prevent outliers, you need a more discrete change (eg. to keep people from quitting, figure out what is making them quit, and get rid of it).

Deming also provides actual techniques - albeit very handwavy techniques that seem to make "real" statisticians cringe - for detecting outliers vs continuous variables. Something about the standard deviation and the interquartile range. This margin is too small, etc.

...

Well, that got long. Maybe I should have just listed the book titles.

An epic treatise on scheduling, bug tracking, and triage

Fri, 30 Nov 2018 21:20:48 +0000

Note 2017-12-29:

news.ycombinator.com discussion of this post

Note 2018-09-01:

an updated and shortened version of this talk

I did a talk at work about the various project management troubles I've seen through the years. People seemed to enjoy it a lot more than I expected, and it went a bit viral among co-workers. It turns out that most of it was not particularly specific to just our company, so I've taken the slides, annotated them heavily, and removed the proprietary bits. Maybe it'll help you too.

Sorry it got... incredibly long. I guess I had a lot to say!

[Obligatory side note: everything I post to this site is my personal opinion, not necessarily the opinion of my or probably any other employer.]

Scaling in and out of control

Way back in 2016, we were having some trouble getting our team's releases out the door. We also had out-of-control bug/task queues and no idea when we'd be done. It was kind of a mess, so I did what I do best, which is plotting it in a spreadsheet. (If what I do best was actually solving the problem, I assume they'd have to pay me more.)

Before 2016 though, we had been doing really quite well. As you can see, our average time between releases was about two months, probably due primarily to my manager's iron fist. You might notice that we had some periodic slower releases, but when I looked into them, it turned out that they crossed over the December holiday seasons when nobody did any work. So they were a little longer in wall clock time, but maybe not CPU time. No big deal. Pretty low standard deviation. Deming would be proud. Or as my old manager would say, "Cadence trumps mechanism."

What went wrong in 2016? Well, I decided to find out. And now we're finally getting to the point of this presentation.

But before we introspect, let's extrospect!

Here's a trend from a randomly selected company. I wanted to see how other companies scale up, so I put together a little analysis of Tesla's production (from entirely public sources, by the way; beware that I might have done it completely wrong). The especially interesting part of this graph (other than how long they went while producing only a tiny number of Roadsters and nothing else) is when they released the Model X. Before that, they roughly scaled employees at almost the same rate as they scaled quarterly vehicle production. With the introduction of the Model X, units shipped suddenly went up rapidly compared to number of employees, and sustained at that level.

I don't know for sure why that's the case, but it says something about improved automation, which they claim to have further improved for the Model 3 (not shown because I made this graph earlier in the year, and anyway, they probably haven't gotten the Model 3 assembly straightened out yet).

Tesla is a neat example of working incrementally. When they had a small number of people, they built a car that was economical to produce in small volumes. Then they scaled up slowly, progressively fixing their scaling problems as they went toward slightly-less-luxury car models. They didn't try to build a Model 3 on day 1, because they would have failed. They didn't scale any faster than they legitimately could scale.

So naturally I showed the Tesla plot to my new manager, who looked at it and asked the above question.

Many people groan when they hear this question, but I think we should give credit where credit is due. Many tech companies are based around the idea that if you hire the smartest people, and remove all the obstacles you can, and get rid of red tape, and just let them do what they feel they need to do, then you'll get better results than using normal methods with normal people. The idea has obviously paid off hugely at least for some companies. So the idea of further boosting performance isn't completely crazy. And I did this talk for an Engineering Productivity team, whose job is, I assume, literally to make the engineers produce more.

But the reason people groan is that they take this suggestion in the most basic way: maybe we can just have the engineers work weekends or something? Well, I think most of us know that's a losing battle. First of all, even if we could get engineers to work, say, on Saturdays without any other losses (eg. burnout), that would only be a 20% improvement. 20%? Give me a break. If I said I was going to do a talk about how to improve your team's efficiency by 20%, you wouldn't even come. You can produce 20% more by hiring one more person for your 5-person team, and nobody even has to work overtime. Forget it.

The above is more or less what I answered when I was asked this question. But on the other hand, Tesla is charging thousands of dollars for their "optional" self-driving feature (almost everybody opts for it), which they've been working on for less time with fewer people than many of their competitors. They must be doing something right, efficiency wise, right?

Yes. But the good news is I think we can fix it without working overtime. The bad news is we can't just keep doing what we've been doing.

Cursed by goals

Here's one thing we can't keep doing. Regardless of what you think of Psychology Today (which is where I get almost all my dubiously sourced pop psychology claims; well, that and Wikipedia), there's ample indication from Real Researchers, such as my hero W. Edwards Deming, that says "setting goals" doesn't actually work and can make things worse.

Let's clarify that a bit. It's good to have some kind of target, some idea of which direction you're going. What doesn't work is deciding when you're going to get there. Or telling salespeople they need to sell 10% more next quarter. Or telling school teachers they need to improve their standardized test scores. Or telling engineers they need to launch their project at Whatever Conference (and not a day sooner or a day later).

What all these bad goals have in common is that they're arbitrary. Some supposedly-in-charge executive or product manager just throws out a number and everyone is supposed to scramble to make it come true, and we're supposed to call that leadership. But anybody can throw out a number. Real leaders have some kind of complicated thought process that leads to a number. And the thought process, once in place, actually gives them the ability to really lead, and really improve productivity, and predict when things really will be done (and make the necessary changes to help them get done sooner). Once they have all that, they don't really need to tell you an arbitrary deadline, because they already know when you'll finish, even if you still don't.

When it works, it all feels like magic. Do you suspect that your competitors are doing magic when they outdo you with fewer people and less money? It's not magic.

I just talked about why goals don't really help. This is a quote about how they can actually hurt, because of (among other things) a famous phenomenon called the "Student Syndrome." (By the way, the book this is taken from, Critical Chain, is a great book on project management.)

We all know how it works. The professor says you have an assignment due in a week. Everyone complains and says that's impossible, so the prof gives them an extension of an extra week. Then everyone starts work the night before anyway, so all that extra time is wasted.

That's what happens when you give engineers deadlines, like launching at Whatever Conference. They're sure they can get done in that amount of time, so they take it easy for the first half. Then they get progressively more panicked as the deadline approaches, and they're late anyway, and/or you ship on time and you get what Whatever Conference always gets, which is shoddy product launches. This is true regardless of how much extra time you give people. The problem isn't how you chose the deadline, it's that you had a deadline at all.

Stop and think about that a bit. This is really profound stuff. (By the way, Deming wrote things like this about car manufacturers in the 1950s, and it was profound then too.) We're not just doing deadlines wrong, we're doing it meta-wrong. The whole system is the wrong system. And there are people who have known that for 50+ years, and we've ignored them.

I wanted to throw this in too. It's not as negative as it sounds at first. SREs (Site Reliability Engineers) love SLOs (Service Level Objectives). We're supposed to love SLOs, and hate SLAs (Service Level Agreements). (Yes, they're different.) Why is that?

Well, SREs are onto something. I think someone over there read some of Deming's work. An SLA requires you to absolutely commit to hitting a particular specification for how good something should be and when, and you get punished if you miss it. That's a goal, a deadline. It doesn't work.

An SLO is basically just a documented measurement of how well the system typically works. There are no penalties for missing it, and everyone knows there are no penalties, so they can take it a bit easier. That's essential. The job of SRE is, essentially, to reduce the standard deviation and improve the average of whatever service metric they're measuring. When they do, they can update the SLO accordingly.

And, to the point of my quote above, when things suddenly get worse - when the standard deviation increases or the average goes in the wrong direction - the SLO is there so that they can monitor it and know something has gone wrong.

(An SLI - Service Level Indicator - is the thing you measure. But it's not useful by itself. You have to monitor for changes in the average and standard deviation, and that's essentially the SLO.)

I think it's pretty clear that SRE is a thing that works. SLOs give us some clues about why:

SLOs are loose predictions, not strict deadlines.
SLOs are based on measured history, not arbitrary targets thrown out by some executive.

We need SLOs for software development processes, essentially.

Schedule prediction is a psychological game

That was a bunch of philosophy. Let's switch to psychology. My favourite software psychologist is Joel Spolsky, of the famous Joel on Software blog. Way back in the early 2000s, he wrote a very important post called "Painless Software Schedules," which is the one I've linked to above. If you follow the link, you'll see a note at the top that he no longer thinks you should take his advice and you should use his newfangled thing instead. I'm sure his newfangled thing is very nice, but it's not as simple and elegant as his old thing. Read his old thing first, and worry about the new thing later, if you have time.

One of my favourite quotes from that article is linked above. More than a decade ago, at my first company, I was leading our eng team and I infamously (for us) provided the second above quote in response.

What Joel meant by "psychological games" was managers trying to negotiate with engineers about how long a project would take. These conversations always go the same way. The manager has all the power (the power to fire you or cancel your bonus, for example), and the engineer has none. The manager is a professional negotiator (that's basically what management is) and the engineer is not. The manager also knows what the manager wants (which is the software to be done sooner). The engineer has made an estimate off the top of their head, and is probably in the right ballpark, but not at all sure they have the right number. So the manager says, "Scotty, we need more power!" and the engineer says, "She canna take much more o' this, captain!" but somehow pulls it off anyway. Ha ha, just kidding! That was a TV show. In real life, Scotty agrees to try, but then can't pull it off after all and the ship blows up. That manager's negotiation skills have very much not paid off here. But they sure psyched out the engineer!

In my silly quote, the psychological games I'm talking about are the ones you want to replace that with. Motivation is psychological. Accurate project estimation is psychological. Feature prioritization is psychological. We will all fail at these things unless we know how our minds work and how to make them do what we want. So yes, we definitely need to play some psychological games. Which brings us to Agile.

I know you probably think Agile is mostly cheesy. I do too. But the problem is, the techniques really work. They are almost all psychological games. But the games aren't there to make you work harder, they're there, to use an unfortunate cliche, to make you work smarter. What are we doing that needs smartening? If we can answer that, do we need all the cheesy parts? I say, no. Engineers are pretty smart. I think we can tell engineers how the psychological games work, and they can find a way to take the good parts.

But let's go through all the parts just to be clear.

Physical index cards. There are reasons these are introduced into Agile: they help people feel like a feature is a tangible thing. They are especially good for communicating project processes with technophobes. (You can get your technophobic customers to write down things they want on index cards more easily than you can get them to use a bug tracking system.) Nowadays, most tech companies don't have too many technophobic employees. They also often have many employees in remote offices, and physical cards are a pain to share between offices. The fedex bills would be crazy. Some people try to use a tool to turn the physical cards into pictures of virtual physical cards, which I guess is okay for what it is, but I don't think it's necessary. We just need a text string that represents the feature.
Stories & story points. These turn out to be way more useful than you think. They are an incredibly innovative and effective psychological game, even if you don't technically write them as "stories." More on that in a bit.
Pair programming. Sometimes useful, especially when your programmers are kinda unreliable. They're like a real-time code review, and thus reduce latency. But mostly not actually something most people do, and that's fine.
Daily standup meetings. Those are just overhead. Agile, surprisingly enough, is not good because it makes you do stupid management fluff every day. It does, but that's not why it's good. We can actually leave out 95% of the stupid management fluff. The amazing thing about Agile is actually that you get such huge gains despite the extra overhead gunk it adds.
Rugby analogies (ie. SCRUM). Not needed. I don't know who decided sports analogies were a reliable way to explain things to computer geeks.
Strict prioritization. This is a huge one that we'll get to next - and so is flexible prioritization. Since everyone always knows what your priorities are (and in Agile, you physically post index cards on the wall to make sure they all know, but there are other ways), then people are more likely to work on tasks in the "right" order, which gets features done sooner than you can change your mind. Which means when you do change your mind, it'll be less expensive. That's one of the main effects of Agile. Basically, if you can manage to get everyone to prioritize effectively, you don't need Agile at all. It just turns out to be really hard.
Tedious progress tracking: also not needed. See daily standups, above. Agile accrues a lot of cruft because people take a course in it and there's inevitably a TPM (Technical Programme Manager, aka Project Manager) or something who wants to be useful by following all the steps and keeping progress spreadsheets. Those aren't the good parts. If you do this right, you don't even need a TPM, because the progress reports write themselves. Sorry, TPMs.
Burndown charts. Speaking of progress reporting, I don't think I'd heard of burndown charts before Agile. They're a simple idea: show the number of bugs (or stories, or whatever) still needing to be done before you hit your milestone. As stories/bugs/tasks/whatever get done, the line goes down, a process which we call "burning" because it makes us feel like awesome Viking warlords instead of computer geeks. A simple concept, right? But when you do it, the results are surprising, which we'll get to in a moment. Burndown charts are our fundamental unit of progress measurement, and the great thing is they draw themselves so you don't need the tedious spreadsheet.
A series of sprints, adding up to a marathon. This is phrased maybe too cynically (just try to find a marathon runner that treats a marathon as a series of sprints), but that's actually what Agile proposes. Sprints are very anti-Deming. They encourage Student Syndrome: putting off stuff until each sprint is almost over, then rushing at the end, then missing the deadline and working overtime, then you're burnt out so you rest for the start of the next sprint, and repeat. Sprints are also like salespeople with quarterly goals that they have to hit for their bonus. Oh, did I just say goals? Right, sprints are goals, and goals don't work. Let's instead write software like marathon runners run marathons: at a nice consistent pace that we can continue for a very long time. Or as Deming would say, minimize the standard deviation. Sprints maximize the standard deviation.

Phew! Okay, that was wordy. Let's draw a picture.

I made a SWE simulator that sends a bunch of autonomous SWE (software engineer) drones to work on a batch of simulated bugs (or subtasks). What we see are five simulated burndown charts. In each of the five cases, the same group of SWEs works on the exact same bug/task database, which starts with a particular set of bug/tasks and then adds new ones slowly over time. Each task has a particular random chance of being part of a given feature. On average, a task is included in ~3 different features. Then the job of the product managers (PMs) is to decide what features are targeted for our first milestone. For the simulation, we assume the first milestone will always target exactly two features (and we always start with the same two).

The release will be ready when the number of open tasks for the two selected features hits zero.

Interestingly, look at how straight those downward-sloping lines are. Those are really easy to extrapolate. You don't even need a real simulator; the simulator is just a way of proving that the simulator is not needed. If the simulator were an AI, it would be having an existential crisis right now. Anyway, those easy-to-extrapolate lines are going to turn out to be kind of important later.

For now, note the effect on release date of the various patterns of PMs changing their minds. Each jump upward on the chart is caused by a PM dropping one feature from the release, and swapping in a new one instead. It's always drop one, add one. The ideal case is the top one: make up your mind at the start and let people finish. That finishes the soonest. The second one is unfortunate but not too bad: they change their mind once near the start. The third chart shows how dramatically more expensive it is to change your mind late in the game: we drop the same feature and add the same other feature, but now the cost is much higher. In fact, the cost is much higher than changing their minds multiple times near the beginning, shown in the fourth chart, where we make little progress at all during the bickering, but then can go pretty fast. And of course, all too common, we have the fifth chart, where the PMs change their minds every couple of months (again, dropping and adding one feature each time). When that happens, we literally never finish. Or worse, we get frustrated and lower the quality, or we just ship whatever we have when Whatever Conference comes.

The purported quote near the top, "When the facts change, I change my mind," I included because of its implied inverse: "When the facts don't change, I don't change my mind." One of the great product management diseases is that we change our minds when the facts don't. There are so many opinions, and so much debate, and everyone feels that they can question everyone else (note: which is usually healthy), that we sometimes don't stick to decisions even when there is no new information. And this is absolutely paralyzing.

If you haven't even launched - in the picture above, you haven't even hit a new release milestone - then how much could conditions possibly have changed? Sometimes a competitor will release something unexpected, or the political environment will change, or something. But usually that's not the case. Usually, as they don't say in warfare, your strategy does survive until first contact with the enemy (er, the customer).

It's when customers get to try it that everything falls out from under you.

But when customers get to try it, we've already shipped! And if we've been careful only to work on shipping features until then, when we inevitably must change our minds based on customer feedback, we won't have wasted much time building the wrong things.

If you want to know what Tesla does right and most of us do wrong, it's this: they ship something small, as fast as they can. Then they listen. Then they make a decision. Then they stick to it. And repeat.

They don't make decisions any better than we do. That's key. It's not the quality of the decisions that matters. Well, I mean, all else being equal, higher quality decisions are better. But even if your decisions aren't optimal, sticking to them, unless you're completely wrong, usually works better than changing them.

If you only take one thing away from reading this talk, it should be that. Make decisions and stick to them.

Now, about decision quality. It's good to make high quality decisions. Sometimes that means you need to get a lot of people involved, and they will argue, and that will take some time. Okay, do that. But do it like chart #4, where the PMs bicker for literally 50 days straight and we run in circles, but the launch delay was maybe 30 days, less than the 50 you spent. Get all the data up front, involve all the stakeholders, make the best decision you can, and stick to it. That's a lot better than chart #3 (just a single change in direction!) or #5 (blowing in the wind).

Sorry for the rant in that last slide. Moving on, we have more ideas to steal, this time from Kanban.

For people who don't know, Kanban was another thing computer people copied from Japanese car manufacturers of the 1950s. They, too, used actual paper index cards, although they used them in a really interesting way that you can read about in wikipedia if you care. You don't have to, though, because Kanban for software engineers doesn't use the index cards the same way, and as with Agile, I'm going to tell you to skip the paper cards anyway and do something digital, for the sake of your remote officemates. But Kanban for software uses the paper cards differently than Agile for software, and that's relevant, which we'll address in the next slide.

Kanban (for software) also uses stories and story points, just like Agile. Nothing much new there. (The car manufacturers in the 1950s didn't use stories. That's a new thing.)

Kanban, like Agile, gives the huge benefit of strict prioritization. It does it differently from Agile though. Where Agile tries to convince you to do things in a particular order, Kanban emphasizes doing fewer things at a time. Kanban is the basis of Just-in-Time (JIT) factory inventory management. Their insight is that you don't bother producing something until right before you need it, because inventory is expensive: not just for storage (which in our software analogy, is pretty cheap) but in wasted time when you inevitably turn out to have produced the wrong thing. And worse, in wasted time when you produced the right thing, but it didn't get used for another 6 months, and it broke in the meantime so you had to fix it over and over.

Going back to Critical Chain, that book will tell you the big secret of Kanban for software: unreleased software is inventory. It's very expensive, it slowly rusts in the warehouse, and worst of all, it means you produced work in the wrong order. You shouldn't have been doing something that wouldn't be needed for six months. You should have either been a) helping with the thing that's needed right now, or b) lounging around in the microkitchen reading books and eating snacks. Your buildup of low-priority inventory is slowing down the people working on the high-priority things, and that's unacceptable.

The classic example of this in Kanban for software is that we break software down into phases, say, requirements, UX design, software design, implementation, testing, etc. Let's say the TL (Tech Lead) is responsible for designing stuff, and the rest of the engineers are responsible for implementing it (the classic story of a TL who used to code but now mostly has meetings). The engineering team has a lot of work to do, so after the TL designs the current thing, they go off and start designing the next thing, and the next one, and eventually you have a buildup of design docs that haven't been implemented yet. Kanban says, this is useless; the design docs are inventory. Of course those designs will be at least partly obsolete by the time the eng team has time to implement them. Maybe goals will have changed by then and you won't implement that design at all. Meanwhile, the TL has sent around the doc for review, and there are meetings to talk about it, all of which distracts from actually finishing the work on the current feature which needs to get done.

Kanban, and JIT manufacturing, says the TL should go take a long lunch and recharge for when they're needed. Or better still, help out on the engineering phase, to push it along so that it will finish sooner. And that is, of course, the magic of Kanban. It plays some psychological games, using index cards, to convince TLs to write code and (further down the chain) engineers to help with testing, and reduce useless meetings, and stop maintaining code you don't yet need. It forces you to multitask less, which is a way of tricking you into prioritizing better.

We're engineers, so we're pretty smart. If we want, we could just, you know, dispense with the psychological games, and decide we're going to strictly prioritize and strictly limit multitasking. It takes some willpower, but it can be done. I happen to be terrible at it. I am the worst about multitasking and producing inventory. Way back at the beginning of this talk, when I showed a slide with our release goals slipping in 2016, yeah, that was us multitasking too much, which was partly my fault, and facilitated by having too much headcount.

What that means is, I know this is a lot harder than it sounds. It's actually really, really hard. It's one of the hardest things in all of engineering. Most people are very bad at it, and need constant reminders to stop multitasking and to work on the most important things. That's why these psychological games, like sprints (artificial deadlines) and index cards and kanban boards were invented. But if we want to become the best engineers we can be, we have to move beyond tricking ourselves and instead understand the underlying factors that make our processes work or not work.

That chart of the slipping release times was what reminded me to follow the rules I already knew: Prioritize. Limit multitasking.

Here's a sample kanban-for-software board, from wikipedia. I just wanted to point out visually how the psychological game works. Basically, there is only a limited amount of space on the board. To advance a story from one phase (horizontally) to the next, there needs to be an empty space in the destination phase. If there isn't, then you must first move a job from that phase (by finishing it) to the phase after that. And if there are no holes there, then something needs to get pushed forward (by finishing it) even further down the line, and so on.

If your job is to work on a phase and there are no holes left for you to fill, then you literally have to stop work, or (if you're flexible), go help with one of the downstream phases. And we enforce that by just not having enough physical space to paste your index cards. It's kinda neat, as a mental trick, because the most primordial parts of your brain understand the idea that physical things take physical space and there's no physical space left to put them in.

[Random side note: Kanban-for-factories actually does things the other way around. They send an index card upstream to indicate that they need more stuff in order to continue work. This avoids the need for an actual "kanban board" that everyone all over the factory, plus your suppliers' factories, would need to all congregate at. Instead, you have these index cards flying all around, indicating when work needs to be done. (To limit multitasking, there's a carefully limited number of cards of each type.) On the software kanban board, you can think of the "holes" moving upstream and doing the job that the "cards" do when moving upstream in a factory. That, in turn, reminds me of how doped semiconductors carry signals around: you can have excess electrons moving around (n-type) or "holes" moving around (p-type). A hole is a missing electron, and it can carry information just as well as an electron can, although slightly slower, for reasons I don't remember. Many people are surprised to hear that p-type semiconductors don't involve moving protons around, just holes.]

Time for another simulation, this time about multitasking. Our simulated SWE team is back, working just as hard as ever (and no harder), but this time we're going to extend the simulation past the first milestone. Furthermore, let's assume (perhaps too optimistically) that we have convinced our PMs to follow the advice in the previous slides, and they no longer change their minds in between milestones. The deal is this: once they set milestone goals, those are locked in, and the engineers will keep working until they're done. (This is like a sprint in Agile, except there is no strict deadline. We work until it's done, and different milestones might have different lengths.) However, PMs don't have to tell us what the next milestone contains until this one is done. They can change their minds day after day, and argue all they like, but only until the milestone starts. Then they have to start arguing about the next one. Got it? Great.

Another concept we need to cover is the idea of "aggregate value delivered." Remember that our simulation involves several features, and each feature contains a bunch of bugs/tasks, and each task can be in one or more features, and new tasks are being filed all the time. Because it's a simulation, we don't know how good these "features" really are, so let's assume they are all worth one unit of awesomeness. When we ship a feature, users don't get all the value from that feature right away; it accrues over the time they use the product. So the total value delivered to customers is a function of the number of awesomeness units that have launched, and how long ago they launched. In the charts, that's the green area under the curve. Of course, this is all very approximate, because it ignores features losing or gaining customers over time, the fact that some features have more value (and are harder to implement) than others, and so on. But the fundamental truths are the same.

Note also that new bugs/tasks are coming in all the time, across all features, whether launched or not. Once you've launched a feature, you still have to keep maintaining it forever. And no, not launching doesn't avoid that problem; the market keeps moving, and if you launch your feature a year later, the customers will have quietly become more demanding during that time. That's why in the simulation, new bugs keep getting filed in non-launched features, even if you're not working on those features yet. In the real world, those bugs wouldn't exist in any bug tracking system, but you'd sure find out about them when you go try to launch. So the simulation is assuming perfect knowledge of the future, which is not very realistic, but it's okay because you're not really acting on it. (Features you're not working on are accruing virtual bugs, but... you're not working on them, so they don't affect the simulation results until later, when you do work on them, in which case you would know about them in real life.)

So anyway, there are four simulations here. The only difference between them is simply a variation in how many features we try to squeeze into each milestone. In other words, a measure of the amount of multitasking. In the first graph, we work on, and then launch, each feature one at a time. In the second, we do two at a time. Then five at a time, and eight at a time, respectively. The simulator then helpfully sums up the total green area in each case.

Notice that in, say, the top two charts, it takes about the same time to launch the first two features and then the next two features. (The slight variations are partly due to simulator jiggliness and partly due to some second-order effects we don't need to go into, but they're small enough to ignore.) In other words, the fourth feature ships at the same time in both cases. But because you ship the first and third and fifth features earlier in the first chart, the total accrued value is higher. That's kind of intuitive, right? If you ship a simple thing sooner, some users can start enjoying it right away. They won't enjoy it as much as the bigger, better thing you ship later, and some users won't enjoy it at all, but a few will still be accruing enjoyment, which hopefully translates into revenue for you, which is great.

Notice how each subsequent release takes longer. That's because you have to keep fixing bugs in the already-launched code even while you add the new stuff. That's pretty realistic, right? And once you've launched a bunch of things, you slow down so much that you're pretty much just maintaining the code forever. You can see that in the first chart after feature #5 ships: the burndown chart is basically just flat. At that point, you either need to fundamentally change something - can you make your code somehow require less maintenance? - or get a bigger team. That goes back to our headcount scalability slides from the beginning. Is your code paying for itself yet? That is, is more value being accrued than the cost of maintenance? If so, you can afford to invest more SWEs. If not, you have to cancel the project or figure out how to do it cheaper. Scaling up a money loser is the wrong choice.

The third chart is interesting. We launch five features before we start accruing any value at all. Remember, that's the same five features as in the first two charts, and we have perfect foreknowledge of all bugs/tasks that will be filed, so it's the exact same work. But look how much less aggregate value is delivered! About 4x less than chart #2, or about 5x less than chart #1. And the only thing we changed is how much we're multitasking.

The mathematicians among you might be thinking, okay, Avery, you're cheating here by cutting off the graph at 1200 days. What if we doubled it to 2400 days? Then the majority of the chart would have 5 features launched (recall that the team isn't able to scale beyond 5 features anyhow), and the relative difference between scores would be much less. So it's artificial, right? As time goes to infinity, there is no difference.

Mathematically, that's true. But it's like big-O notation in evaluating algorithms. When n goes to infinity, every O(n) algorithm is indistinguishable. But when n is a small finite number, then different O(n) algorithms are completely different, and sometimes an O(n) algorithm completely beats an O(log n) algorithm for small n. The schedule is like that. Your software won't be accruing all that value forever. Competitors will release something, customers will move on. You have a limited window to extract value from your existing features, usually only a few years. By releasing features later, you are throwing away part of that window.

And of course there's chart #4, where we simply never launch at all, because there are too many features with too many integration points and the team just can't handle it; every time they fix one thing, another thing breaks. I don't want to name particular projects, but I have certainly seen this happen lots of times.

So, can you really get a 5x revenue improvement just by reducing multitasking? Well, I don't know. Maybe that's way too high in the real world. But how about 2x? I think I'm willing to claim 2x. And I know I can do infinitely better than chart #4.

...by the way, none of these simulated SWEs were doing any kind of simulated Kanban with simulated index cards and simulated boards with holes. We just had an informal agreement with PMs about how many features they were going to try to launch at a time (where a feature is, say, a story or a set of stories; we'll get to that soon). Kanban is just a psychological game to enforce that agreement. But if the engineers and PMs trust each other, you don't need all that fluff.

Now that we've seen the simulations, we can go back to Painless Software Schedules to emphasize one of the points Joel makes a big deal about: you can't negotiate about how fast people will work. Engineers work at different rates at different times, but it really isn't very variable. Remember those surprisingly straight downward slopes in the simulations? It's just as surprising in real life projects - I'll get to that in a bit. But that straight line makes a clear statement: that yes, I really can extrapolate to predict how long it will take to finish a particular set of tasks, and no, I really cannot do it faster and I probably will not do it slower, either. End of story, conversation over.

Once we know that - and burndown charts are quite good at convincing people that it's true - then we can have the real conversation, which is, what things actually need to go into the release and which things don't? Once we know the work rate - the slope of that line - we can move the requirements around to hit whatever date we want. We want to do this as early as possible of course, because late requirements changes are expensive, as we saw in the simulator. But since we do know the work rate, we can predict the completion date immediately, which means it's possible to have those discussions really early, rather than as we approach the deadline.

Suddenly, the manager and the engineers are on equal footing. The manager is a trained negotiator and can fire you, but the engineers have empirical truth on their side: this is not a matter of opinion, and the facts are not going to change. Better still, the engineers mostly don't care too much what order they do the features in (within reason). So the manager or PM, who is now making the scheduling decisions, doesn't need to negotiate with engineers. They can negotiate with sales or marketing or bizdev teams or whoever, who have actual feature requirements and date requirements, and agree on a combination of features and dates that is physically possible in the real world.

And then, crucially, they will not tell those dates to the engineering team. The engineering team doesn't need to know. That would be setting a goal, and goals are bad. The engineers just do the work in priority order, and don't multitask too much, and let statistics handle things. No student syndrome, no Whatever Conference rush, no working weekends, no burnout, and no lazy days needed to recover from burnout.

It sounds too good to be true, but this actually works. It's math. Coming up, we have a bunch of slides where I continue to try to convince you that it's just math and real life matches the simulation.

Actually putting it into practice

...but first, it's time to finally start talking about stories and story points.

(Most ideas from this section are blatantly stolen from Estimating Like and Adult.)

I'm going to be honest, I find the way "stories" are done in Agile to be kinda juvenile and faintly embarrassing. You make up these fake "personas" that have names, and pets, and jobs, and feelings, and you describe how they're going to go about their day until they bump into some problem, which your software is then going to solve. But it doesn't yet, of course, because the story isn't true yet, it's fiction, that's what makes it a story, and your job as an engineer is to make the persona's dreams come true.

I'm sorry, I tried to remove the snark from that paragraph, but I just couldn't do it. I think this format, with realistic people and feelings and such, is extremely useful while PMs discuss designs with UX designers (who think about character archetypes) and UX researchers (who tell literal stories about literal people who tried to use the product or mocks of the product). No problem there. That's not juvenile, that's empathy, and it's essential to good UX. I don't know exactly how they do things at Apple, but there's no way they aren't writing excruciatingly detailed stories about their users' product interactions.

But eventually the stories need to get explained to stereotypically un-empathetic software engineers, who often got into this business because they wanted to have a job that mostly did not involve empathizing. Not all of them, but many of them. And let's be honest, the engineer working on a new feature for Gmail's backend probably just doesn't need to know all those personal details. "User will be able to search for emails by keyword, and the results will be returned in no more than 2000ms, and results will be ranked by relevance" is fine, thanks, and gets to the point a lot faster.

But here's the reason Agile pushes those stories so hard. Stories are, by necessity, about a person who does something. That person is always the customer. We don't write stories about what the backend engineer does (which is how we often write bugs/tasks, and that's fine). We write stories about what the customer does. That's absolutely essential. At the level of abstraction we're talking about - features that will deliver value, like in our simulation - only things that affect the customer are allowed, because things that don't affect the customer do not deliver value to the customer.

There are lots of things engineers have to do, in order to deliver value, that do not themselves deliver value to the customer. You might have to do 10 of those to deliver value. Well, in that case, you have 10 bugs (or tasks), making up one story (feature). That's fine; that's what we did in the simulator, too. Engineers were fixing bugs for days and days before a feature was done enough to launch.

For the following discussion, that's what you need to know. Stories don't have to be written like a story, but they have to truly reflect something the user will benefit from. None of this works otherwise.

Agile doesn't just talk about stories, though, they talk about story points. Story points are an absolute breakthrough in project estimation; I might make fun of any other part of Agile (and sometimes I do), but this part is genius. You can tell it's genius because it sounds so trivial, and yet fixes numerous problems that you probably didn't even know you had. But don't worry, of course I'm going to tell you all about those problems.

In my talk, I popped up this slide on the big screen and asked people to do an experiment with me. I didn't know for sure if it would work, but it ended up working out even better than I had hoped. I asked a simple question: "How tall is the third rectangle from the left?"

One person immediately piped up, "Three!" "Ok, three what?" I asked. "Units." "What units? Tell me the height in, say, centimeters." And now people had a problem. "Can we just measure it?" someone asked. "The size depends on what screen you display it on," said another. On my laptop screen, it was maybe 4cm tall; on the projector, maybe 2 feet. "How tall would it be for, say, a typical L5 screen that Meets Expectations?"

It was surprising how quickly people understood this point for estimating rectangles, although it's somehow harder to understand for estimating tasks. Estimating absolute sizes is really tough. Estimating relative sizes is almost magically easy. Now, there was someone in the audience who thought it looked more like 4 units than 3, a disagreement which was actually excellent for multiple reasons, as we'll see in the next slide.

But before we go to the next slide, I want to say more about the problems of estimating tasks using absolute sizes. Don't experienced engineers get pretty good at estimating how long tasks will take? Let's say I'm going to write a basic Android app, and I've done a couple of basic Android apps before. Is this one easier or harder? How much harder? 2x, 3x? Okay, let's say it took 3 months to launch the last one, and this is 2x harder. Then say 6 months, right?

Well. Maybe. First of all, from when did we start counting? Conception, design, first day writing code? And hold on, are you going to hold me to that 6 month estimate? Maybe I should add a safety factor just in case. Let's say 8 or 9 months. But even worse, now we've set a goal, another one of those evil, productivity destroying goals. I have a project that I'm pretty sure will take 6 months, but maybe it's not as hard as I thought, and anyway I said 8 months, so I can relax for the first two at least. Maybe more if I'm willing to rush near the end, because I kinda remember rushing near the end of the last one. Student syndrome returns. And now we can guarantee it won't take less than the 8 month estimate, but it will almost certainly take longer, even though it was only a 6 month project. And that's why the more experienced a person gets, the longer their estimates get.

What happened here? We let the engineer know the due date. Never do that.

People have all these weird hangups about goals and about time. Story points have the astonishing ability to just bypass all that. Nobody sets a "goal" that the project will take 5 points to complete. What does that even mean? It's a five point story, it will always take 5 points to complete, no matter how long that turns out to be. It is 5 points, by definition. What are we arguing about?

With story points, there is no late, and there is no early. There's just how long it takes. Which we can measure surprisingly accurately, because story point estimates turn out to be surprisingly accurate. We just have to cheat by not telling the engineers the date we estimated, because when we do, their brains go haywire.

In the talk, we had this useful disagreement about whether the rectangle was really 3 or 4 units high, which was excellent for showing the next step: the actual process of coming to an estimate. For this, we steal a concept taught in an excellent corporate project planning class (not sure if it's still taught), called Planning Poker. In turn, he apparently stole it from some variants of Agile or Scrum.

Now, the method he teaches involves a physical deck of cards with numbers printed on them in a pseudo-fibonacci sequence. (You know this must be at least a fairly common game if you can buy these cards, which you can.) But playing cards are a pain for people in remote offices, because you have to do silly things like trying to squint at cards over a videoconference, etc. So my co-worker, Dr. Held, created a little spreadsheet version of the game which works pretty well.

You start by loading is.gd/estulator and making a copy of the spreadsheet for your team. Then you delete all the per-user tabs, and add one for each person on your team. Then you add those users' names in the first column of the Results tab.

For each round of voting, you set the Vote Code field to a new number. (You can just increment it, or use whatever formula you want. The only important thing is that it isn't the same as the last round of voting.) Then, each user opens their own tab in the spreadsheet on their own laptop, and enters the vote. Only once all users have voted (with the right vote code) are the votes revealed on the Results page.

Now, it's actually very important that you don't bias people by discussing the number of points before the vote; you want their honest answers. We screwed that up in this exercise, because we'd already talked a lot about whether the box was size 3 or 4. But oh well, we also had troublemakers who intentionally voted differently, so it worked out this time for the demo. (We still saw the results of bias: as soon as the votes were revealed, a bunch of people rushed to change their answers to make them agree with the consensus. Don't do that!)

Note that the "3 vs 4" discussion was moot anyway, because of the "pseudo-fibonacci sequence" rule. You can't vote for 4, because it's too close to 3. You have to choose 3 or 5. The lucky disagreement about 3 or 4 was a really convenient way to show why the fibonacci sequence is useful for cutting down on needless precision.

So anyway, people all vote and then you see their results. If not everyone agrees, then the tradition is that you ask one of the people with the highest vote (the one who has spoken least recently), what they think is necessary in order to do the task. Then you ask one of the people with the lowest vote. Then, you discuss more if needed, or else go immediately to a revote, and you repeat until you all agree. As a shortcut, if the votes are split between two consecutive values, you can just pick the higher one of the two. (Mathematically it turns out not to matter what the rule is for dealing with split votes, as long as you're consistent. If you bias on the high side or the low side, the formula for converting points to time, ie the velocity, will just compensate. If you bias inconsistently, you'll get increased variability and thus an increased error margin.)

The really interesting thing about this method is that nobody has any particularly vested interest in a particular story being a particular number of points. After all, nobody knows the size of a point anyway, and nobody is setting a deadline or making promises of a completion date. So there's no reason to argue. Instead of fighting to be right about the exact size, people can instead focus on why two people have such a widely varying (at least two fibonacci slots) difference of opinion. When that happens, usually it's because there's missing information about the scope of the story, and that kind of missing information is what really screws up your estimates if you don't resolve it. The ensuing discussion often uncovers some very important misunderstanding (or unstated assumptions) in the story itself, which you can fix before voting again.

Another neat feature of the voting game is that you don't spend time discussing stories where everyone agrees on the size. Maybe not everyone understands every detail of the story right now, but they apparently understand well enough to estimate the size (and maybe they disagree on various details, but apparently those details don't affect the size much). So just accept the story size, and move on.

This combination of features makes it a great way to spend your weekly or biweekly team meeting:

You talk about things that definitely matter to customers
The person who has spoken least recently gets to talk, which reduces various social biases
You don't talk about stuff that everyone already knows about
You end the meeting early if you've gone through all the stories.

There are lots of ways to estimate story points, but you might want to give this one a try.

Okay, so now you have a bunch of stories and their sizes, in points. How do you track those stories and their progress? This is where I sell you a tool, right?

Nope! We still don't need a special tool for this. The thing about stories is that they're pretty big - it takes quite a while to deliver a unit of useful functionality to a user. At least days, maybe weeks or even months. The size of a story depends on how your team wants to do things. But don't make them unnecessarily small; it doesn't help much with accuracy, and it makes the estimation a lot more work. You want estimation to be so quick and easy that you end up estimating a lot of tasks that never get scheduled - because when the PM realizes how much work they are, they realize there's something more effective to be working on.

Note that this is a departure from the Joel on Software method, where he recommends estimating line items of no more than 4 hours each. Those aren't stories, though (his method mostly pre-dates Agile and stories), those are bugs (tasks). We'll get to those later. Stories are way bigger than 4 hours, and that's perfectly fine.

Since stories are so big, that means we don't have very many of them. Since we don't have very many of them, they are easy to keep track of using any method you want. Agile people use paper index cards, and they don't get overwhelmed. The course I took recommends just keeping a spreadsheet. That works for me.

The spreadsheet is just a bunch of rows that list the stories and their estimates, but most importantly, the sequence you're planning to do them in. I suggest working on no more than one at a time, if at all possible. Since each one, when implemented, is made up of a bunch of individual tasks (bugs), it is probably possible to share the work across several engineers. That's how you limit multitasking, like kanban says to do. If your team is really big or your stories are small, you'll have to work on several stories at a time. But try not to.

Note also what the spreadsheet does not need to contain:

Estimates of how much time has already been spent working on each story
Estimates of how much time is remaining
Pointers to which bugs are in the story
Expected completion date of the story

You don't need that stuff. This spreadsheet is for the PMs and executives to write down what they want, and in what order. The engineers estimate the relative effort each of those things will take. Then the PMs re-sequence those things to their heart's delight, except for the first unfinished one, because it's too inefficient to change the goals mid-milestone (as we saw in the simulation slides earlier).

But you still need status updates, right? Everyone loves status updates. How will we track status updates in a mere spreadsheet, without endless tedium and overhead?

We'll get to that. Hang on.

Bathroom break

That was a lot of serious story estimation business, and this was the spot where we had a designated bathroom break, so I included this meditation slide where I correlated the rainfall in various sites in Peru as part of some random analytics for a team I was working with. I thought it was neat.

Also, the NOAA keeps data sets for the rainfall amount in every 8km x 8km region, between 60N and 60S latitude, every 30 minutes, going back decades. Impressive!

Better bug (or task) tracking

We've spent a lot of time talking about Agile, and Kanban, and Joel, and Stories, but now we have to diverge from all of them. We're going to steal another trick from that project management course I took: we're going to fundamentally break our planning into two completely different layers of abstraction.

The course called these, I think, Business Tasks (or maybe Business Goals) and Engineering Tasks, but those are too hard to pronounce, so let's call them Stories and Bugs.

Agile mostly deals with stories (big). Joel mostly deals with bugs (small). The big innovation in the project management course I took is that we want to do both at once. We'll make stories that are a bit bigger than the stories usually used in Agile (but maybe not as big as an "Epic"), and then we'll split each of them into numerous bugs, on about the scale Joel uses for his individual tasks.

The nicest thing about this split is it allows for a clean division of labour. PMs come up with stories; engineers estimate them; PMs sequence them; engineers then work on them in order, ideally only one at a time. On the other hand, PMs don't care at all about bugs. Only engineers care about bugs, and do them in whatever order they want, with whatever level of multitasking they think is appropriate. Engineers can just avoid attending the story sequencing meetings, and PMs can avoid attending the bug processing meetings. It really cuts coordination overhead a lot.

However, one place where we'll depart from the course I took is in handling bugs. They recommended using a spreadsheet and Planning Poker (with different point scales) for both stories and bugs. I'm going to recommend against estimating bugs at all (see next slide). I also think for bugs, you can skip using a spreadsheet and file them straight in the bug tracker. In applying the method from the course I took, I found there was a lot of tedium in keeping things synced with a bug tracker, which comes down to a common software engineering problem: if you have a bug database and a spreadsheet, then you have two sources of truth, and synchronization between them is a lot of work and tends to break down. Just use the bug tracker. For that we're going to have to use bug trackers effectively, which we'll get to a bit later.

I really like this simplification, because it makes bug fixing a first class activity. That's different from most Agile methods, which tend to treat bugs as overhead (you need to do it, but it's not accomplishing a new story, so we don't really track it and thus don't reward it). Lots of teams use bug trackers a lot, so we need a method that works with what they already do.

To explain why estimating bugs is unnecessary, I first need to convince you that all bugs are effectively the same size. For that, I made yet another simulation. This one is very simple. First (in the top chart), I created 5 stories, and gave them sizes between 1 and 13 points. If we finish them in sequence with no multitasking, we burn down from 5 stories to 4, 3, 2, 1, 0 over time. As you can see, the top line is very erratic. If you tried to estimate story completion dates by drawing a straight line from (0, 5) to (350, 0), you'd be pretty far off, most of the time.

On the other hand, the second chart has a bunch of bugs, distributed between 0.1 days and 10 days in length (with shorter times being more common; essentially a Poisson distribution). That's a range of 2 orders of magnitude, or 100x duration difference from the shortest bug to the longest. Pretty huge, right? The top chart was only one order of magnitude; much less variable.

And yet, the bottom chart is more or less a straight line. Like the top chart, it assumes no multitasking; that is, there is only a single engineer working on exactly one bug at a time, and sometimes it takes them 10 days, but sometimes they get lucky with a stream of 0.1-day bugs. But it doesn't matter! There are so many bugs that it just all averages out. If you draw a straight line from beginning to end, you're only off by a few days at any given point in the sequence. That's way too small to matter. If you add in multiple engineers on a team, the variations get even less, because even while one poor sucker works on the 10-day bug, the rest of the team keeps making progress on 0.5, 1, and 2 day bugs. [Random side note: this is also why restaurants with at least two bathroom stalls are a good idea.]

Why does that happen? The underlying reason is the Central Limit Theorem (linked above), which says that if you sum enough samples from nearly any random distribution, it ends up converging on the normal (Gaussian) distribution. A Gaussian distribution has a mean and a standard deviation. The more samples you sum together, the smaller the standard deviation is as a fraction of the average. Which is a long-winded way of saying if you have a lot of bugs, they are all about the same size. Or close enough not to matter.

You might not believe me, because this is a simulation. Good. Let's (finally!) look at some real patterns from some real projects.

Here's one from my old team's project bug tracker. Note that we're not just looking at one or two stories here; this is literally the whole component. I'm showing the burnup chart here instead of the burndown chart because for this particular team, it makes some characteristics a little more obvious. (In a burnup chart, the red dots correspond to new bugs being added, and green dots correspond to bugs being fixed. The difference between them would be a burndown chart.)

First of all, notice the distinct phases. We launched in late 2012; leading up to that, we were in heavy dogfood, so bugs were getting furiously filed and fixed. After launch, interestingly, bugs started getting filed a little more slowly (perhaps the most serious flaws had finally been resolved, which is why we launched) and fixed more slowly still, resulting in an increasing deviation between the two lines. (This corresponds to more and more bugs outstanding. In and of itself, that's pretty normal; most launched projects find that their user base is finding bugs faster than the engineers can fix them.)

In late 2015 there was a declared bug bankruptcy (long story). I have some choice words about bug bankruptcies but I'll save them for another slide. The main thing to notice is that the bankruptcy improved neither the filing nor the fixing rate. It caused a one-time bump in fixed bugs (a lying bump, of course, since they weren't really fixed, just forgotten). But the lines' slopes stayed the same.

Finally, we reorged the project in early 2017 and moved things elsewhere, which meant a lot fewer bugs in this component got fixed (nobody to fix them) but also a lot fewer got filed (nobody to file them). By mid-2017, a new team took over, and you can see bugs getting filed and fixed again, but at a much lower rate than before.

Cool story, right? But here's what you need to notice. Look how straight those lines are. There are defined segments corresponding to major events (launch; reorg; recovery). But for literally years at a time, the bug filing/fixing rates are remarkably stable. Just like in the simulations.

Here's another team I've been working with, that shall remain nameless. This chart is a little different: instead of a whole bug tracker project, this tracks only the particular milestone (hotlist) they were working on, starting in late July 2017. As you can see from the y axis labels, we're now talking about an order of magnitude fewer bugs, so you'd expect more variability.

The milestone was being worked on in July and finalized around August 4th. It includes all bugs that were still open as of August 4th, as long as they were relevant to the particular set of stories included in this milestone. The bugs could have been filed at any time back in the past (in this case, as far back as late 2014). Somewhat confusingly, bugs which were relevant to this milestone, but which were already fixed by triage time, were probably not added to the milestone. That's probably why you see this weird behaviour where the creation rate increases at first, before it stabilizes into a straight line. Notice that they explicitly stopped adding bugs to the milestone (ie. froze the requirements) around mid-to-late August. That's why the "Created" dots stop at that time, while the Resolved line continues to increase. However, it's pretty clear from the slopes of the two lines that even if they had continued to add new bugs at the same rate, they would have converged eventually. Story-relevant bugs were being found slower than fixed, which is what you need if you ever intend to launch.

Once again, look how straight those lines are. When the milestone was under construction during July, the work rate was already higher than it had been, but once it was fully committed in August, the engineers got right to work and productivity increased dramatically. You can clearly see the before, during, and after segments of the chart.

The most important observation is this: the engineers were, most likely, all working at about the same rate before the milestone goals were decided. (They were fixing a lot of bugs that don't show up in this filter: creating inventory.) But once everyone focused on the agreed-upon priority, performance on the highest priority items more than doubled. Just eyeballing it, maybe even 3x or 4x.

Well-defined strict priorities, communicated to everyone, can get you to launch 2x-4x faster than just letting engineers work on whatever seemed important at the time.

One more very artificial query: the entire internal bug tracker for a huge corporate project.

Again, even though we don't even really know what the project is, aren't the patterns interesting? It's pretty clear that the lines are, again, surprisingly straight, except for some major transitions. I'm guessing something launched around late 2012, and something else changed in mid-to-late 2014. Then something disastrous happened in early 2017, because the number of remaining open bugs (bottom chart) started shooting up out of control, with no end in sight. Observe also, though, that at the same time, the bug creation rate slowed a little (and even more over time), while the bug fixing rate slowed much more. To me, this sounds like a team being destaffed while a project is still in heavy use - a dangerous strategy at the best of times, and a recipe for angry customers.

But oh well. Even here, the key observation is that for very long stretches, other than transitions at major milestones (launch, another launch, destaffing), the lines are again surprisingly straight. I wouldn't feel too guilty taking a ruler and extrapolating a straight line to make a rough time estimate for how long something might take to finish, at any point in that chart. In fact, even after destaffing, the Remaining line is still straight. It'll never finish, because it's upward instead of downward, but it's straight and therefore predictable.

[Random side note: check out the scale on the top graph. They were handling about 15000 internally-facing bugs per year! Handling dogfood feedback for a popular project can be a nightmare. But it also gives us a clue that the reliability is probably not high enough.]

One more chart, for another project that shall remain nameless. This team did at least two bug bankruptcies (possibly another in 2012, though maybe it was just a manual re-triage since the slope isn't as steep). The problem is, once again, that the slopes of the lines didn't change.

Teams that find themselves under a seemingly endless deluge of bugs certainly have a problem: the ingress rate of their bug queue is faster than their egress rate. That will certainly cause an ever-increasing pile to build up. If your project is popular and understaffed, that's unavoidable. (Recall our simulation earlier of trying to continually launch features with the same team: after a while, you just can't make progress anymore because you're stuck maintaining all the old stuff.)

Unfortunately, some of those teams think they can improve their situation by correcting the absolute value of the number of open bugs. This doesn't work; if you can't improve the slope of the line, then a bug bankruptcy is at best a temporary fix. You might as well not be filing the bugs in the first place.

Look again at the previous slide. That team was handling 15000 bugs per year, and the number of bugs remaining open, while it fluctuated a bit, was neither increasing nor decreasing. Impressively, they had the deluge under control. So, we have evidence that it is, in fact, possible to keep your head above water. The team on this slide (and my own old team, three slides ago) was constantly falling behind, unfortunately. They're pretty normal; the more popular your project is, the more likely you are to experience it. But bankruptcy helped not at all. They need a real solution.

Let's use this as the motivator for the next section of my talk: bug triage. In other words, dealing with the inevitability that you can't fix bugs as fast as they get found.

But first, here's my simulation again. Remember chart #4? When they had all 8 features going at once, the open bug count just kept going up instead of down. I bet that team felt doomed. They could have declared bankruptcy, but that wouldn't have helped either.

The first three charts give us a clue. The same exact team, working on the same product, was able to not get into the ever-rising-tide situation. How? By strictly prioritizing, and by controlling the level of multitasking. The trick is that the first three plots don't show the count of all bugs outstanding: they just show the count of bugs related to the project's current priorities.

That's how we deal with the deluge situation: we only count the bugs that really matter. Which leaves only the question of how we decide which ones matter. That's triage.

Triage: sorting bugs

Before I try to give some tips for bug triage, I want to go back to psychology and talk a bit about what does and doesn't work well in a bug tracker.

First of all, in my corner of the world, priority levels have been made almost completely meaningless by well-intentioned teams. Depending on the project, P0 often ends up paging someone, which is rarely what you want, so you usually don't use P0. P1 often has a stupid bot attached, which pings the bug if you stop responding, trying to enforce some kind of short SLO, so the pattern is often that you file a bug, then someone looks at it for a while, makes little progress, gets annoyed by the bot, and lowers the priority to P2, only to be interrupted by the next round of new P1 bugs. It maximizes distraction without necessarily getting more bugs resolved. Oops! P2 is the default, which is fine, but of course, almost all bugs are P2 so your bug pile is undifferentiated. And at this point everyone knows that P3 and P4 both mean "will never implement." I don't know why we need two separate levels for that, but okay, that's how it is.

Some people complain about the tyranny of bug priority schemes chosen by other teams, and how they shouldn't affect my team because we want to use something less silly, but that doesn't work. Everyone has been trained that when they file a bug, those are the levels. Trying to be different will just confuse your users, because they can't keep track of every team's special snowflake policy.

"Severity," a field available in some bug tracking tools, is even weirder. I've never found anyone who can explain what it's actually for. I know what it supposedly means: there are definitions about how many end users it's affecting, and how badly. Like, if there's a typo in the menu bar of a major app, that's definitely S0 because it affects a lot of people and it's super embarrassing. Whereas a crash that affects 10 users out of a billion is maybe S4. Okay, sure. But nobody ever sorts by severity; they sort by priority. Those could just as easily have been P0 and P4. What is a P4 bug with an S0 severity? A placebo. It made someone feel good because there is a field for "yes, this bug is actually important!" and another person decided "but not important enough to work on." Maybe this method works, but I don't know for who. Anyway, I advise just ignoring severity fields.

More psychology: resolving as "Working as Intended" (the infamous "WAI") always just makes your customer angry. So does mass-resolving bugs as obsolete without looking at them. But pro tip: setting them to P3 or P4 does not create this effect. It's a useful mystery. It means the same thing, but customers are happier. Might as well do it. Instead of a bug bankruptcy, let's call it a bug refinancing.

Of course, that advice about leaving more bugs open, and all of them at P2, and ignoring the severity, leaves you with a giant undifferentiated ever-growing mass of bugs. That's no good. On the other hand, that's nothing new; if you're like one of those popular teams on the other slides, you were going to end up with that anyway, because your ingress rate is simply greater than your egress rate, no matter what. We need a system to deal with this.

Step one is understanding the triage step. It's totally different from the fixing step, for all the reasons listed in the slide.

Many teams try to take every incoming bug and make sure it's assigned to someone on the team, under the assumption that the person will reassign it or fix it eventually. Of course, that doesn't work when ingress > egress; each person's queue will be ever-growing, not ever-shrinking, and eventually the Assignee field also loses all meaning, and you have an even bigger problem. That's not what triage is. That's just giving everything high priority, which as we all know, means nothing is high priority. (Incidentally, that's the mistake we made on my old team: skipping triage and just assigning every bug to someone. I had to ask around a lot to find out that successful teams were all using a triage method that actually works!)

Okay Avery, so now you're saying don't assign the bugs to people, don't close them, set them all to average priority, and ignore severity, all while accepting that the pile will be ever-growing? This is not sustainable.

[This slide is a visual representation of unsustainable clutter. I started with an image search for "overrun by bugs" and quickly learned my lesson. Don't do that.]

Moving on to the real advice.

Step one: never just look at an entire project-wide bug tracker. It's like looking at every file on your filesystem, or every web site indexed by a search engine. Beyond a certain size, it just doesn't work anymore. Your project is almost certainly at that size. Bug trackers have queries. It's time to learn how to use them to solve your problem.

Second, a huge surprise that I didn't realize until I started asking around to the teams that were using bug trackers effectively. Bug tracker "components" (aka dividing by "project" or "area") are almost useless! They are only good for one thing: helping your end users point your bug at the right triage team. Each triage team triages one or more components. Each component is triaged by exactly one triage team. If you only have one triage team, there's no real reason to have a dozen sub-components, no matter how much sense that seems to make, because it will just confuse your end users (they won't know which component to use when filing a new bug).

Instead, use hotlists (aka "milestones" or "labels" or "tags" depending what bug tracker you use). Your triage team can figure out what a bug is about and how important it is. Their main job it so assign it to the right label(s), then move on to the next one. Once it's in a milestone or label, you no longer care what component it's in, because the eng teams working on features are working on a strictly-prioritized set of stories (remember!) and the stories are defined by one of those milestones. And the high priority labels are not depressingly ever-increasing in size.

[To give some concrete examples, I'm going to use Google Issue Tracker (issuetracker.google.com), which is not very popular, but happens to be something I use a lot. In Issue Tracker, you can create and view "hotlists" (aka "labels", specific lists of bugs that you have tagged) and "saved searches" (search queries that you have named and saved for later use, which update their contents as bugs change). Every bug tracker has something like these two features, if you explore around. But the syntax of these specific example queries is from Google Issue Tracker.]

Now all you need is a way to tell the triage team which bugs they haven't yet triaged. That's easy enough: make a bug query for "everything in my interesting set of components" but subtract "everything in my hotlist called ‘Triaged'". I call this query "Needs Triage."

Then the triage team, for each bug, will: - Assign the bug to at least one hotlist (either a release milestone, aka story, or a backlog of not-yet-scheduled features; maybe it's even relevant to more than one feature hotlist) - Assign the bug to the Triaged hotlist to make it disappear from their queue.

That's it! No need to assign it to anyone - the eng team pulling from a given hotlist can do that.

If you have a giant bug tracker that already has a lot of bugs, and you want to get started using this method, add a "modified>DATE" query filter. That will make it so that only bugs edited after the given date are subject to triage. But any old bug, if someone pings it, will immediately make it appear, so that you can triage it right away. Much better than declaring bankruptcy!

If you have a huge backlog and are thinking of declaring bankruptcy, another trick is to make a separate query that again looks at all the bugs in all your interested components, and subtracts out everything in the Triaged list. But this one, instead of having a modified>DATE absolute cutoff, uses a relative cutoff date. In this example I picked 730 days, or two years. What this means is that any bug created more than two years ago, but untriaged, will show up in the query, inviting you to triage it now. Over the course of the next two years, this moving window will slowly pass over all your remaining bugs. (If you have more than two years of backlog, increase the date accordingly. It'll take longer to go over the whole set, but then again, you already had super old bugs, so this won't make it any worse!)

Most people faced with triaging a huge bug backlog just give up. Argh, thousands of bugs? It's just too frustrating. Kill them all with fire (bug bankruptcy). This re-triage query is disproportionately much more pleasant than trying to triage them all in a burst or an ad-hoc series of bursts. If you can triage all your new incoming bugs - and I assure you that you can, using this method, because triaging bugs is way easier than fixing bugs - then you can also re-triage your old bugs. After all, the rate of ingress of bugs two years ago was almost certainly <= the rate of ingress today. Plus, you've already closed a lot of the old bugs from that time, and you'll only see the remaining ones. Plus, many old bugs will be obviously obsolete and can be politely closed right away. (My own experience, and experience of at least one other person I talked to, is that the rate tends to be roughly half the bugs are obsolete vs. still important. I've heard some people estimate 90% obsolete, but I haven't seen that be true in practice. Even if it were, closing bugs as obsolete is easy, and losing the remaining 10% of real bugs is pretty bad.)

If you're like me, you might find that the re-triage window actually moves a little too slowly, and you have extra capacity to re-triage more bugs on some days, between meetings or whatever. If so, you can just edit the query to reduce the 730d to a smaller number, a bit at a time. Then it'll take even less than two years to go through them all.

One more tip: sometimes when triaging, you need more information. People often make one of two mistakes when this happens:

They resolve the bug as "not reproducible", which makes the bug filer angry (even if you ask them to reopen it when they've added better information) or
They send the bug back to the reporter for more information, without triaging it. If you have no triage hotlist, and the original reporter never responds, or worse, they respond but forget to send it back to you, you now have an undead bug floating around in your bug tracker with no way to discover and rescue it. Or, if you do use the Triage hotlist and Needs Triage queries, you have the annoying behaviour that it will keep showing up in your triage queue until the customer responds - which may be never.

The solution is the "Needs Discussion" hotlist. When an untriaged bug needs more discussion before it can be adequately triaged, add it to this hotlist and assign it back to the sender. Change your "Needs Triage" query to exclude everything in the "Needs Discussion" hotlist. But now, add a "Needs Discussion (stalled") query with the magic "-modified:6d" filter shown above. What that does is it makes the bug disappear from your queries for 6 days, or as long as people are still communicating on the bug. Once it goes idle for 6 days, it reappears, at which time you can a) remember to continue the discussion; or b) close the bug Not Reproducible, because the person who filed it is non-responsive, and they have lost the right to be angry; or c) if you have all the information you need, finally triage it.

You can use a shorter time than 6 days if you prefer. Longer is probably a bad idea.

[Side note: it would be really nice if bug trackers had some kind of "discussion" flow feature that could track whose job it is to respond. Ideally we could just wake up right away when it's our team's turn again, rather than using the hardcoded delay.]

In the last few slides, we've made all these hotlists and queries. Now we have to put them all together. For this, you want to use your bug tracker's "dashboard" or "bookmark groups" features, which (to generalize) are generally just ordered sequences of hotlists and queries. I recommend the overall order shown above.

"...Release Milestones..." are hotlists representing stories you have scheduled for upcoming releases. "...Backlogs..." are hotlists representing stories or bugs you haven't scheduled for any release yet (and in some cases, probably never will).

Congratulations, you made it to the end! If you thought it took a long time to read all this, just imagine how long it took me to write. Phew.

Event horizon

Sun, 15 Oct 2017 20:47:05 +0000

One of the dangers of working at a big company that does a little bit of everything, even if that big company has a pretty open culture, is it's really hard to remember which things are still secret and which have been released. Since I started working for such a company back in 2011, I've still had as many opinions as ever, but I've contained them mostly to an audience at work, so that I don't accidentally leak something. People who used to follow my writing have probably noticed that there has been a lot less of it than there used to be.

Well! Luckily(?) for you, I had a bit of spare time last weekend, and managed to extract, then filter and review, my posts from the last 6 years. They've now been "seamlessly" merge-sorted into apenwarr.ca, including their original dates, so you too can see the slow evolution of my thought processes (especially regarding wifi) interspersed with the previously-published content when I finally got something that overcame the friction of making a public post.

Useless trivia: this import more than doubled the total number of posts, but only increased the total word count by about 50%. It's probably because there are a bunch of articles that are essentially just sharing a link to someone else's content, with a line or two of commentary. Such is the New Way of the Internet, I guess.

Anyway, I had some misgivings about this, as the new content is maybe not as useful, on average, as the old content. Or maybe it is. I don't know. You don't have to read it. No one, as they used to say at work, is forcing you.

More useless trivia: the automatic "Related" and "Unrelated" links generated by the latest version of the software that runs this site tends to consider the old-style (mostly long-form) content "Related" to itself, and the new, shorter-style content "Related" to itself, with few overlaps between the two. Apparently my writing style was different enough to be detected by even my very naive clustering algorithm.

git basically gives you superpowers. However, like any new superhero

Sun, 15 Oct 2017 20:24:57 +0000

, you will need to go through a montage of embarrassing and destructive mistakes as you learn to use those superpowers.

People keep telling me I should write a book with all this stuff I

Sun, 15 Oct 2017 20:24:57 +0000

keep posting about. But my deep dark secret is... I mostly got it from reading books in the first place.

SimSWE part 2: The perils of multitasking

Sat, 15 Sep 2018 18:34:09 +0000

I updated my SWE simulator[1] from a few weeks ago. This time, instead of PMs changing their minds about features, we have PMs and execs who choose not to make up their minds at all, letting launch requirements get bloated.

These four charts have the same eng team, same set of features, same pattern of bugs getting filed, same work rate, etc. The only difference is how many "features" (one "bug" can be tagged as required for one or more features) are required for a particular milestone. In this version of the simulator, when we hit a given milestone (reach ~0 bugs in the related features), we launch the product with those features intact. From then on, we have to maintain the product (bugs in already-launched features take precedence) as well as trying to develop new features, all with the same team. Thus, each subsequent feature takes longer to launch, and eventually new features can't launch at all without expanding the team size. (This is why products need bigger teams as they launch more features. In turn, that's why they need more and more revenue.)

When a feature launches, we consider that to be "value" provided to customers. It's a rough proxy for revenue, or increased customer retention, or whatever. Providing value takes time: if I buy a product, I don't get value for it instantaneously. I expect it to keep delivering value as time passes. So the height of the green line is the instantaneous value being delivered (in this simulation, it's simply proportional to number of features shipped). The green area under the curve (the integral of instantaneous product value over time) is the total value that has been delivered so far.

Using this (admittedly oversimplified) model, for this hypothetical product, shipping two features at a time would deliver 4x as much value, after 1200 days, as shipping 5 features at a time. With the same engineering team and the same features! In short, that's what makes SpaceX efficient and others inefficient. SpaceX might work engineers a little harder, but that's not the real benefit. The real benefit is that SpaceX has a clear idea of what really, really needs to happen next, and they deliver it incrementally. Others, mostly, don't.

(You might wonder what happens if we extend the plot to, say, 2000 days. Eventually, the team launches all the features it's capable of maintaining, and just keeps churning away at bugs. The value ratio between chart#2 and chart#3 keeps decreasing, but the absolute difference never goes away. Also note that value delivery usually drops off at a fixed point in the future, when your feature isn't needed anymore (eg. someone else builds a better one). So you can't just extend the time horizon forever to try to make it look better.)

Not shown: the benefits of being able to change your direction, based on customer feedback, after launch #1. The assumption that we are building the same features, but in a different order, is not very realistic. In reality, what you learn after launch #1 is so valuable that you end up changing product direction quite a lot, so early time spent working on features for future milestones is largely wasted.

(The team that completely fails to focus, chart #4, never launches anything because there are so many moving parts that new bugs are found faster than you can fix them. The usual response is to just launch something crappy and plan to fix it in v2. Notice how different that is from the first three charts, where we make no quality sacrifices at all, for a limited subset of functionality, and still launch much earlier. And then we "unrealistically" commit to fixing the bugs in already-launched features before launching new features. You could say that's the difference between many Apple products and other products.)

[1] SWE simulator v1