Chris Farris, Cloud Security Nerd at PrimeHarbor Technologies, LLC, joins Corey on Screaming in the Cloud to discuss his new project, breaches.cloud, and why he feels having a centralized location for cloud security breach information is so important. Corey and Chris also discuss what it means to dive into entrepreneurship, including both the benefits of not having to work within a corporate structure and the challenges that come with running your own business. Chris also reveals what led him to start breaches.cloud, and what he’s learned about some of the biggest cloud security breaches so far. About ChrisChris Farris is a highly experienced IT professional with a career spanning over 25 years. During this time, he has focused on various areas, including Linux, networking, and security. For the past eight years, he has been deeply involved in public-cloud and public-cloud security in media and entertainment, leveraging his expertise to build and evolve multiple cloud security programs.Chris is passionate about enabling the broader security team’s objectives of secure design, incident response, and vulnerability management. He has developed cloud security standards and baselines to provide risk-based guidance to development and operations teams. As a practitioner, he has architected and implemented numerous serverless and traditional cloud applications, focusing on deployment, security, operations, and financial modeling.He is one of the organizers of the fwd:cloudsec conference and presented at various AWS conferences and BSides events. Chris shares his insights on security and technology on social media platforms like Twitter, Mastodon and his website https://www.chrisfarris.com.Links Referenced: fwd:cloudsec: https://fwdcloudsec.org/ breaches.cloud: https://breaches.cloud Twitter: https://twitter.com/jcfarris Company Site: https://www.primeharbor.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. My returning guest today is Chris Farris, now at PrimeHarbor, which is his own consultancy. Chris, welcome back. Last time we spoke, you were a Turbot, and now you’ve decided to go independent because you don’t like sleep anymore.Chris: Yeah, I don’t like sleep.Corey: [laugh]. It’s one of those things where when I went independent, at least in my case, everyone thought that it was, oh, I have this grand vision of what the world could be and how I could look at these things, and that’s going to just be great and awesome and everyone’s going to just be a better world for it. In my case, it was, no, just there was quite literally nothing else for me to do that didn’t feel like an exact reframing of what I’d already been doing for years. I’m a terrible employee and setting out on my own was important. It was the only way I found that I could wind up getting to a place of not worrying about getting fired all the time because that was my particular skill set. And I look back at it now, almost seven years in, and it’s one of those things where if I had known then what I know now, I never would have started.Chris: Well, that was encouraging. Thank you [laugh].Corey: Oh, of course. And in sincerity, it’s not one of those things where there’s any one thing that stops you, but it’s the, a lot of people get into the independent consulting dance because they want to do a thing and they’re very good at that thing and they love that thing. The problem is, when you’re independent, and at least starting out, I was spending over 70% of my time on things that were not billable, which included things like go and find new clients, go and talk to existing clients, the freaking accounting. One of the first hires I made was a fractional CFO, which changed my life. Up until that, my business partner and I were more or less dead reckoning of looking at the bank account and how much money is in there to determine if we could afford things. That’s a very unsophisticated way of navigating. It’s like driving by braille.Chris: Yeah, I think I went into it mostly as a way to define my professional identity outside of my W-2 employer. I had built cloud security programs for two major media companies and felt like that was my identity: I was the cloud security person for these companies. And so, I was like, ehh, why don’t I just define myself as myself, rather than define myself as being part of a company that, in the media space, they are getting overwhelmed by change, and job security, job satisfaction, wasn’t really something that I could count on.Corey: One of the weird things that I found—it’s counterintuitive—is that when you’re independent, you have gotten to a point where you have hit a point of sustainability, where you’re not doing the oh, I’m just going to go work for 40 billable hours a week for a client. It’s just like being an employee without a bunch of protections and extra steps. That doesn’t work super well. But now, at the point where I’m at where the largest client we have is a single-digit percentage of revenue, I can’t get fired anymore, without having a whole bunch of people suddenly turn on me because I’ve done something monstrous, in which case, I probably deserve not to have business anymore, or there’s something systemic in the macro environment, which given that I do the media side and I do the cost-cutting side, I work on the way up, I work on the way down, I’m questioning what that looks like in a scenario that doesn’t involve me hunting for food. But it’s counterintuitive to people who have been employees their whole life, like I was, where, oh, it’s risky and dangerous to go out on your own.Chris: It’s risky and dangerous to be, you know, tied to a single, yeah, W-2 paycheck. So.Corey: Yeah. The question I’d like to ask is, how many people need to be really pissed off before you have one of those conversations with HR that doesn’t involve giving you a cup of coffee? That’s the tell: when you don’t get coffee, it’s a bad conversation.Chris: Actually, that you haven’t seen [unintelligible 00:04:25] coffee these days. You don’t want the cup of coffee, you know. That’s—Corey: Even when they don’t give you the crappy percolator navy coffee, like, midnight hobo diner style, it’s still going to be a bad meeting because [unintelligible 00:04:37] pretend the coffee’s palatable.Chris: Perhaps, yes. I like not having to deal with my own HR department. And I do agree that yeah, getting out of the W-2 space allows me to work on side projects that interests me or, you know, volunteer to do things like continuing the fwd:cloudsec, developing breaches.cloud, et cetera.Corey: I’ll never forget, one of my last jobs I had a boss who walked past and saw me looking at Reddit and asked me if that was really the best use of my time. At first—it was in, I think, the sysadmin forum at the time, so yes, it was very much the best use of my time for the problem I was focusing on, but also, even if it wasn’t, I spent an inordinate amount of time on social media, just telling stories and building audiences, on some level. That’s the weird thing is that what counts as work versus what doesn’t count as work gets very squishy when you’re doing your own marketing.Chris: True. And even when I was a W-2 employee, I spent a lot of time on Twitter because Twitter was an intel source for us. It was like, “Hey, who’s talking about the latest cloud security misconfigurations? Who’s talking about the latest data breach? What is Mandiant tweeting about?” It was, you know—I consider it part of my job to be on Twitter and watching things.Corey: Oh, people ask me that. “So, you’re on Twitter an awful lot. Don’t you have a newsletter to write?” Like, yeah, where do you think that content comes from, buddy?Chris: Exactly. Twitter and Mastodon. And Reddit now.Corey: There’s a whole argument to be had about where to find various things. For me at least, because I’m only security adjacent, I was always trying to report the news that other people had, not make the news myself.Chris: You don’t want to be the one making the news in security.Corey: Speaking of, I’d like to talk a bit about what you just alluded to breaches.cloud. I don’t think I’ve seen that come across my desk yet, which tells me that it has not been making a big splash just yet.Chris: I haven’t been really announcing it; it got published the other night and so basically, yeah, is this is sort of a inaugural marketing push for breaches.cloud. So, what we’re looking to do is document all the public cloud security breaches, what happened, why, and more importantly, what the companies did or didn’t do that led to the security incident or the security breach.Corey: How are you slicing the difference between broad versus deep? And what I mean by that is, there are some companies where there are indictments and massive deep dives into everything that happens with timelines and blows-by-blows, and other times you wind up with the email that shows up one day of, “Security is very important to us. Now, listen to how we completely dropped the ball on it.” And it just makes the biggest description that they can get away with of what happened. Occasionally, you find out oh, it was an open S3 buckets, or they’ll allude to something that sounds like it. Does that count for inclusion? Does it not? How do you make those editorial decisions?Chris: So, we haven’t yet built a page around just all of the recipients of the Bucket Negligence Award. We’re looking at the specific ones where there’s been something that’s happened that’s usually involving IAM credentials—oftentimes involving IAM credentials found in GitHub—and what led to that. So, in a lot of cases, if there’s a detailed company postmortem that they send their customers that said, “Hey, we goofed up, but complete transparency—” and then they hit all the bullet points of how they goofed up. Or in the case of certain others, like Uber, “Hey, we have court transcripts that we can go to,” or, “We have federal indictments,” or, “We have court transcripts, and federal indictments and FTC civil actions.” And so, we go through those trying to suss out what the company did or did not do that led to the breach. And really, the goal here is to be able to articulate as security practitioners, hey, don’t attach S3 full access to this role on EC2. That’s what got Capital One in trouble.Corey: I have a lot of sympathy for the Capital One breach and I wish they would talk about it more than they do, for obvious reasons, just because it was not, someone showed up and made a very obvious dumb decision, like, “Oh, that was what that giant red screaming thing in the S3 console means.” It was a series of small misconfigurations that led to another one, to another one, to another one, and eventually gets to a point where a sophisticated attacker was able to chain them all together. And yes, it’s bad, yes, they’re a bank and the rest, but I look at that and it’s—that’s the sort of exploit that you look at and it’s okay, I see it. I absolutely see it. Someone was very clever, and a bunch of small things that didn’t rise to the obvious. But they got dragged and castigated as if they basically had a four-character password that they’d left on the back of the laptop on a Post-It note in an airport lounge when their CEO was traveling. Which is not the case.Chris: Or all of the highlighting the fact that Paige Thompson was a former Amazon employee, making it seem like it was her insider abilities that lead to the incident, rather than she just knew that, hey, there’s a metadata service and it gives me creds if I ask it.Corey: Right. That drove me nuts. There was no maleficence as an employee. And to be very direct, from what I understand of internal AWS controls, had there been, it would have been audited, flagged, caught, interdicted. I have talked to enough Amazonians that either a lot of them are lying to me very consistently despite not knowing each other, or they’re being honest when they say that you can’t get access to customer data using secret inside hacks.Chris: Yeah. I have reasonably good faith in AWS and their ability to not touch customer data in most scenarios. And I’ve had cases that I’m not allowed to talk about where Amazon has gone and accessed customer data, and the amount of rigmarole and questions and drilling that I got as a customer to have them do that was pretty intense and somewhat, actually, annoying.Corey: Oh, absolutely. And, on some level, it gets frustrating when it’s a, look, this is a test account. I have nothing of sensitive value in here. I want the thing that isn’t working to start working. Can I just give you a whole, like, admin-powered user account and we can move on past all of this? And their answer is always absolutely not.Chris: Yes. Or, “Hey, can you put this in our bucket?” “No, we can’t even write to a public bucket or a bucket that, you know, they can share too.” So.Corey: An Amazonian had to mail me a hard drive because they could not send anything out of S3 to me.Chris: There you go.Corey: So, then I wound up uploading it back to S3 with, you know, a Snowball Edge because there’s no overkill like massive overkill.Chris: No, the [snowmobile 00:11:29] would have been the massive overkill. But depending on where you live, you know, you might not have been able to get a permit to park the snowmobile there.Corey: They apparently require a loading dock. Same as with the outposts. I can’t fake having one of those on my front porch yet.Chris: Ah. Well, there you go. I mean, you know it’s the right height though, and you don’t mind them ruining your lawn.Corey: So, help me understand. It makes sense to me at least, on some level, why having a central repository of all the various cloud security breaches in one place that’s easy to reference is valuable. But what caused you to decide, you know, rather than saying it’d be nice to have, I’m going to go build that thing?Chris: Yeah, so it was actually right before the last time we spoke, Nicholas Sharp was indicted. And there was like, hey, this person was indicted for, you know, this cloud security case. And I’m like, that name rings a bell, but I don’t remember who this person was. And so, I kind of realized that there’s so many of these things happening now that I forget who is who. And so, when a new piece of news comes along, I’m like, where did this come from and how does this fit into what my knowledge of cloud security is and cloud security cases?So, I kind of realized that these are all running together in my mind. The Department of Justice only referenced ‘Company One,’ so it wasn’t clear to me if this even was a new cloud incident or one I already knew about. And so basically, I decided, okay, let’s build this. Breaches.cloud was available; I think I kind of got the idea from hackingthe.cloud.And I had been working with some college students through the Collegiate Cyber Defense Competition, and I was like, “Hey, anybody want a spring research project that I will pay you for?” And so yeah, PrimeHarbor funded two college students to do quite a bit of the background research for me, I mentored them through, “Hey, so here’s what this means,” and, “Hey, have we noticed that all of these seem to relate to credentials found in GitHub? You know, maybe there’s a pattern here.” So, if you’re not yet scanning for secrets in GitHub, I recommend you start scanning for secrets in your GitHub, private and public repos.Corey: Also, it makes sense to look at the history. Because, oh, I committed a secret. I’m going to go ahead and revert that commit and push that. That solves the problem, right?Chris: No, no, it doesn’t. Yes, apparently, you can force push and delete an entire commit, but you really want to use a tool that’s going to go back through the commit history and dig through it because as we saw in the Uber incident, when—the second Uber incident, the one that led to the CSOs conviction—yeah, the two attackers, [unintelligible 00:14:09] stuffed a Uber employee’s personal GitHub account that they were also using for Uber work, and yeah, then they dug through all the source code and dug through the commit histories until they found a set of keys, and that’s what they used for the second Uber breach.Corey: Awful when that hits. It’s one of those things where it’s just… [sigh], one thing leads to another leads to another. And on some level, I’m kind of amazed by the forensics that happen around all of these things. With the counterpoint, it is so… freakishly difficult, I think, for lack of a better term, just to be able to say what happened with any degree of certainty, so I can’t help but wonder in those dark nights when the creeping dread starts sinking in, how many things like this happen that we just never hear about because they don’t know?Chris: Because they don’t turn on CloudTrail. Probably a number of them. Once the data gets out and shows up on the dark web, then people start knocking on doors. You know, Troy Hunt’s got a large collection of data breach stuff, and you know, when there’s a data breach, people will send him, “Hey, I found these passwords on the dark web,” and he loads them into Have I Been Pwned, and you know, [laugh] then the CSO finds out. So yeah, there’s probably a lot of this that happens in the quiet of night, but once it hits the dark web, I think that data starts becoming available and the victimized company finds out.Corey: I am profoundly cynical, in case that was unclear. So, I’m wondering, on some level, what is the likelihood or commonality, I suppose, of people who are fundamentally just viewing security breach response from a perspective of step one, make sure my resume is always up to date. Because we talk about these business continuity plans and these DR approaches, but very often it feels like step one, secure your own mask before assisting others, as they always say on the flight. Where does personal preservation come in? And how does that compare with company preservation?Chris: I think down at the [IaC 00:16:17] level, I don’t know of anybody who has not gotten a job because they had Equifax on their resume back in, what, 2017, 2018, right? Yes, the CSO, the CEO, the CIO probably all lost their jobs. And you know, now they’re scraping by book deals and speaking engagements.Corey: And these things are always, to be clear, nuanced. It’s rare that this is always one person’s fault. If you’re a one-person company, okay, yeah, it’s kind of your fault, let’s be clear here, but there are controls and cost controls and audit trails—presumably—for all of these things, so it feels like that’s a relatively easy thing to talk around, that it was a process failure, not that one person sucked. “Well, didn’t you design and implement the process?” “Yes. But it turned out there were some holes in it and my team reported that those weren’t there and it turned out that they were and, well, live and learn.” It feels like that’s something that could be talked around.Chris: It’s an investment failure. And again, you know, if we go back to Harry Truman, “The buck stops here,” you know, it’s the CEO who decides that, hey, we’re going to buy a corporate jet rather than buy a [SIIM 00:17:22]. And those are the choices that happen at the top level that define, do you have a capable security team, and more importantly, do you have a capable security culture such that your security team isn’t the only ones who are actually thinking about security?Corey: That’s, I guess, a fair question. I saw a take on Twitter—which is always a weird thing—or maybe was Blue-ski or somewhere else recently, that if you don’t have a C-level executive responsible for security with security in their title, your company does not take security seriously. And I can see that past a certain point of scale, but as a one-person company, do you have a designated CSO?Chris: As a one-person company and as a security company, I sort of do have a designated CSO. I also have, you know, the person who’s like, oh, I’m going to not put MFA on the root of this one thing because, while it’s an experiment and it’s a sandbox and whatever else, but I also know that that’s not where I’m going to be putting any customer data, so I can measure and evaluate the risk from both a security perspective and a business existential investment perspective. When you get to the larger the organization, the more detached the CEO gets from the risk and what the company is building and what the company is doing, is where you get into trouble. And lots of companies have C-level somebody who’s responsible for security. It’s called the CSO, but oftentimes, they report four levels down, or even more, from the chief executive who is actually the one making the investment decisions.Corey: On some level, the oh yeah, that’s my responsibility, too, but it feels like it’s a trap that falls into. Like, well, the CTO is responsible for security at a publicly traded company. Like, well… that tends to not work anymore, past certain points of scale. Like when I started out independently, yes, I was the CSO. I was also the accountant. I was also the head of marketing. I was also the janitor. There’s a bunch of different roles; we all wear different hats at different times.I’m also not a big fan of shaming that oh, yeah. This is a universal truth that applies to every company in existence. That’s also where I think Twitter started to go wrong where you would get called out whenever making an observation or witticism or whatnot because there was some vertex case to which it did not necessarily apply and then people would ‘well, actually,’ you to death.Chris: Yeah. Well, and I think there’s a lot of us in the security community who are in the security one-percenters. We’re, “Hey, yes, I’m a cloud security person on a 15-person cloud security team, and here’s this awesome thing we’re doing.” And then you’ve got most of the other companies in this country that are probably below the security poverty line. They may or may not have a dedicated security person, they certainly don’t have a SIIM, they certainly don’t have anybody who’s monitoring their endpoints for malware attacks or anything else, and those are the companies that are getting hit all the time with, you know, a lot of this ransomware stuff. Healthcare is particularly vulnerable to that.Corey: When you take a look across the industry, what is it that you’re doing now at PrimeHarbor that you feel has been an unmet need in the space? And let me be clear, as of this recording earlier today, we signed a contract with you for a project. There’s more to come on that in the future. So, this is me asking you to tell a story, not challenging, like, what do you actually do? This is not a refund request, let’s be very clear here. But what’s the unmet need that you saw?Chris: I think the unmet need that I see is we don’t talk to our builder community. And when I say builder, I mean, developers, DevOps, sysadmins, whatever. AWS likes the term builder and I think it works. We don’t talk to our builder community about risk in a way that makes sense to them. So, we can say, “Hey, well, you know, we have this security policy and section 24601 says that all data’s classifications must be signed off by the data custodian,” and a developer is going to look at you with their head tilted, and be like, “Huh? What? I just need to get the sprint done.”Whereas if we can articulate the risk—and one of the reasons I wanted to do breaches.cloud was to have that corpus of articulated risk around specific things—I can articulate the risk and say, “Hey, look, you know how easy it is for somebody to go in and enumerate an S3 bucket? And then once they’ve enumerated and guessed that S3 bucket exists, they list it, and oh, hey, look, now that they’ve listed it, they know all of the objects and all of the juicy PII that you just made public.” If you demonstrate that to them, then they’re going to be like, “Oh, I’m going to add the extra story point to this story to go figure out how to do CloudFront origin access identity.” And now you’ve solved, you know, one more security thing. And you’ve done in a way that not just giving a man a fish or closing the bucket for them, but now they know, hey, I should always use origin access identity. This is why I need to do this particular thing.Corey: One of the challenges that I’ve seen in a variety of different sites that have tried to start cataloging different breaches and other collections of things happening in public is the discoverability or the library management problem. The most obvious example of this is, of course, the AWS console itself, where when it paginates things like, oh, there are 3000 things here, ten at a time, through various pages for it. Like, the marketplace is just a joke of discoverability. How do you wind up separating the stuff that is interesting and notable, rather than, well, this has about three sentences to it because that’s all the company would say?Chris: So, I think even the ones where there’s three sentences, we may actually go ahead and add it to the repo, or we may just hold it as a draft, so that we know later on when, “Hey, look, here’s a federal indictment for Company Three. Oh, hey, look. Company Three was actually this breach announcement that we heard about three months ago,” or even three years ago. So like, you know, Chegg is a great example of, you know, one of those where, hey, you know, there was an incident, and they disclosed something, and then, years later, FTC comes along and starts banging them over the head. And in the FTC documentation, or in the FTC civil complaint, we got all sorts of useful data.Like, not only were they using root API keys, every contractor and employee there was sharing the root API keys, so when they had a contractor who left, it was too hard to change the keys and share it with everybody, so they just didn’t do that. The contractor still had the keys, and that was one of the findings from the FTC against Chegg. Similar to that, Cisco didn’t turn off contractors’ access, and I think—this is pure speculation—I think the poor contractor one day logged into his Google Cloud Shell, cd’ed into a Terraform directory, ran ‘terraform destroy’, and rather than destroying what he thought he was destroying, it had the access keys back to Cisco WebEx and took down 400 EC2 instances that made up all of WebEx. These are the kinds of things that I think it’s worth capturing because the stories are going to come out over time.Corey: What have you seen in your, I guess, so far, a limited history of curating this that—I guess, first what is it you’ve learned that you’ve started seeing as far as patterns go, as far as what warrants inclusion, what doesn’t, and of course, once you started launching and going a bit more public with it, I’m curious to hear what the response from companies is going to be.Chris: So, I want to be very careful and clear that if I’m going to name somebody, that we’re sourcing something from the criminal justice system, that we’re not going to say, “Hey, everybody knows that it was Paige Thompson who was behind it.” No, no, here’s the indictment that said it was Paige Thompson that was, you know, indicted for this Capital One sort of thing. All the data that I’m using, it all comes from public sources, it’s all sited, so it’s not like, hey, some insider said, “Hey, this is what actually happened.” You know? I very much learned from the Ubiquiti case that I don’t want to be in the position of Brian Krebs, where it’s the attacker themselves who’s updating the site and telling us everything that went wrong, when in fact, it’s not because they’re in fact the perpetrator.Corey: Yeah, there’s a lot of lessons to be learned. And fortunately, for what it’s s—at least it seems… mostly, that we’ve moved past the battle days of security researchers getting sued on a whim from large companies for saying embarrassing things about them. Of course, watch me be tempting fate and by the time this publishes, I’ll get sued by some company, probably Azure or whatnot, telling me that, “Okay, we’ve had enough of you saying bad things about our security.” It’s like, well, cool, but I also read the complaint before you file because your security is bad. Buh-dum-tss. I’m kidding. I’m kidding. Please don’t sue me.Chris: So, you know, whether it’s slander or libel, depending on whether you’re reading this or hearing it, you know, truth is an actual defense, so I think Microsoft doesn’t have a case against you. I think for what we’re doing in breaches, you know—and one of the reasons that I’m going to be very clear on anybody who contributes—and just for the record, anybody is welcome to contribute. The GitHub repo that runs breaches.cloud is public and anybody can submit me a pull request and I will take their write-ups of incidents. But whatever it is, it has to be sourced.One of the things that I’m looking to do shortly, is start soliciting sponsorships for breaches so that we can afford to go pull down the PACER documents. Because apparently in this country, while we have a right to a speedy trial, we don’t have a right to actually get the court transcripts for less than ten cents a page. And so, part of what we need to do next is download those—and once we’ve purchased them, we can make them public—download those, make them public, and let everybody see exactly what the transcript was from the Capital One incident, or the Joey Sullivan trial.Corey: You’re absolutely right. It drives me nuts that I have to wind up budgeting money for PACER to pull up court records. And at ten cents a page, it hasn’t changed in decades, where it’s oh, this is the cost of providing that data. It’s, I’m not asking someone to walk to the back room and fax it to me. I want to be very clear here. It just feels like it’s one of those areas where the technology and government is not caught up and it’s—part of the problem is, of course, having no competition.Chris: There is that. And I think I read somewhere that the ent—if you wanted to download the entire PACER, it would be, like, $100 million. Not that you would do that, but you know, it is the moneymaker for the judicial system, and you know, they do need to keep the lights on. Although I guess that’s what my taxes are for. But again, yes, they’re a monopoly; they can do that.Corey: Wildly frustrating, isn’t it?Chris: Yeah [sigh]… yeah, yeah, yeah. Yeah, I think there’s a lot of value in the court transcripts. I’ve held off on publishing the Capital One case because one, well, already there’s been a lot of ink spilled on it, and two, I think all the good detail is going to be in the trial transcripts from Paige Thompson’s trial.Corey: So, I am curious what your take is on… well, let’s called the ‘FTX thing.’ I don’t even know how to describe it at this point. Is it a breach? Is it just maleficence? Is it 15,000 other things? But I noticed that it’s something that breaches.cloud does talk about a bit.Chris: Yeah. So, that one was a fascinating one that came out because as I was starting this project, I heard you know, somebody who was tweeting was like, “Hey, they were storing all of the crypto private keys in AWS Secrets Manager.” And I was like, “Errr?” And so, I went back and I read John J. Ray III’s interim report to the creditors.Now, John Ray is the man who was behind the cleaning up of Enron, and his comment was “FTX is the”—“Never in my career have I seen such a complete failure of corporate controls and such a complete absence of trustworthy information as occurred here.” And as part of his general, broad write-up, they went into, in-depth, a lot of the FTX AWS practices. Like, we talk about, hey, you know, your company should be multi-account. FTX was worse. They had three or four different companies all operating in the same AWS account.They had their main company, FTX US, Alameda, all of them had crypto keys in Secrets Manager and there was no access control between any of those. And what ended up happening on the day that SBF left and Ray came in as CEO, the $400 million worth of crypto somehow disappeared out of FTX’s wallets.Corey: I want to call this out because otherwise, I will get letters from the AWS PR spin doctors. Because on the surface of it, I don’t know that there’s necessarily a lot wrong with using Secrets Manager as the backing store for private keys. I do that with other things myself. The question is, what other controls are there? You can’t just slap it into Secrets Manager and, “Well, my job is done. Let’s go to lunch early today.”There are challenges [laugh] around the access levels, there are—around who has access, who can audit these things, and what happens. Because most of the secrets I have in Secrets Manager are not the sort of thing that is, it is now a viable strategy to take that thing and abscond to a country with a non-extradition treaty for the rest of my life, but with private keys and crypto, there kind of is.Chris: That’s it. It’s like, you know, hey, okay, the RDS database password is one thing, but $400 million in crypto is potentially another thing. Putting it in and Secrets Manager might have been the right answer, too. You get KMS customer-managed keys, you get full auditability with CloudTrail, everything else, but we didn’t hear any of that coming out of Ray’s report to the creditors. So again, the question is, did they even have CloudTrail turned on? He did explicitly say that FTX had not enabled GuardDuty.Corey: On some level, even if GuardDuty doesn’t do anything for you, which in my case, it doesn’t, but I want to be clear, you should still enable it anyway because you’re going to get dragged when there’s inevitable breach because there’s always a breach somewhere, and then you get yelled at for not having turned on something that was called GuardDuty. You already sound negligent, just with that sentence alone. Same with Security Hub. Good name on AWS’s part if you’re trying to drive service adoption. Just by calling it the thing that responsible people would use, you will see adoption, even if people never configure or understand it.Chris: Yeah, and then of course, hey, you had Security Hub turned on, but you ignore the 80,000 findings in it. Why did you ignore those 80,000 findings? I find Security Hub to probably be a little bit too much noise. And it’s not Security Hub, it’s ‘Compliance Hub.’ Everything—and I’m going to have a blog post coming out shortly—on this, everything that Security Hub looks at, it looks at it from a compliance perspective.If you look at all of its scoring, it’s not how many things are wrong; it’s how many rules you are a hundred percent compliant to. It is not useful for anybody below that AWS security poverty line to really master or to really operationalize.Corey: I really want to thank you for taking the time to catch up with me once again. Although now that I’m the client, I expect I can do this on demand, which is just going to be delightful. If people want to learn more, where can they find you?Chris: So, they can find breaches.cloud at, well https://breaches.cloud. If you’re looking for me, I am either on Twitter, still, at @jcfarris, or you can find me and my consulting company, which is www.primeharbor.com.Corey: And we will, of course, put links to all of that in the [show notes 00:33:57]. Thank you so much for taking the time to speak with me. As always, I appreciate it.Chris: Oh, thank you for having me again.Corey: Chris Farris, cloud security nerd at PrimeHarbor. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry, insulting comment that you’re also going to use as the storage back-end for your private keys.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Josh Doody, Owner of Fearless Salary Negotiation, joins Corey on Screaming in the Cloud to discuss how to successfully negotiate your salary, and why it’s important to do so even in times of economic uncertainty. Corey and Josh chat about some of the hidden reasons why salary negotiation is critical to job seekers, and what goes into determining salary bands behind the scenes. Josh also reveals why he feels there’s some stagnancy in the big tech job market, and why it’s critical for job seekers to have a balanced view of the value that they provide to employers when negotiating salary. Josh also describes some of the unexpected ways salary negotiations can come up throughout the interview process, and how to best handle the discomfort of negotiation. About JoshJosh is a salary negotiation coach who works with senior software engineers and engineering managers to negotiate job offers with big tech companies. He also wrote Fearless Salary Negotiation: A Step-by-Step Guide to Getting Paid What You're Worth, and recently launched Salary Negotiation Mastery to help folks who aren't able to work with him 1-on-1.Links Referenced: Company website: https://fearlesssalarynegotiation.com Twitter: https://twitter.com/joshdoody LinkedIn: https://www.linkedin.com/in/joshdoody/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Developers are responsible for more than ever these days. Not just the code they write, but also the containers and cloud infrastructure their apps run on. And probably the billing on top of that - which is neither here nor there. And a big part of that responsibility is app security — from code to cloud.That’s where Snyk comes in. Snyk is a frictionless security platform that meets teams where they are, automating application security controls across their existing tools, workflows, and the AWS application stack — including seamless integrations with AWS CodePipeline, Amazon EKS, Amazon Inspector and several others.Deploy on AWS. Secure with Snyk. Learn more at snyk.co/scream. That’s S-N-Y-K-dot-C-O/scream. And my thanks to them for sponsoring this ridiculous nonsense!Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I have a returning guest today who hasn’t been on for a couple of years, at least. Josh Doody is the owner of fearlesssalarynegotiation.com and focuses on a problem that’s near and dear to my heart from my previous life as an employee, salary negotiation, specifically emphasizing software engineers, if I have that right. Josh, thanks for joining me.Josh: Yeah. You have it exactly right. It’s great to be here, and good to talk to you again, Corey.Corey: I used to be practiced at doing salary negotiations, which is a very roundabout way of saying I got fired a lot, so I got lots of practice at doing it. And I found that it was a very strange experience that was completely orthogonal to anything else that I did in the course of my day-to-day. Now, of course, I you know, negotiate AWS bills for a living among many other things, and do a lot of sales work, and yeah okay, now it’s a lot more germane. But back in my engineering life, it was the one time I got to really negotiate that wasn’t, you know, haggling with some vendor somewhere when I’m trying to buy a burrito, was salary negotiation, and I felt utterly unprepared for it.Josh: Yeah, I think most people feel that way and you summarized pretty well why that is. And that's, you know, let's say you have a really robust career, you're around for decades, you know, you’re working in lots of companies, you might have, I don’t know, let’s say ten, a dozen job offers that you negotiate, you know, give or take. And that’s not very many reps for doing something that’s as consequential as, you know, negotiating your actual pay. Which, depending on how senior you are, could be literally negotiating, like, you know, multiple cars’ worth of value per year that you’re going to [laugh] that you’re going to earn. But you don’t get the reps, so most people just kind of—I think they kind of don’t even think about it until they have to think about it until it’s directly in front of them. And then they just kind of power through it, get it over with or even totally ignore it and just get back to the thing that they’re doing in their career, which is why you show up to work.Corey: It feels, on some level, like it’s one of those areas where people wind up thinking about it long after they really should have. These days, it feels like salary negotiation process, more or less should start when you start debating, huh, maybe I’ll change jobs. Like, it feels like it’s really that early, not when you have an offer sitting in your inbox that needs a response by the end of the week. Right, wrong, or am I just thinking about this in ridiculous ways?Josh: No, I think you’re right. So, you can start thinking about it, you know, you get a job offer in your inbox and you can start thinking how do I negotiate this now, but you know, you’re going to be in a less secure position to do a strong negotiation at that moment than you would have had you begun thinking about it when you mentioned, which is, like, you’re actually thinking about changing jobs, or, you know, maybe you just got a cold call from a recruiter and they’re at a company that you’re kind of interested in working with. So, maybe I will talk to this recruiter instead of just blocking them or whatever. And so, the whole process can begin at that moment when they say, “Hey, you know, we have this opportunity that we think you might be interested in. What do you think?”And then, you know, early on, they’ll even kind of officially start the negotiation, at least in my mind, where they’ll say, “But before we really go too far on this, like, what are you hoping to make here? You know, what are your salary expectations if you come work here?” And you’re kind of off to the races at that moment. But even if they don’t say that out loud, that’s something that you should be thinking about from the beginning, which is, you know, maybe most broadly, how do I position myself to get the best possible version of the job offer that they’re willing to give and to leave myself the most latitude to improve that job offer to be the maximum that they can afford to pay me or the maximum that their budget allows or however you want to frame that. So, short answer, yeah, I think you’re right that most people think about it as sort of an afterthought, either after they’ve already started a job and they go, “Huh, I wonder if that guy over there is making more than I am?” Or, you know, “Shoot. I think I moved too fast there. Maybe I should have done something a little bit better.” When they could think about it way earlier in the process than that.Corey: Since I was last on the job market, there have been some changes, at least here in California, that have had a somewhat significant impact, to my understanding. First, job salary changes need to be posted in job ads, which I think is great—and that’s occurring in a number of different states—and also it is now against the law, in California—or at least against public policy—to ask what someone’s current compensation is and your salary history and dive into that. Now, that’s all well and good, but I also have been asked a number of questions that are not exactly… green, when it comes to being in the middle of an interview. And, “You’re not legally allowed to ask me that question,” is that a heck of a pushback.Josh: Yeah, I think that I’ve had a couple conversations about this recently, but also over the past few years, especially on the—you know, you mentioned the two prongs of that idea: what’s your current salary, what are you making now? And, you know, what is the salary that you expect to make? And so, kind of one by one, states are outlawing potential employers’ ability to ask about what you’re currently making. And then I’ve also heard some agitation lately that there might be some federal legislation that’s coming down that might just kind of take that off the table. As you mentioned, recruiters, companies, organizations, however you want to model them are very clever, and so there are always ways, you know, even if they’re indirect questions, you know, you don’t ask them what they’re currently making, but you ask them something that gives you some insight into what they currently might be thinking.Also, if you’re in the big tech world—which you mentioned you negotiate AWS contracts—in the big tech world, they don’t necessarily have to ask you what you’re currently making if they know that you’re an L4 software engineer at Google. They can probably approximate it pretty well. And of course, they know that because you’re going to have to tell them, you know, with a resume or when you’re interviewing, that’s kind of how you get in the door. So, that’s an interesting thing. But I still say, avoid that. Try to avoid giving him as much information as possible.And I think the most important thing with the current salary idea is, you just don’t want to say it out loud. You want to make sure that they can’t quite grab onto that because you make it too easy for them when they know what your current salary is to just do sort of a cost-plus version of offering you a job, which is, “Well, you’re making this much now. We’ll just add 10% to that,” and that’s your new job offer, when you know, that’s not how you level up quickly and in big ways.And then you mentioned the salary expectations. I do think it’s great that a lot of job offers now will have a salary range in them. That’s a question that I see a lot is, like, how do I know that I’m not going to waste 25 hours of my personal time and maybe a trip across the country for a job, where when they finally make the offer, it’s just laughably low? And the answer is, you know, hopefully, they have something that you can grab onto in the actual job description that says, here’s what the range looks like. But even then, you’ll notice if you look carefully—I saw one yesterday, and I don’t remember where I saw it, but it was like, “Yeah, our range of salary is, you know, 120k up to 290k, depending on geographic region.”And it’s like… I mean, technically, that’s a salary range, but they don’t tell you what the regions are, how they map, and all that stuff. So, you’re not getting a lot of information there; you’re just getting sort of an approximate number. But it’s still helpful to know that information. And it’s also helpful to not disclose that information. If you have a number in mind that you’re hoping for, it’s not in your best interest to share that with the company.So, I think at least what you can do is look at the job description. If they have some kind of a range, take a look at it, see if it feels like, okay, this is something I can work with or if it’s just, you know, there’s no way that that would ever work for me and you can just pass on it and save yourself some time.Corey: For me, one of the things that always frustrated me was that at the start of looking into a job, there’s always the big question that they ask that has been the socially acceptable paths at screwing you over, and the knowing how to answer that is important. But I still bungled it a number of times whenever I was out of practice, which is quite simply, “Okay, what is it that you expect to make in your next role? What are your compensation requirements?” And it feels like answering that at the beginning of the process just completely sets your course for how the rest of that process is going to go.Josh: It does and it’s something that’s very subtle and clever because most people will not perceive that to be a negotiation tactic when it is. And also you mentioned earlier in the context of, like, asking you what your current salary is that it can be perceived as sort of a gatekeeping question. Like you mentioned, you know, you’re in the middle of an interview and somebody pops a question at you like, “What is your [laugh] what’s your current salary?” And you’re looking at an interviewer and you’re thinking, “If I don’t give him this information, then I’m saying no to an interviewer, and how’s that going to go over?”This is the same kind of thing. When, you know, at any point in the process, they might ask you what your salary expectations are, it could be on the first screening call, it could be right before, they like to hold this till right before they make an offer where you go through the whole interview process and then right before they’re going to extend an offer, they say, “Hey, you know, I’m going to go to the hiring committee and make a recommendation that we hire. But before I do that, you know, what are you hoping to make if we actually do extend an offer and I go talk to the comp team?” And it can feel like, well, gosh, I better tell them the answer to that question because they literally just said, basically, like, “I have an offer for you, but first, I need this information from you.” And it can feel really kind of daunting to say, “No, I’m not going to give you that.”So, the question is, you know, should you give them that and how? You shouldn’t, as I mentioned earlier. Giving them salary expectations, I’ll give kind of a brief summary of why it’s not a good idea. I think a good way to reframe that question, you know, what are you what are you hoping to make if you join our team, is, you know, “Hey, you know, we have a giant company here. We’ve got tens of thousands of employees. We’ve got thousands of engineers that are at your level and doing your kind of work. We have salary surveys that we run once a quarter, or once a month, that are super expensive. We know what everybody else in the industry is doing. We know what the value of this role is to our company. We know how many other people are applying for this job. We know how many open seats we have. You don’t know any of that stuff, but even though you don’t know any of that stuff, why don’t you take a wild guess what we would pay you to do this job at this company at this moment?”Corey: And then of course, we’re going to use it against you later, when you wind up having what you view as a negotiation, like, “Ah, but you said at the beginning of the process that this would be sufficient.”Josh: Yeah. So, that’s the problem, right? As you take that wild guess and you’re going to do one of two things. It’s basically 0% that you’re going to hit the nail on the head in terms of you guessed the actual maximum compensation that they would pay you to do the job, it’s very unlikely.Corey: You’re either going to guess too high and then basically get yourself disqualified—Josh: Yeah.Corey: —you’re going to get too low and leave money on the table, or you’re going to get it exactly right, but you’ll never know whether you got it exactly right or whether you guessed low.Josh: Right. Even if you do guess exactly right, you won’t know that you did. And so, of course, if you guess low, like you said, you leave money on the table. And the really pernicious thing is, you could guess low and still feel great about the result and never know it or not find out until the next time you get a job, which is to say, you know, you say a number that’s well below the bottom of the minimum that they could pay you and so you say, I don’t know, to use round numbers, you say $100,000. And they go, “Great, how about 120?”And you say, “Wow, they must really like me. They’re going to pay, I just said 100 and they said 120. That’s amazing.” And really what’s going on is they’re looking at, you know, their internal pay structure and they’re like, we can’t pay less than 120, like, the pay structure starts at 120. So, we’ll pay 120, which is the literal bottom that you could make.You feel like you got a huge win of a 20% bump, but the reality is, you’re probably not anywhere near the middle of that pay range and you’re way behind the eight ball already. And of course, you could overshoot. And the worst-case scenario is you overshoot so far that you basically disqualify yourself from the process early. So, it’s like, if it’s on that first screening call, and you say—Corey: And they view you as being fundamentally unserious, where it’s a, okay, the compensation for this role is 100 to 130, for example—to use made-up numbers—and you come in asking for 340. It’s… okay like, there’s no point in even doing a counter and having a negotiation at this point. We are so far apart, that it doesn’t work out that way.Josh: Right, which on the surface, seems like oh, well, I just saved a bunch of time. But in reality, what you may have done is sort of like knocked yourself out of the entire hiring funnel for them, when what could have happened is perhaps you could have as you interviewed, you could have aligned better with a more senior role that would have had a higher pay range that you would have been a better fit for, you could have changed their budget based on the way that you present in your interviews and what they perceive from you. And who knows, maybe you actually do get an offer that looks like 340 because they say, “Oh, wow, we had you leveled as a, you know, an L6 and really should be, like, at an L7. So, how about this, you know, this senior or principal or lead role over here that we’ve been trying to fill for six months, we now realize you might be a good fit for that role. Why don’t you go talk to that hiring manager, and if we have to, we’ll just put you into that hiring stream?” Instead of, you set a giant number and we got to kick you out because there’s no room for you here.Corey: This is all well and good and we’re talking about effectively cash comp and salaries, but so many companies these days seem to tie a fair part of their compensation to the equity portion of it. And because remember, everything’s up and to the right. Always. The end. Until one day, it’s very much not.And now we’re taking a look and seeing that, for example, Amazon stock has largely been in the toilet for a couple of years. It’s what, 50% off of what it was at the peak.Josh: Yeah.Corey: So it’s, on some level, when you’re negotiating comp, it feels like you’re being asked to predict the future of how well the company does. And at these multibillion-dollar company scales, are you really going to be in a position personally to meaningfully impact the stock price? Like, well, not positively anyway. And it just feels like it’s a bit of a shell game where if you can’t spot the sucker, it’s probably you. Because I wanted to be an engineer, not a stockbroker.Josh: Yeah, I mean, first of all, you’re right, that no individual engineer is really going to be impacting the bottom line of Google.Corey: Unless I take the site down.Josh: Right. Well, I was just [laugh]—man, you beat me to the punch on that one. Yeah. So, there is a possibility that one engineer could have a dramatic impact, but not the kind that you would hope if [laugh] you’re also tied to their stock price, right? So, there’s a couple of ways that I think about this.One of them, you mentioned the Amazon stock going down. So, one thing that’s really interesting about that is really what Amazon is doing is they’re targeting a total annual compensation number with their stock. And so, they start with their current known stock value—I don’t know if they’re doing this now, but for many years, they were just kind of building in a year-over-year growth number of 10 to 15%. So, we’re going to give you this much total comp and we’re targeting 300k total comp per year. And if you kind of map it out based on the base salary and the equity that vests and the signup bonuses they give you in years one and two, then it looks like a pretty flat, like, 300k a year when you build in that stock growth.So, the magic question that I started talking to—and had a couple of internal recruiter friends, like, last year, mid-year last year when things were looking pretty bad, and the question that I don’t think that they had an answer to at the time and now they have answered is, well, what do we do when the stock doesn’t grow 10 to 15% and actually kind of collapses, like, takes a huge nosedive? And the answer is that Amazon is still targeting a total comp of 300k a year. And they go back and they say, “Well, here’s some more RSUs at the current value to kind of makeup for that. Here’s your new vesting schedule on these.” They essentially are giving refreshers, and here’s the new vesting schedule.And so, at least in Amazon’s case, they did kind of try to right the ship. But the reason is that something you alluded to, you’re not really getting equity in the company because you impact the company; you’re getting equity in the company because it’s another way for them to kind of generate, quote-unquote, “Cash flow” of some kind or comp, that isn’t, you know, dollars coming off the books. So, this is something I think that’s kind of a TBD is, Amazon has now answered this, which is we’re going to give him—because otherwise, they’re going to have a mass exodus, right, like, if you thought you’re going to make 300k a year and you’re actually going to make 180k a year, that’s a huge dropoff, and you’re probably going to be looking elsewhere. So, they say, “Well, here’s some more RSUs.”The question is, you know, what will other companies do? All of this is, you know, we’re talking about public companies here. So, there’s a big difference between, like, Amazon stock, Google stock, whatever—or GSUs, whatever you want to call them—and then private, pre-IPO equity, and all these different things. I see those as much more in the category of what you described, which is, you know, if you’re getting stock options on, like, an early stage, you know, like, an early stage startup, right, they’re raising, like, their first or second or third round, you are going to have maybe kind of a large impact on the trajectory of the company, but on the price of that you have almost no agency whatsoever because of all the options that they have for dilution and all that other stuff that can go on and whether you even have shares that are going to be liquid at some point and all that stuff. So, I see that as much more like, you’ve just got to look at the company, the cash that they’re paying you, how you feel about that, how you feel about the mission of the company, and understand that you’ve got, you know, you’ve got some lotto tickets in that company and who knows, maybe it goes to the moon and you get to go along for the ride, but much less certain than, you know, like I said, like, an Amazon-type situation where they actually will give you even more RSUs if the stock tanks over the course of the year.Corey: What are you seeing these days in terms of the macroeconomic conditions as a result? Like, some wit on Twitter said that the correction in the market has identified the grim reality that there are more engineers making $600,000 a year than there are engineering problems that need $600,000 engineers to fix them. So, there’s a certain, are people being overcompensated? Is there a correction in the market? Is that changing the world of salary negotiation and peoples’ job mobility?Josh: I think—working backwards—yes, job mobility is affected right now. I mean, I’ve seen you know, even in my own business, there are just fewer people reaching out and saying, “Hey, I have an offer at a big tech company.” Which is, you know, all over the news, layoffs. First, it was hiring freezes, right? This is late last year, October last year-ish, Q3, Q4, last year. They kind of said, “Oh, we’re going to hire—we’re going to slow down for a little bit on this hiring.”And then it was layoffs. And so, the last several months have been layoff after a layoff, you know, 5% here, 10% there at lots of different companies. Paradoxically, a lot of those companies are still up into the right, if you’re looking at their stock price, lately. And I think a lot of that is back to the first thing that you said, which is, you know, do we have more engineers that are kind of sitting around looking for problems to solve than there are problems to solve? And I think the answer was probably, yes.Certainly, the pandemic, interest rates where they were, and all these other kind of macro-economic things, which I won’t opine on too much because I’m not super-educated on them, but I understand them well enough to understand that basically, it was a better investment for a big company to hire an engineer, than necessarily to try to find somewhere to invest that money because interest rates were so low, so it’s hard to find a nice quote-unquote, “Risk-free” return on the investment, so they said, “Why not? We’ll just hire some engineers and maybe we’ll get a bigger ROI there. We’ll try a bunch of different projects, we’ll put a bunch of people and maybe we’ll go to the moon.”Corey: A lot of speculative or strategic hiring—Josh: Yeah.Corey: —and then okay, then you have—something that companies do when they have extra money is they greenlit additional projects. And when things get tight, they wind up effectively removing some of those projects from the table. And what I think people misunderstand in many cases is that compensation of employees is always more expensive than the infrastructure they work on, with very rare exceptions. So, the AWS bill is always secondary to payroll expenses, and fixing AWS cost takes time, effort, and engineering work, whereas laying people off requires a couple of difficult conversations—that companies increasingly seem to be bungling—and that’s the end.Josh: Yeah. I think you’re right about that. I mean, payroll, it’s an old saw in businesses is that payroll is the biggest expense, right? Like, it’s very expensive to hire people. But it could be the kind of thing, like you said, “We’ll just fire up a bunch of these projects. We’ve been thinking about them anyway. We can’t really invest this money anywhere else for a good return, so we’ll take some shots here.” Right?But then interest rates go up and oh, there are places that I can get a nice return on this investment of cash, so maybe, you know, some of these projects that aren’t going so well, we’re going to shut them down. We’re going to lean up a little bit. We’re going to increase our margins, reduce our payroll costs, and just kind of ride this economic turmoil out and see how it goes. And who knows, maybe they’ll fire some of those projects up later. But yes, it’s much easier to say we’re laying off 10% of our workforce tomorrow than it is to make a lot of other changes, especially on the expenses side.That’s one of the few expenses I think that a company has direct control over and can simply reduce if they choose to. And that’s kind of where we are right now, I think. And so, you mentioned economic mobility or job mobility. It’s definitely way down. And I think the reason is that, you know, I mean, if I’d been through layoffs at companies that I worked at before, right?It’s a really uncomfortable feeling, where the person that was sitting next to you in the office next to you gets laid off and you’re sitting there wondering, “Am I going to be next?” And the last thing that you’re going to do is start kind of poking your head up and looking for jobs and making it known that you’re shopping, or even go ask for a raise or something because you’re just trying to keep your head down and maybe the scythe will pass over me [laugh], right? Maybe they’re going to miss me in this next round of layoffs if I just keep my mouth shut and I keep typing away here on my keyboard. So, I think a lot of that is going on where people are, if they’re still employed, they’re happy to be there and they’re just going to kind of hunker down. And then if they’re not employed, there’s not a lot of them, you know, especially if you’re coming from big tech, you would want to go most of the time to another big tech company.Like, that’s why you’re there, a lot of people aspire to work for big tech, they want to be in that ecosystem. But if all the big tech companies are laying people off or freezing hiring, there’s nowhere to go. And so, there’s nowhere to move if they want to. They don’t want to make it known that they’re looking to move because they don’t want to draw attention to themselves if they’re still employed. And if they’re unemployed, the options for them to go somewhere are slim, but they probably have a severance package that they’re kind of going to milk for a little bit and see if things kind of warm up again and they can go find somewhere to move to. So, everything feels, in the big tech level, there’s a lot of inertia right now. People are just kind of sitting back, and there’s a lot of friction, and they’re just kind of hanging back to see what happens.Corey: And also, at least from my somewhat naive perspective, it feels like when people do get offers and they have made the decision to move on, there’s an increasing sense of they should be thankful for what they get and not rock the boat by asking for more. But I vehemently disagree, to be very clear on this. I think that negotiate for the best package you can get. Do it in good faith and be responsible about it, but money that is life-changing to you is a rounding error at best for a lot of these companies. You will always be more invested in this than the counterparty that you’re negotiating against. But it just really throws me and on some level, makes me sad watching people take less than they could be getting.Josh: Yeah. I mean, I think that’s just the nature of people who are spooked when the economy is doing weird stuff. And it’s an understandable reaction to it, but I agree with you. Just yesterday—you know, I’m in a bunch of [laugh] a bunch of different developer Slacks. I don’t know which one this was, but I was in a developer Slack—and somebody was saying exactly that.They’re like, “Yeah, I got this offer, it seems pretty good. I don’t know if I should bother negotiating it, you know? Like, I, I—shouldn’t I just be, you know, pretty satisfied with this thing that I got?” And I wrote a long response, which was, the short version of it was basically, “No.” And the reason is, think about all the costs that the company has incurred just to get to the point where they made you an offer?It was expensive for them. Believe me, a lot of money has been spent. They’ve gotten all the way to the finish line with you. I mean, the number is at least in the thousands of dollars; it’s probably in the tens of thousands of dollars, especially if they flew out for an onside or something. If you went through an interview loop, just do the math on, well, I talked to six people for about an hour apiece. That’s six hours right there of really expensive time probably at, like [laugh], you know, senior manager and above pay rates.So, they put a lot of money into trying to fill this role. They want to fill the role, especially in this environment. If you’re that deep in the process, they’ve got a role that they probably feel is pretty crucial to be filled. So, you’ve got a lot of reasons that you should be optimistic about the value that you’re bringing to that role and I think it’s a mistake to not see what the maximum value is that you can get in return for the work that you’re going to provide for them. So, I do think that being scared is not the right response there, again because they’ve made a significant investment to get to the point of making an offer.And remember their fallback, right, if you negotiate with them and they don’t want to give you any more, I have never seen—and I underline the word ‘never—I’ve never seen that a big tech company, somebody negotiates, and the big tech company says, “Nevermind. Get out of here.” Job offer went away. I’ve never seen it.Corey: I was about to ask that because I’ve heard about it at startups. And back in years when I was on Twitter a lot more than I am now, I periodically have people messaging me saying that this happened to them. What should they do? Do I want to put the company on blast and the rest? It’s something I learned relatively early on in that process was before I go off half-cocked—which I’m thrilled to do—can I get a screenshot of that email exchange back and forth?Because it hasn’t happened often, but once or twice, what I have clearly seen is that the company makes an offer in good faith and the person comes back with what they believe is the professional way to negotiate for more money and it is such a screaming red flag that is basically fists-of-ham-powered here that companies are like, “Oh, thank God. We just learned this giant red flag. We can get out of this super easy by rescinding the offer because of the negotiation, rather than asking them who they think they’re speaking to like that.” And that is the way of getting out of it in those cases. I don’t think that’s particularly common, and as you say, I don’t suspect that happens at big tech companies.Josh: I mean, it’s not a good look, right? There was a period last year where a big tech company… [laugh] I don’t know if this is privileged information or not, but they were actually resending offers, and it’s because they had gotten out over their skis. They were hiring way ahead of where they should have been, and then of course, everything turned and they had to start reducing headcount. So, they did, and then they started actually res—Corey: I can think of at least three companies off the top of my head that would qualify for that story. A lot of it came, but no one made an announcement that we’re rescinding offers, but it doesn’t take much on Twitter when you start seeing wow, 15 people all popped up at the same time claiming that. I wonder if they’re telling—Josh: Weird.Corey: —the truth, given they’ve never—Josh: It’s a pattern.Corey: —interacted with each other?Josh: Yes.Corey: Yeah…Josh: So, without putting them on blast, obviously, the reason I’m not saying their names is I would be putting them on bla—it’s not a good look, right? Nobody wants to know that they’re in the interview process for a company who is known for rescinding offers. And so, you know it wasn’t a decision they took lightly. And so, to your point, companies are not just going to willy-nilly start pulling back offers because that’s really terrible PR. I mean, it’s just not a good idea.So, it’s either what you said, which is—and this is something, like when I say, “I’ve never seen it; underline the word never,” right, what I mean is I work with people one-on-one for a living; that’s what I do. None of my clients have ever had a job offer rescinded from big tech company. That’s not to say it hasn’t happened for reasons like you mentioned.Corey: Yeah, I have to imagine that the emails you help them craft to respond to these things don’t start off with, “Now, listen here, asshole…”Josh: Right.Corey: Like, I sort of get the sense that that’s not quite the negotiating tone that you take, most days.Josh: [laugh]. No. There’s no, like, you know, “I’ve CC'd my lawyer on this email… and blah, blah—” you know, that’s not how I negotiate; it’s not a good way to negotiate if you want to get good results and build rapport with people. So, in general, if you follow what I would call, like, kind of good negotiating practices—which is self-serving because I would say that I’ve created a lot of them for salary negotiations, right—and if you’re following the best practices there, everybody’s understanding that we’re having a professional, business conversation among, you know, [unintelligible 00:26:52] professionals. We’re trying to find the best result, that’s good for everybody and we’re going to get there.And so, as long as you’re not—you know, you mentioned, you know [laugh], I say, you know, pounding your fist on the desk and making ultimatums and stuff, like, that’s not how I negotiate; you can hear it in the way I talk. You’re going to be fine. They’re not going to be rescinding offers and therefore, you have pretty much carte blanche to, in good faith, negotiate with them to see if there’s more room to negotiate. And how aggressive you’re being and what you’re asking for, these are all things that are dependent on the situation, right? There’s some cases where asking for another half a million a year would be completely absurd; in some cases where it’s totally appropriate [laugh] and it just depends on what your situation is.Corey: For some roles, if you just accept the offer as given, you will lose status in their eyes, on some level. For example, one of the challenges we’ve had with contract negotiation has been when we hire folks to work on negotiations. It’s one of those, like, “Okay, do we want somebody who accepts the first offer or do we want someone who really fights us tooth and nail over every aspect of it?” And it’s, on some level, it’s an extension to the interviewing process there.Josh: Yeah.Corey: I don’t know what the right answer is on that I mostly shrug and make that my business partner’s problem.Josh: I think it’s a good metric to see, especially in your business, like, you want to know not only, like, can they negotiate contracts and all this stuff, but you want to know, like, how savvy are they in terms of business? And I think, in general, a person who just accepts the first offer they get in business, I will not say that they are not savvy because I don’t know that, but it’s not a signal of savviness, I think, to just outright accept the first thing that comes your way in business, in general.Corey: Oh, when I wind up interviewing people in person and telling them about offers and whatnot, in years past, it was always a, would you like me to sign it right now? It’s… to be honest, I’m actually starting to reconsider having given it to you at all because only someone who is deranged is going to sign a contract they haven’t read, and we don’t try to hire for that.Josh: Right. Yeah, I mean, that’s just not—especially when your job is negotiating—you want to know that this person is running a number of filters when they’re considering, you know, what is probably a kind of a life-altering decision for them, right? And so, one of those filters is, “Are the terms of this contract good for me? Is there anything dangerous in here?” And one of the filters is, “Am I being appropriately compensated for the value that I’m going to bring?” That’s the big one that I focus on, right?And there’s a number of those filters and I think—you know, when I’m coaching someone, the first thing that we always say when a job offer comes in is, “Hey, thanks for the offer. I appreciate it. If you wouldn’t mind, I’d like”—Corey: Yeah, acknowledge receipt.Josh: Yeah, yeah, “Thank you. I got the offer. Thank you for that.” And also acknowledge it and be thankful. Like, you know, “Hey, I appreciate it.” Like, “We have now made a significant step forward in this whole process that we’re going through. I appreciate what you’ve done to get us here. I appreciate the fact that you’re giving me an offer. That demonstrates a lot of trust and all these things. And if you don’t mind, I’d like to take a day or two to think it over.” And then the last thing is, “Would you mind sending me a bullet-point summary in email of the numbers that you said, so I make sure I don’t mess them up?”Because you’re trying to avoid the very unlikely chance that they said numbers and you heard different numbers and then you start negotiating based on the different numbers and everything just kind of go sideways. So, that’s the first three things: “Thank you for the offer.” “Can I have a day or two?” “Would you send me a bullet-point summary?” It doesn’t have to be formal; just bullet points is fine.Corey: Would always irked me—and I you tend to see this a lot more with early career folks, but there’s also this is a common failure mode as well among people who have been in one job for a while where they have gotten completely rusty at doing the interview dance. And they tend to view jobs as being this benevolent gift bestowed upon them by the employer and they become falling over themselves, just thanking them for the opportunity and the rest. And no, no, no, no, no. A job is a mutual exchange of value. You are solving a problem that the company has, and in turn, they are bringing you in and giving you a not inconsiderable amount of money—presumably—to wind up solving that problem for them, you both come out better than you were independently. That is what a job is. Confusing the power dynamic for something else feels, to me at least, like it’s the wrong way to view things.Josh: Yeah. I’ve always not liked even the meta sort of way that we talk about jobs as, like, jobs created, jobs destroyed, somebody gave me a job. I don’t know when that term—I would be curious actually, to kind of know the etymology of that term, but like, when we started describing jobs is the thing that was given or taken or—and instead, what it is, is it’s a verbal contract or written contract. It’s like, “Hey, I’m going to do work for you because I bring value. You’re going to pay me because I’m creating value and because it’s valuable to you. And we’re going to figure out, you know, what’s the meet-in-the-middle number, basically, that makes us both feel good about that business transaction.”You as a company can’t do what you’re doing without people like me. And I as a person have found a good place to flex that particular muscle at your company. That’s great for both of us. Let’s figure out how, you know, we can both be happy with it. So, it’s definitely not that, you know, nobody’s really doing anybody favors there. You’re both entering into a mutual exchange of value for business reasons.And of course, your business reasons are different than theirs, but that’s what they are. So yeah, I like the way that you frame that and you think about it. And I do think it can be a little harmful for people to have that perspective, especially like if they’re in a position where they’re thinking, “Oh, I’m so thankful that this company is willing to give me this job.” You know, “They’re gifting me with this job and they’re creating this job for me.” That’s actually not what’s happening.Corey: Something that I want to talk about, just because I’ve gone through this process myself as an employee, who interviewed a lot, negotiated a lot, and got hired a lot. Then I started this place and I’ve been on the other side of the table. And it turns out that it’s not that hard to be a human being when you’re the hiring manager and making these decisions. And understand the fact that yeah, you may be hiring five people this month, but these people aren’t accepting five job offers a month—you hope—and going through that entire process themselves. And extending grace is just not that hard.Like, one thing that we’ve done since day one here has always been to put our salary compensation for the job in the job posting so we don’t waste anyone’s time. Where, like, “Well, what do you want to make?” It’s like, if someone walks in to buy a car, the salesperson doesn’t say, “Well, how much do you want to pay for it?” It doesn’t work that way. It’s, “This is the thing we’re offering. This is the compensation we can build here. We don’t do equity, so there’s no funny money stories.”And yeah, I know you’d like to make three times more. So, would we, but without growth, that doesn’t become sustainable. So, let’s talk about how to get there. And being a responsible, decent human being is not that hard in the hiring space, but no one tells those stories because it’s more fun, and outrage goes around the internet three times while the truth is still putting its boots on, where the idea of these horrible companies with people who don’t know what they’re doing just completely kicking themselves.Josh: Yeah, you know, it’s funny, I thought, two things kind of flashed in my mind while you were talking. And the first one was, you know, I was a hiring manager for a while. And a lot of the sort of philosophy that I built around, like, asking for raises and promotions, right? Like, I have a process for that that’s different than negotiating job offers, but the way that I developed it was as a hiring manager, my employees would say, you know—in their one-on-one or something—like, “Hey, you know, I feel like, you know, here’s what happened when I started the company. For these reasons, I feel like I’m like, way behind where I should be in pay. Can you help?”And so, the way that I kind of approached that was, yes, I want to help them, but I cannot really do that on my own. I need a lot of information that I don’t have for them. So, what information do I need from them to have them help me help them to get them a raise or get them or promotion, right? And so, I started thinking about it from the manager side of, like, essentially, kind of like a compassionate approach to, like, I need you to give me information and I will do what I can for you. And that was like, my whole philosophy with that, which is, I think I agree with you, but I need you to kind of prove to me that you should be paid more. Not because I don’t believe you, but because I can’t get you more money if I can’t make that case, and I’m not able to make that case on my own, right?And so, I think that there is room for hiring managers to be compassionate in terms of like you said, just putting numbers in a job description, just so the person knows, like, yeah, this seems like it’s probably approximately for me. Or you know, like I said, as a hiring manager saying, “Hey listen, I need you to bring me these three things. If you bring me those three things, that’d be the information that I need to go to finance or to HR or whoever and see if we can get you a raise or get you a promotion.” And if we can’t, then I’ll figure out, like, what are the next steps for you to get there to do that thing. And I think that in general, that’s just removing friction from, like, forward-moving business processes and that’s a good way to go.I think for you, right, you’re saving yourself time, by putting those numbers in the job description, you’re saving your applicants time by putting the numbers in the job description, and you’re also kind of setting the terms for, like, the conversation that you’re going to have, in addition to the abilities that they’re bringing, the skills that they’re going to bring, the things they’re going to do for your company, you’re also saving time on what the pay is going to be, what the compensation is going to look like approximately for that role, so that you can say, “Are we having a conversation whose parameters are known to us and that we agree upon to start with? Yes? Okay, great. Let’s keep talking.” Otherwise, no, and maybe they should go somewhere else or maybe you need to rework your job listings because nobody is [laugh] applying for that job, right?But it’s all data. It’s a feedback loop. And it can be done compassionately. It doesn’t have to be this, kind of, aggressive, you know, Shark Tank-style, like, I’m going to beat you over the head with this thing and get my result that I want, regardless of how you feel about it or, you know, how it makes you feel as a person.Corey: One last thing that I want to comment on this is that I’ve done this a fair bit, but if I wound up finding myself on the job market, I would absolutely reach out to you for coaching on the salary piece of it, just because you are a dispassionate third-party who is very aware of what the current state of the market is, you have a bunch of different offerings these days that range from a bunch of free articles on your website all the way up to individualized personalized coaching. I have bullied friends of mine into becoming your clients with a, “If he doesn’t justify his fee, I will pay it instead of you.”Josh: [laugh]. Thank you for that, by the way.Corey: Of course. And I’ve never had to do it because you know what you’re doing and the results absolutely speak for themselves. But my question is, what are you doing these days that’s between the everything free on a website if you read it and, individualized one-on-one coaching? Are there now points in between those two extremes?Josh: Yeah. I think you actually summarized the whole spectrum pretty well. I mean, I’ve made—since I started my business seven-and-a-half years ago, one of the primary things that I did to start was, I’m going to create as much free content as I can and make it publicly available, just so that people can find it. Because there’s no way that I can talk to tens of thousands, hundreds of thousands of people one-on-one. And so, that’s there on fearlesssalarynegotiation.com.The other end is my one-on-one coaching which I developed because, frankly, people were reaching out and saying, “Hey, will you coach me through this?” And I said, “Sure.” And I developed that business. And then in between is, I created a program… three or four months ago, I launched it. It’s called Salary Negotiation Mastery, but it is essentially me sitting down late last year with an instructional designer and asking the question, how can I teach the methodology I use in my one-on-one coaching to people who can’t afford to hire me or just aren’t inclined to hire a consultant to help them do something? And how can I teach that to them in a way that they can execute it on their own to get a good result, or possibly, you know, they’re just at an income bracket right now where it doesn’t make sense for them to hire me?And so, that’s kind of the middle ground there is it’s a coaching program, but it’s wrapped in a do-it-yourself thing, where you have, you know, worksheets and workbooks and things that you can use to do it yourself using exactly the methodology, even the templates and things that I use with my clients. And the only thing, of course, that you’re missing is my brain, but I’ve put as much of that as I can into the program as possible. So, that’s the spectrum is: free articles, Salary Negotiation Mastery in the middle, and then the top tier offering that I have is, like you said, one-on-one bespoke coaching, where I work with somebody one-on-one. And I don’t do a lot of that, just because it requires a lot of time and I like to give a ton of focus to everybody that I work with.Corey: Which makes sense because it also feels like it’s a very time-sensitive issue as well. Like with AWS bill, great people want it fixed now, but then procurement can slow things down. But that’s okay; there’s another bill coming next month. Job offers, speaking as a hiring manager, if you accept the job, terrific, that’s great. If you don’t. Then okay, that’s unfortunate, but it happens. But either way, let us know so we can either continue speaking to other people or begin planning for you to show up. So, it feels like there’s very much a strong sense of urgency around the entirety of what you do.Josh: Yeah, especially for the coaching. And the whole offering for my coaching offering is really designed to make sure that I have enough bandwidth available for someone to call me. I mean, literally, as we’re in this recording right now, I could have gotten an application in the email that would say, “Hey, I have a job offer in hand from Google. It’s for this much money, it’s for this level, can you help?”Corey: “And they’re on the other line. Please respond immediately.”Josh: Yes. And their recruiter is pressuring me for an answer. They want to get back to the hiring manager. And so, I need to be able to respond quickly, get back to that person, have an intro call, get to know them, see if I can help in their situation, kickoff, you know, this afternoon or tomorrow morning, get a counteroffer over in the next 24 to 36 hours, that kind of thing. And so, in order to do that, I’ve got to build an offering that allows me to have enough bandwidth and, kind of, agency over my schedule so that I can just sort of jump in immediately into the middle of a process that’s ongoing and help the person get the best result possible.So, I enjoy that to be honest with you. I like, kind of, being called on in emergency situations like that. It’s really good. But of course, I had to structure the offering so that it facilitates it so that I’m not, you know, already booked on the phone eight hours a day and unable to even look at my email until tomorrow or something because it just wouldn’t work.Corey: Yeah. There’s something to be said for being able to take a vacation.Josh: Yes. Which takes some planning, but can be done [laugh]. And it means I just have to turn off the application sometimes [laugh].Corey: Glad to see things are still going well for you. You started your business a few months before I started mine and it’s great to see that we’re both still failing to go out of business every month.Josh: [laugh]. That’s how I see it, too. I’m still here. Now, what [laugh]? That’s every month on the first when I do the books.Corey: [laugh]. I hear you. I really want to thank you for taking the time to speak with me. If people want to learn more—and if they’re changing jobs, they absolutely should—where should they go to find out?Josh: fearlesssalarynegotiation.com is the first place to go. I’m also on Twitter. I don’t tweet a lot kind of actively, I probably should do better on that, but I’m at @joshdoody on Twitter and I’m very responsive on there. So, you could ping me on there or, you know, connect with me on LinkedIn if you wanted to; I’m also joshdoody there. But fearlesssalarynegotiation.com is the best place to go, especially if you’re kind of in a time crunch. Everything is just right there for you to jump in and kind of grab, you know, the free resource that you might need or apply to work with me as a coaching client.Corey: Oh, the template emails are glorious.Josh: Yes. Those are one of my favorite things on the site. They don’t look like other emails that people write, and something I take a lot of pride in is communicating well and creating good email templates that help a lot of people.Corey: Oh, in TextExpander, for a decade now, I’ve had a fill-in-the-blanks templated resignation letter, which it turns out, most people don’t have. But I don’t need it much these days, but it is useful to wind up giving to people from time to time. Like, “So, how do I tell my boss to take this job and shove it?” It’s like—Josh: Well—Corey: —life is long and the industry is small. Go vent to your friends over beers. But there’s very little upside and huge potential downside, so write the formal thing. Here you go. And it turns out that it’s sort of cathartic, just filling that out. And it’s like, oh, that’s what this [unintelligible 00:42:05]. And it often helps people step back from the ledge sometimes. Or pushes them right off, depending.Josh: I think that’s a useful service.Corey: But yeah, the [unintelligible 00:42:11] template emails are way better than mine.Josh: [laugh]. Well thanks, I appreciate it. It means a lot to me.Corey: [laugh]. Josh Doody the owner of fearlesssalarynegotiation.com. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment and be sure to include your salary expectations.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Rich Mogull, SVP of Cloud Security at FireMon, joins Corey on Screaming in the Cloud to discuss his career in cybersecurity going back to the early days of cloud. Rich describes how he identified that cloud security would become a huge opportunity in the early days of cloud, as well as how cybersecurity parallels his other jobs in aviation and emergency medicine. Rich and Corey also delve into the history of Rich’s involvement in the TidBITS newsletter, and Rich unveils some of his insights into the world of cloud security as a Gartner analyst. About RichRich is the SVP of Cloud Security at FireMon where he focuses on leading-edge cloud security research and implementation. Rich joined FireMon through the acquisition of DisruptOps, a cloud security automation platform based on his research while as CEO of Securosis. He has over 25 years of security experience and currently specializes in cloud security and DevSecOps, having starting working hands-on in cloud over 12 years ago. He is also the principle course designer of the Cloud Security Alliance training class, primary author of the latest version of the CSA Security Guidance, and actively works on developing hands-on cloud security techniques. Prior to founding Securosis and DisruptOps, Rich was a Research Vice President at Gartner on the security team. Prior to his seven years at Gartner, Rich worked as an independent consultant, web application developer, software development manager at the University of Colorado, and systems and network administrator.Rich is the Security Editor of TidBITS and a frequent contributor to industry publications. He is a frequent industry speaker at events including the RSA Security Conference, Black Hat, and DefCon, and has spoken on every continent except Antarctica (where he's happy to speak for free -- assuming travel is covered).Links Referenced: FireMon: https://www.firemon.com/. Twitter: https://twitter.com/rmogull Mastodon: [https://defcon.social/@rmogull](https://defcon.social/@rmogull) FireMon Blogs: https://www.firemon.com/blogs/ Securosis Blogs: https://securosis.com/blog TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. My guest today is Rich Mogull, SVP of Cloud Security over at FireMon now that I’m a bit too old to be super into Pokémon, so I forget which one that is. Rich, thanks for joining me. I appreciate it.Rich: Thank you. Although I think we need to be talking more Digimon than Pokémon. Not that I want to start a flame war on the internet in the first two minutes of the conversation.Corey: I don’t even have the level of insight into that. But I will say one of the first areas where you came to my notice, which I’m sure you’ll blame yourself for later, is that you are the security editor behind TidBITS, which is, more or less, an ongoing newsletter longer than I’ve been in the space, to my understanding. What is that, exactly?Rich: So, TidBITS is possibly the longest-running—one of the longest-running newsletters on the internet these days and it’s focused on all things Apple. So, TidBITS started back in the very early days as kind of more of an email, I think like, 30 years ago or something close to that. And we just write a lot about Apple and I’ve been reading about Apple security there.Corey: That’s got to be a bit of an interesting experience compared to my writing about AWS because people have opinions about AWS, particularly, you know, folks who work there, but let’s be clear, there is nothing approaching the zealotry, I think I want to call it, of certain elements of the Apple ecosystem whenever there is the perception of criticism about the company that they favor. And I want to be clear here to make sure I don’t get letters myself for saying this: if there’s an Apple logo on a product, I will probably buy it. I have more or less surrounded myself with these things throughout the course of the last ten years. So, I say this from a place of love, but I also don’t wind up with people threatening me whenever I say unkind things about AWS unless they’re on the executive team.Rich: So, it’s been a fascinating experience. So, I would say that I’m on the tail end of being involved with kind of the Mac journalist community. But I’ve been doing this for over 15 years is kind of what I first started to get involved over there. And for a time, I wrote most of the security articles for Macworld, or a big chunk of those, I obviously was writing over a TidBITS. I’ve been very lucky that I’ve never been on the end of the death threats and the vitriol in my coverage, even though it was balanced, but I’ve also had to work a lot—or have a lot of conversations with Apple over the years.And what will fascinate you is at what point in time, there were two companies in the world where I had an assigned handler on the PR team, and one was Apple and then the other was AWS. I will say Apple is much better at PR than [laugh] AWS, especially their keynotes, but we can talk about re:Invent later.Corey: Absolutely. I have similar handlers at a number of companies, myself, including of course, AWS. Someone has an impossible job over there. But it’s been a fun and exciting world. You’re dealing with the security side of things a lot more than I am, so there’s that additional sensitivity that’s tied to it.And I want to deviate for a second here, just because I’m curious to get your take on this given that you are not directly representing one of the companies that I tend to, more or less, spend my time needling. It seems like there’s a lot of expectation on companies when people report security issues to them, that you’re somehow going to dance to their tune and play their games the entire time. It’s like, for a company that doesn’t even have a public bug bounties process, that feels like it’s a fairly impressively high bar. On some level, I could just report this via Twitter, so what’s going on over there? That feels like it’s very much an enterprise world expectation that probably means I’m out of step with it. But I’m curious to get your take.Rich: Out of step with which part of it? Having the bug bounty programs or the nature of—Corey: Oh, no. That’s beside the point. But having to deal with the idea of oh, an independent security researcher shows up. Well, now they have to follow our policies and procedures. It’s in my world if you want me to follow your policies and procedures, we need a contract in place or I need to work for you.Rich: Yeah, there is a long history about this and it is so far beyond what we likely have time to get into that goes into my history before I even got involved with dealing with any of the cloud pieces of it. But a lot about responsible disclosure, coordinated disclosure, no more free bugs, there’s, like, this huge history around, kind of, how to handle these pieces. I would say that the core of it comes from, particularly in some of the earlier days, there were researchers who wanted to make their products better, often as you criticize various things, to speak on behalf of the customer. And with security, that is going to trigger emotional responses, even among vendors who are a little bit more mature. Give you an example, let’s talk about Apple.When I first started covering them, they were horrific. I actually, some of the first writing I did that was public about Apple was all around security and their failures on security disclosures and their inability to work with security researchers. And they may struggle still, but they’ve improved dramatically with researcher programs, and—but it was iterative; it really did take a cultural change. But if you really want to know the bad stories, we have to go back to when I was writing about Oracle when I was a Gartner analyst.Corey: Oh, dear. I can only imagine how that played out. They have been very aggressive when it comes to smacking down what they perceive to be negative coverage of anything that they decide they like.Rich: Yeah, you know, if I would look at how culturally some of these companies deal with these things when I was first writing about some of the Oracle stuff—and remember, I was a Gartner analyst, not a vulnerability researcher—but I’m a hacker; I go to Blackhat and DEF CON. I’m friends with the people who are smarter than me at that or have become friends with them over the years. And I wrote a Gartner research note saying, “You probably shouldn’t buy any more Oracle until they fix their vulnerability management process.” That got published under the Gartner name, which that may have gotten some attention and created some headaches and borderline legal threats and shade and all those kinds of things. That’s an organization that looks at security as a PR problem. Even though they say they’re more secure, they look at security as a PR problem. There are people in there who are good at security, but that’s different. Apple used to be like that but has switched. And then Amazon is… learning.Corey: There is a lot of challenge around basically every aspect of communication because again, to me, a big company is one that has 200 people. I think that as soon as you wind up getting into the trillion-dollar company scale, everything you say gets you in trouble with someone, somehow, somewhere, so the easiest thing to do is to say nothing. The counterpoint is that on some point of scale, you hit a level where you need a fair bit of scrutiny; it’s deserved at this point because you are systemically important, and them’s the breaks.Rich: Yeah, and they have improved. A lot of the some of the larger companies have definitely improved. Microsoft learned a bunch of those lessons early on. [unintelligible 00:07:33] the product in Azure, maybe we’ll get there at some point. But you have to—I look at it both sides a little bit.On the vendor side, there are researchers who are unreasonable because now that I’m on the vendor side for the first time in my career, if something gets reported, like, it can really screw up plans and timing and you got to move developer resources. So, you have outside influences controlling you, so I get that piece of it. But the reality is if some researcher discovered it, some China, Russia, random criminals are going to discover it. So, you need to deal with those issues. So, it’s a bit of control. You lose control of your messaging and everything; if marketing gets their hands in this, then it becomes ugly.On the other hand, you have to, as a vendor, always realize that these are people frequently trying to make your products better. Some may be out just to extort you a little bit, whatever. That’s life. Get used to it. And in the end, it’s about putting the customers first, not necessarily putting your ego first and your marketing first.Corey: Changing gears slightly because believe it or not, neither you nor I have our primary day jobs focused on, you know, journalism or analyst work or anything like that these days, we focus on these—basically cloud, for lack of a better term—through slightly different lenses. I look at it through cost—which is of course architecture—and you look at it through the lens of security. And I will point out that only one of us gets called at three in the morning when things get horrible because of the bill is a strictly business-hours problem. Don’t think that’s an accident as far as what I decided to focus on. What do you do these days?Rich: You mean, what do I do in my day-to-day job?Corey: Well, it feels like a fair question to ask. Like, what do you do as far as day job, personal life et cetera. Who is Rich Mogull? You’ve been a name on the internet for a long time; I figured we’d add some color and context to it.Rich: Well, let’s see. I just got back from a flying lesson. I’m honing in on my getting ready for my first solo. My side gig is as a disaster response paramedic. I dressed up as a stormtrooper for the 501st Legion. I’ve got a few kids and then I have a job. I technically have two jobs. So—Corey: I’m envious of some of those things. I was looking into getting into flying but that path’s not open to me, given that I have ADHD. And there are ways around it in different ways. It’s like no, no, you don’t understand. With my given expression of it, I am exactly the kind of person that should not be flying a plane, let’s be very clear here. This is not a regulatory thing so much as it is a, “I’m choosing life.”Rich: Yeah. It’s a really fascinating thing because it’s this combination of a physical and a mental challenge. And I’m still very early in the process. But you know, I cracked 50, it had always been a life goal to do this, and I said, “You know what? I’m going to go do it.”So, first thing, I get my medical to make sure I can actually pass that because I’m over 50, and then from there, I can kind of jump into lessons. Protip though: don’t start taking lessons right as summer is kicking in in Phoenix, Arizona, with winds and heat that messes up your density altitude, and all sorts of fun things like that because it’s making it a little more challenging. But I’m glad I’m doing it.Corey: I have to imagine. That’s got to be an interesting skill set that probably doesn’t have a huge amount of overlap with the ins and outs of the cloud business. But maybe I’m wrong.Rich: Oh God, Corey. The correlations between information security—my specialty, and cloud security as a subset of that—aviation, and emergency medicine are incredible. These are three areas with very similar skill sets required in terms of thought processes. And in the case of both the paramedic and aviation, there’s physical skills and mental skills at the same time. But how you look at incidents, how you process things algorithmically, how you—your response times, checklists, the correlations.And I’ve been talking about two of those three things for years. I did a talk a couple years ago, during Covid, my Blackhat talk on the “Paramedics Guide to Surviving Cybersecurity,” where I talked a lot about these kinds of pieces. And now aviation is becoming another part of that. Amazing parallels between all three. Very similar mindsets are required.Corey: When you take a look at the overall sweep of the industry, you’ve been involved in cloud for a fairly long time. I have, too, but I start off as a cynic. I started originally when I got into the space, 2006, 2007, thinking virtualization was a flash in the pan because of the security potential impact of this. Then cloud was really starting to be a thing and pfff, that’s not likely to take off. I mean, who’s going to trust someone else to run all of their computing stuff?And at this point, I’ve learned to stop trying to predict the future because I generally get it 180 degrees wrong, which you know, I can own that. But I’m curious what you saw back when you got into this that made you decide, yeah, cloud has legs. What was that?Rich: I was giving a presentation with this guy, Chris Hoff, a good friend of mine. And Chris and I joined together are individual kind of research threads and were talking about, kind of, “Disruptive Innovation and the Future of Security.” I think that was the title. And we get that at RSA, we gave that at SOURCE Boston, start kind of doing a few sessions on this, and we talked about grid computing.And we were looking at, kind of, the economics of where things were going. And very early, we also realized that on the SaaS side, everybody was already using cloud; they just didn’t necessarily know it and they called them Application Service Providers. And then the concepts of cloud in the very early days were becoming compelling. It really hit me the first time I used it.And to give you perspective, I’d spent years, you know, seven years as a Gartner analyst getting hammered with vendors all the time. You can’t really test those technologies out because you can never test them in a way that an enterprise would use them. Even if I had a lab, the lab would be garbage; and we know this. I don’t trust things coming out of labs because that does not reflect operational realities at enterprise scale. Coming out of Gartner, they train me to be an enterprise guy. You talk about a large company being 200? Large companies start at 3000 to 5000 employees.Corey: Does that map to cloud services the way that AWS expresses? Because EKS, you’re going to manage that differently in an enterprise environment—or any other random AWS service; I’m just picking EKS as an example on this. But I can spin up a cluster and see what it’s like in 15 minutes, you know, assuming the cluster gets with the program. And it’s the same type of thing I would use in an enterprise, but I’m also not experiencing it in the enterprise-like way with the processes and the gating and the large team et cetera, et cetera, et cetera. Do you think it’s still a fair comparison at that point?Rich: Yeah, I think it absolutely is. And this is what really blew my mind. 11 or 12 years ago, when I got my first cloud account setup. I realized, oh, my God. And that was, there was no VPC, there was no IAM. It was ephemeral—and—no, we just had EBS was relatively new, and IAM was API only, it wasn’t in the console yet.Corey: And the network latency was, we’ll charitably call it non-deterministic.Rich: That was the advantage of not running anything at scale, wasn’t an issue at the time. But getting the hands-on and being able to build what I could build so quickly and easily and with so little friction, that was mind-blowing. And then for me, the first time I’ve used security groups I’m like, “Oh, my God, I have the granularity of a host firewall with the manageability of a network firewall?” And then years later, getting much deeper into how AWS networking and all the other pieces were—Corey: And doesn’t let it hit the host, which I always thought a firewall that lets—Rich: Yes.Corey: —traffic touch the host is like a seatbelt that lets your face touch the dashboard.Rich: Yeah. The first thing they do, they go in, they’re going to change the rules. But you can’t do that. It’s those layers of defense. And then I’m finding companies in the early days who wanted to put virtual appliances in front of everything. And still do. I had calls last week about that.But those are the things that really changed my mind because all of a sudden, this was what the key was, that I didn’t fully realize—and it’s kind of something that’s evolved into something I call the ‘Grand Unified Theory of Cloud Governance,’ these days—but what I realized was those barriers are gone. And there is no way to stop this as people want to build and test and deploy applications because the benefits are going to be too strong. So, grab onto the reins, hold on to the back of the horse, you’re going to get dragged away, and it’s your choice if your arm gets ripped off in the process or if you’re going to be able to ride that thing and at least steer it in the general direction that you need it to go in.Corey: One of the things that really struck me when I started playing around with cloud for more than ten minutes was everything you say is true, but I can also get started today to test out an idea. And most of them don’t work, but if something hits, suddenly I don’t have the data center constraints, whereas today, I guess you’d call it, I built my experiment MVP on top of a Raspberry Pi and now I have to wait six weeks for Dell to send me something that isn’t a piece of crap that I can actually take production traffic on. There’s no okay, and I’ll throw out the junky hardware and get the good stuff in once you start hitting a point of scale because you’re already building on that stuff without the corresponding massive investment of capital to get there.Rich: Yeah well, I mean, look, I lived this, I did a startup that was based on demos at a Blackhat—sorry, at a Blackhat. Blackhat. Did some demos on stage, people were like, “We want your code.” It was about cloud security automation. That led to doing your startup, the thing called DisruptOps, which got acquired, and that’s how I ended up at FireMon. So, that’s the day job route where I ended up.And what was amazing for that is, to add on to what you said, first of all, the friction was low; once we get the architecture right, scalability is not something we are hugely concerned with, especially because we’re CI/CD. Oh, no, we hit limits. Boom, let’s just stand up a new version and redirect people over there. Problem solved. And then the ability to, say, run multiple versions of our platform simultaneously? We’re doing that right now. We just had to release an entirely free version of it.To do that. It required back-end architectural changes for cost, not for scalability so much, but for a lot around cost and scheduling because our thing was event-driven, we’re able to run that and run our other platform fully in parallel, all shared data structures, shared messaging structures. I can’t even imagine how hard that would have not been to do in a traditional data center. So, we have a lot of freedom, still have those cost constraints because that’s [laugh] your thing, but the experimentation, the ability to integrate things, it’s just oh, my God, it’s just exciting.Corey: And let’s be clear, I, having spent a lot of time as a rat myself in these data centers, I don’t regret handing a lot of that responsibility off, just because, let’s not kid ourselves, they are better at replacing failed or failing hardware than I will ever be. That’s part of the benefit you get from the law of large numbers.Rich: Yeah. I don’t want to do all of that stuff, but we’re hovering around something that is kind of—all right, so former Gartner analyst means I have a massive ego, and because of that, I like to come up with my own terms for things, so roll with me here. And it’s something I’m calling the ‘Grand Unified Theory of Cloud Governance’ because you cannot possibly get more egotistical than referring to something as your solution to the biggest problem in all of physics. The idea is, is that cloud, as we have just been discussing, it drops friction and it decentralizes because you don’t have to go ask somebody for the network, you don’t have to ask somebody for the server. So, all of a sudden, you can build a full application stack without having to call somebody for help. We’ve just never had that in IT before.And all of our governance structures—and this includes your own costs, as well as security—are built around scarcity. Scarcity of resources, natural choke points that evolved from the data center. Not because it was bad. It wasn’t bad. We built these things because that’s what we needed for that environment at the data center.Now, we’ve got cloud and it’s this whole new alien technology and it decentralizes. That said, particularly for us on security, you can build your whole application stack, of course, we have completely unified the management interfaces in one place and then we stuck them on the internet, protected with nothing more than a username and password. And if you can put those three things together in your head, you can realize why these are such dramatic changes and so challenging for enterprises, why my kids get to go to Disney a fair bit because we’re in demand as security professionals.Corey: What does FireMon do exactly? That’s something that I’m not entirely up to speed on, just because please don’t take this the wrong way, but I was at RSA this year, and it feels like all the companies sort of blend together as you walk between the different booths. Like, “This is what you should be terrified of today.” And it always turns into a weird sales pitch. Not that that’s what you do, but it at some point just blinds me and overloads me as far as dealing with any of the cloud security space.Rich: Oh, I’ve been going to RSA for 20 years. One of our SEs, I was briefly at our booth—I’m usually in outside meetings—and he goes, “Do you see any fun and interesting?” I go—I just looked at him like I was depressed and I’m like, “I’ve been to RSA for 20 years. I will never see anything interesting here again. Those days are over.” There’s just too much noise and cacophony on that show floor.What do we do? So—Corey: It makes re:Invent’s Expo Hall look small.Rich: Yeah. I mean, it’s, it’s the show over at RSA. And it wasn’t always. I mean, it was—it’s always been big as long as I’ve been there, but yeah, it’s huge, everyone is there, and they’re all saying exactly the same thing. This year, I think the only reason it wasn’t all about AI is because they couldn’t get the printers to reprint the banners fast enough. Not that anybody has any products that would do anything there. So—you look like you want to say something there.Corey: No, no. I like the approach quite a bit. It’s the, everything was about AI this year. It was a hard pivot from trying to sell me a firewall, which it seems like everyone was doing in the previous year. It’s kind of wild. I keep saying that there’s about a dozen companies that exhibit at RSA. A guess, there are hundreds and hundreds of booths, but it all distills down to the same 12 things. They have different logos and different marketing stories, but it does seem like a lot of stuff is very much just like the booth next to it on both sides.Rich: Yeah. I mean, that’s—it’s just the nature. And part of—there’s a lot of reasons for this. We used to, when I was—so prior to doing the startup thing and then ending up at FireMon, I did Securosis, which was an analyst firm, and we used to do the Securosis guide to RSA every year where we would try and pick the big themes. And the reality is, there’s a reason for that.I wrote something once the vendors lied to you because you want them to. It’s the most dysfunctional relationship because as customers, you’re always asking, “Well, what are you doing for [unintelligible 00:22:16]? What are you doing for zero trust? What are you doing for AI?” When those same customers are still just working on fundamental patch management and firewall management. But it doesn’t stop them from asking the questions and the vendors have to have answers because that’s just the nature of that part of the world.Corey: I will ask you, over are past 12 years—I have my own thoughts on this, but I want to hear your take on it—what’s changed in the world of cloud security?Rich: Everything. I mean, I was one of the first to be doing this.Corey: Oh, is that all?Rich: Yeah. So, there’s more people. When I first started, very few people doing it, nobody knew much about it outside AWS, we all knew each other. Now, we’ve got a community that’s developed and there’s people that know what they’re doing. There’s still a shortage of skills, absolutely still a shortage of skills, but we’re getting a handle on that, you know? We’re getting a bit of a pipeline.And I’d say that’s still probably the biggest challenge faced. But what’s improved? Well, it’s a give-and-take. On one hand, we now have strategies, we have tools that are more helpful, unfortunately—I’ll tell you the biggest mistake I made and it ties to the FireMon stuff in my career, in a minute; relates directly to this question, but we’re kind of getting there on some of the tool pieces.On the other hand, that complexity is increasing faster. And that’s what’s made it hard. So, as much as we’re getting more skilled people, better at tooling, for example, we kind of know—and we didn’t have CloudTrail when I started. We didn’t have the fundamental things you need to actually implement security at the start of cloud. Most of those are there; they may not be working the way we wish they always worked, but we’ve got the pieces to assemble it, depending on which platform you’re on. That’s probably the biggest change. Now, we need to get into the maturity phase of cloud, and that’s going to be much more difficult and time-consuming to kind of get over that hump.Corey: It’s easy to wind up saying, “Oh, I saw the future so clearly back then,” but I have to ask, going back 12 years, the path the world would take was far from certain. Did you have doubts?Rich: Like, I had presented with Chris Hoff. We—we’re still friends—presented stuff together, and he got a job that was kind of clouding ancillary. And I remember calling him up once and going, “Chris, I don’t know what to do.” I was running my little analyst firm—little. We were doing very, very well—I could not get paid to do any work around cloud.People wanted me to write shitty papers on DLP and take customer inquiries on DLP because I had covered that at the Gartner days, and data encryption and those pieces. That was hard. And fortunately, a few things started trickling in. And then it was a flood. It completely changed our business and led to me, you know, eventually going down into the vendor path. But that was a tough day when I hit that point. So, absolutely I knew it was the future. I didn’t know if I was going to be able to make a living at it.Corey: It would seem that you did.Rich: Yeah. Worked out pretty well [laugh].Corey: You seem sprightly to me. Good work. You’re not on death’s door.Rich: No. You know, in fact, the analyst side of it exploded over the years because it turns out, there weren’t people who had this experience. So, I could write code to the APIs, but they’ll still talk with CEOs and boards of directors around these cloud security issues and frame them in ways that made sense to them. So, that was wonderful. We partnered up with the Cloud Security Alliance, I actually built a bunch of the CSA training, I wrote the current version of the CSA guidance, we’re writing the next version of that, did a lot of research with them. They’ve been a wonderful partner.So, all that went well. Then I got diverted down onto the vendor path. I had this research idea and then it came out, we ended up founding that as a startup and then it got, as I mentioned, acquired by FireMon, which is interesting because FireMon, you asked what we did, it’s firewall policy management is the core of the company. Yet the investors realize the company was not going in the right direction necessarily, to deal with the future of cloud. They went to their former CEO and said, “Hey, can you come back”—the founder of the company—“And take this over and start moving us in the right direction?”Well, he happened to be my co-founder at the startup. And so, we kind of came in and took over there. And so, now it’s a very interesting position because we have this one cloud-native thing we built for all these years. We made one mistake with that, which I’ll talk about which ties back to your predicting the future piece if you want to go into it, but then we have the network firewall piece now extending into hybrid, and we have an asset management moving into the attack surface management space as well. And both of those products have been around for, like, 15-plus years.Corey: No, I’m curious to your thoughts on it because it’s been one of those weird areas where there’s been so much change and so much evolution, but you also look at today’s “OWASP Top 10” list of vulnerabilities, and yeah, they updated a year or so ago, but it still looks basically like things that—from 2008—would have made sense to me when I’m looking at this. Well, insomuch as they do now. I didn’t know then, nor do I now what a cross-site scripting attack might be, but other than that, I find that there’s, “Oh, you misconfigured something and it winds up causing a problem.” Well, no kidding. Imagine that.Rich: Yeah. Look, the fundamentals don’t change, but it’s still really easy to screw up.Corey: Oh, having done so a lot, I believe you.Rich: There’s a couple of principles, and I’ll break it into two sides. One is, a lot of security sounds simple. There’s nothing simple at scale. Nothing simple scales. The moment you get up to even 200 employees, everything just becomes ridiculously harder. That’s the nature of reality. Simplicity doesn’t scale.The other part is even though it’s always the same, it’s still easy to think you’re going to be different this time and you’re not going to screw it up, and then you do. For example, so cloud, we were talking about the maturity. I assumed CSPM just wasn’t going to be a thing. For real. The Cloud Security Posture Management. Because why would the cloud providers not just make that problem go away and then all the vulnerability assessment vendors and everybody else? It seemed like it was an uninteresting problem.And yet, we were building a cloud security automation thing and we missed the boat because we had everything we needed to be one of the very first CSPM vendors on the market and we’re like, “No, no. That problem is going to go away. We’ll go there.” And it ties back to what you said, which is it’s the same stuff and we just outsmarted ourselves. We thought that people would go further faster. And they don’t and they aren’t.And that’s kind of where we are today. We are dramatically maturing. At the same time, the complexity is increasing dramatically. It’s just a huge challenge for skills and staffing to adjust governance programs. Like I think we’ve got another 10 to 20 years to go on this cloud security thing before we even get close. And then maybe we’ll get down to the being bored by the problems. But probably not because AI will ruin us.Corey: I’d like to imagine, on some level, that AI could be that good. I mean, don’t get me wrong. It has value and it is transformative for a bunch of things, but I also think a lot of the fear-mongering is more than a little overblown.Rich: No, I agree with you. I’m trying to keep a very close eye on it because—I can’t remember if you and I talked about this when we met face-to-face, or… it was somebody at that event—AI is just not just AI. There’s different. There’s the LLMs, there’s the different kinds of technologies that are involved. I mean, we use AI all over the place already.I mean my phone’s got it built in to take better pictures. It’s a matter of figuring out what the use cases and the, honestly, some of the regulatory structure around it in terms of copyright and everything else. I’m not worried about Clippy turning into Skynet, even though I might make jokes about that on Mastodon, maybe someday there will be some challenges, but no, it’s just going to be another tech that we’re going to figure out over time. It is disruptive, so we can’t ignore that part of it.Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where’s the best place to find you that isn’t one of the Disney parks?Rich: That really is kind of the best place to find—no. So, these days, I do technically still have a Twitter presence at @rmogull. I’m not on there much, but I will get DMs if people send those over. I’m more on Mastodon. It’s at @rmogull defcon.social. I write over at FireMon these days, as well as occasionally still over Securosis, on those blogs. And I’m in the [Cloud Security Slack community 00:30:49] that is now under the banner for CloudSec. That’s probably the best place if you want to hit me up and get quick answers on anything.Corey: And I will, of course, include links to all of that in the show notes. Thank you so much for taking the time to speak with me today. I really appreciate it.Rich: Thanks, Corey. I was so happy to be here.Corey: Rich Mogull, SVP of Cloud Security at FireMon. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment talking about how at Dell these days, it does not take six weeks to ship a server. And then I will get back to you in six to eight weeks.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Kelly Shortridge, Senior Principal Engineer at Fastly, joins Corey on Screaming in the Cloud to discuss their recently released book, Security Chaos Engineering: Sustaining Resilience in Software and Systems. Kelly explains why a resilient strategy is far preferable to a bubble-wrapped approach to cybersecurity, and how developer teams can use evidence to mitigate security threats. Corey and Kelly discuss how the risks of working with complex systems is perfectly illustrated by Jurassic Park, and Kelly also highlights why it’s critical to address both system vulnerabilities and human vulnerabilities in your development environment rather than pointing fingers when something goes wrong.About KellyKelly Shortridge is a senior principal engineer at Fastly in the office of the CTO and lead author of "Security Chaos Engineering: Sustaining Resilience in Software and Systems" (O'Reilly Media). Shortridge is best known for their work on resilience in complex software systems, the application of behavioral economics to cybersecurity, and bringing security out of the dark ages. Shortridge has been a successful enterprise product leader as well as a startup founder (with an exit to CrowdStrike) and investment banker. Shortridge frequently advises Fortune 500s, investors, startups, and federal agencies and has spoken at major technology conferences internationally, including Black Hat USA, O'Reilly Velocity Conference, and SREcon. Shortridge's research has been featured in ACM, IEEE, and USENIX, spanning behavioral science in cybersecurity, deception strategies, and the ROI of software resilience. They also serve on the editorial board of ACM Queue.Links Referenced: Fastly: https://www.fastly.com/ Personal website: https://kellyshortridge.com Book website: https://securitychaoseng.com LinkedIn: https://www.linkedin.com/in/kellyshortridge/ Twitter: https://twitter.com/swagitda_ Bluesky: https://shortridge.bsky.social TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Have you listened to the new season of Traceroute yet? Traceroute is a tech podcast that peels back the layers of the stack to tell the real, human stories about how the inner workings of our digital world affect our lives in ways you may have never thought of before. Listen and follow Traceroute on your favorite platform, or learn more about Traceroute at origins.dev. My thanks to them for sponsoring this ridiculous podcast. Corey: Welcome to Screaming in the Cloud, I’m Corey Quinn. My guest today is Kelly Shortridge, who is a Senior Principal Engineer over at Fastly, as well as the lead author of the recently released Security Chaos Engineering: Sustaining Resilience in Software and Systems. Kelly, welcome to the show.Kelly: Thank you so much for having me.Corey: So, I want to start with the honest truth that in that title, I think I know what some of the words mean, but when you put them together in that particular order, I want to make sure we’re talking about the same thing. Can you explain that like I’m five, as far as what your book is about?Kelly: Yes. I’ll actually start with an analogy I make in the book, which is, imagine you were trying to rollerblade to some destination. Now, one thing you could do is wrap yourself in a bunch of bubble wrap and become the bubble person, and you can waddle down the street trying to make it to your destination on the rollerblades, but if there’s a gust of wind or a dog barks or something, you’re going to flop over, you’re not going to recover. However, if you instead do what everybody does, which is you know, kneepads and other things that keep you flexible and nimble, the gust you know, there’s a gust of wind, you can kind of be agile, navigate around it; if a dog barks, you just roller-skate around it; you can reach your destination. The former, the bubble person, that’s a lot of our cybersecurity today. It’s just keeping us very rigid, right? And then the alternative is resilience, which is the ability to recover from failure and adapt to evolving conditions.Corey: I feel like I am about to torture your analogy to death because back when I was in school in 2000, there was an annual tradition at the school I was attending before failing out, where a bunch of us would paint ourselves green every year and then bike around the campus naked. It was the green bike ride. So, one year I did this on rollerblades. So, if you wind up looking—there’s the bubble wrap, there’s the safety gear, and then there’s wearing absolutely nothing, which feels—Kelly: [laugh]. Yes.Corey: —kind of like the startup approach to InfoSec. It’s like, “It’ll be fine. What’s the worst that happens?” And you’re super nimble, super flexible, until suddenly, oops, now I really wish I’d done things differently.Kelly: Well, there’s a reason why I don’t say rollerblade naked, which other than it being rather visceral, what you described is what I’ve called YOLOSec before, which is not what you want to do. Because the problem when you think about it from a resilience perspective, again, is you want to be able to recover from failure and adapt. Sure, you can oftentimes move quickly, but you’re probably going to erode software quality over time, so to a certain point, there’s going to be some big incident, and suddenly, you aren’t fast anymore, you’re actually pretty slow. So, there’s this, kind of, happy medium where you have enough, I would like security by design—we can talk about that a bit if you want—where you have enough of the security by design baked in and you can think of it as guardrails that you’re able to withstand and recover from any failure. But yeah, going naked, that’s a recipe for not being able to rollerblade, like, ever again, potentially [laugh].Corey: I think, on some level, that the correct dialing in of security posture is going to come down to context, in almost every case. I’m building something in my spare time in the off hours does not need the same security posture—mostly—as we are a bank. It feels like there’s a very wide gulf between those two extremes. Unfortunately, I find that there’s a certain tone-deafness coming from a lot of the security industry around oh, everyone must have security as their number one thing, ever. I mean, with my clients who I fixed their AWS bills, I have to care about security contractually, but the secrets that I hold are boring: how much money certain companies pay another very large company.Yes, I’ll get sued into oblivion if that leaks, but nobody dies. Nobody is having their money stolen as a result. It’s slightly embarrassing in the tech press for a cycle and then it’s over and done with. That’s not the same thing as a brief stint I did running tech ops at Grindr ten years ago where, leak that database and people will die. There’s a strong difference between those threat models, and on some level, being able to act accordingly has been one of the more eye-opening approaches to increasing velocity in my experience. Does that align with the thesis of your book, since my copy has not yet arrived for this recording?Kelly: Yes. The book, I am not afraid to say it depends on the book, and you’re right, it depends on context. I actually talk about this resilience potion recipe that you can check out if you want, these ingredients so we can sustain resilience. A key one is defining your critical functions, just what is your system’s reason for existence, and that is what you want to make sure it can recover and still operate under adverse conditions, like you said.Another example I give all the time is most SaaS apps have some sort of reporting functionality. Guess what? That’s not mission-critical. You don’t need the utmost security on that, for the most part. But if it’s processing transactions, yeah, probably you want to invest more security there. So yes, I couldn’t agree more that it’s context-dependent and oh, my God, does the security industry ignore that so much of the time, and it’s been my gripe for, I feel like as long as I’ve been in the industry.Corey: I mean, there was a great talk that Netflix gave years ago where they mentioned in passing, that all developers have root in production. And that’s awesome and the person next to him was super excited and I looked at their badge, and holy hell, they worked at an actual bank. That seems like a bad plan. But talking to the Netflix speaker after the fact, Dave Hahn, something that I found that was extraordinarily insightful, was that, yeah, well we just isolate off the PCI environment so the rest and sensitive data lives in its own compartmentalized area. So, at that point, yeah, you’re not going to be able to break much in that scenario.It’s like, that would have been helpful context to put in talk. Which I’m sure he did, but my attention span had tripped out and I missed that. But that’s, on some level, constraining blast radius and not having compliance and regulatory issues extending to every corner of your environment really frees you up to do things appropriately. But there are some things where you do need to care about this stuff, regardless of how small the surface area is.Kelly: Agreed. And I introduced the concept of the effort investment portfolio in the book, which is basically, that is where does it matter to invest effort and where can you kind of like, maybe save some resources up. I think one thing you touched on, though, is, we’re really talking about isolation and I actually think people don’t think about isolation in as detailed or maybe as expansively as they could. Because we want both temporal and logical and spatial isolation. What you talked about is, yeah, there are some cases where you want to isolate data, you want to isolate certain subsystems, and that could be containers, it could also be AWS security groups.It could take a bunch of different forms, it could be something like RLBox in WebAssembly land. But I think that’s something that I really try to highlight in the book is, there’s actually a huge opportunity for security engineers starting from the design of a system to really think about how can we infuse different forms of isolation to sustain resilience.Corey: It’s interesting that you use the word investment. When fixing AWS bills for a living, I’ve learned over the last almost seven years now of doing this that cost and architecture and cloud are fundamentally the same thing. And resilience is something that comes with a very real cost, particularly when you start looking at what the architectural choices are. And one of the big reasons that I only ever work on a fixed-fee basis is because if I’m charging for a percentage of savings or something, it inspires me to say really uncomfortable things like, “Backups are for cowards.” And, “When was the last time you saw an entire AWS availability zone go down for so long that it mattered? You don’t need to worry about that.” And it does cut off an awful lot of cost issues, at the price of making the environment more fragile.That’s where one of the context thing starts to come in. I mean, in many cases, if AWS is having a bad day in a given region, well does your business need that workload to be functional? For my newsletter, I have a publication system that’s single-homed out of the Oregon region. If that whole thing goes down for multiple days, I’m writing that week’s issue by hand because I’m going to have something different to talk about anyway. For me, there is no value in making that investment. But for companies, there absolutely is, but there’s also seems to be a lack of awareness around, how much is a reasonable investment in that area when do you start making that investment? And most critically, when do you stop?Kelly: I think that’s a good point, and luckily, what’s on my side is the fact that there’s a lot of just profligate spending in cybersecurity and [laugh] that’s really what I’m focused on is, how can we spend those investments better? And I actually think there’s an opportunity in many cases to ditch a ton of cybersecurity tools and focus more on some of the stuff he talked about. I agree, by the way that I’ve seen some threat models where it’s like, well, AWS, all regions go down. I’m like, at that point, we have, like, a severe, bigger-than-whatever-you’re-thinking-about problem, right?Corey: Right. So, does your business continuity plan account for every one of your staff suddenly quitting on the spot because there’s a whole bunch of companies with very expensive consulting, like, problems that I’m going to go work for a week and then buy a house in cash. It’s one of those areas where, yeah, people are not going to care about your environment more than they are about their families and other things that are going on. Plan accordingly. People tend to get so carried away with these things with tabletop planning exercises. And then of course, they forget little things like I overwrote the database by dropping the wrong thing. Turns out that was production. [laugh]. Remembering for [a me 00:10:00] there.Kelly: Precisely. And a lot of the chaos experiments that I talk about in the book are a lot of those, like, let’s validate some of those basics, right? That’s actually some of the best investments you can make. Like, if you do have backups, I can totally see your argument about backups are for cowards, but if you do have them, like, maybe you conduct experiments to make sure that they’re available when you need them, and the same thing, even on the [unintelligible 00:10:21] side—Corey: No one cares about backups, but everyone really cares about restores, suddenly, right after—Kelly: Yeah.Corey: —they really should have cared about backups.Kelly: Exactly. So, I think it’s looking at those experiments where it’s like, okay, you have these basic assumptions in place that you assume to be invariance or assume that they’re going to bail you out if something goes wrong. Let’s just verify. That’s a great place to start because I can tell you—I know you’ve been to the RSA hall floor—how many cybersecurity teams are actually assessing the efficacy and actually experimenting to see if those tools really help them during incidents. It’s pretty few.Corey: Oh, vendors do not want to do those analyses. They don’t want you to do those analyses, either, and if you do, for God’s sakes, shut up about it. They’re trying to sell things here, mostly firewalls.Kelly: Yeah, cybersecurity vendors aren’t necessarily happy about my book and what I talk about because I have almost this ruthless focus on evidence and [unintelligible 00:11:08] cybersecurity vendors kind of thrive on a lack of evidence. So.Corey: There’s so much fear, uncertainty, and doubt in that space and I do feel for them. It’s a hard market to sell in without having to talk about here’s the thing that you’re defending against. In my case, it’s easy to sell the AWS bill is high because if I don’t have to explain why more or less setting money on fire as a bad thing, I don’t really know what to tell you. I’m going to go look for a slightly different customer profile. That’s not really how it works in security, I’m sure there are better go-to-market approaches, but they’re hard to find, at least, ones that work holistically.Kelly: There are. And one of my priorities with the book was to really enumerate how many opportunities there are to take software engineering practices that people already know, let’s say something like type systems even, and how those can actually help sustain resilience. Even things like integration testing or infrastructure as code, there are a lot of opportunities just to extend what we already do for systems reliability to sustain resilience against things that aren’t attacks and just make sure that, you know, we cover a few of those cases as well. A lot of it should be really natural to software engineering teams. Again, security vendors don’t like that because it turns out software engineering teams don’t particularly like security vendors.Corey: I hadn’t noticed that. I do wonder, though, for those who are unaware, chaos engineering started off as breaking things on purpose, which I feel like one person had a really good story and thought about it super quickly when they were about to get fired. Like, “No, no, it’s called Chaos Engineering.” Good for them. It’s now a well-regarded discipline. But I’ve always heard of it in the context of reliability of, “Oh, you think your site is going to work if the database falls over? Let’s push it over and see what happens.” How does that manifest in a security context?Kelly: So, I will clarify, I think that’s a slight misconception. It’s really about fixing things in production, and that’s the end goal. I think we should not break things just to break them, right? But I’ll give a simple example, which I know it’s based on what Aaron Rinehart conducted at UnitedHealth Group, which is, okay, let’s inject a misconfigured port as an experiment and see what happens, end-to-end. In their case, the firewall only detected the misconfigured port 60% of the time, so 60% of the time, it works every time.But it was actually the cloud, the very common, like, Cloud configuration management tool that caught the change and alerted responders. So, it’s that kind of thing where we’re still trying to verify those assumptions that we have about our systems and how they behave, again, end-to-end. In a lot of cases, again, with security tools, they are not behaving as we expect. But I still argue security is just a subset of software quality, so if we’re experimenting to verify, again, our assumptions and observe system behavior, we’re benefiting software quality, and security is just a subset of that. Think about C code, right? It’s not like there’s, like, a healthy memory corruption, so it’s bad for both the quality and security reason.Corey: One problem that I’ve had in the security space for a while is—let’s [unintelligible 00:14:05] on this to AWS for a second because that is the area in which I spend the most of my time, which probably explains a lot about my personality challenges. But the problem that I keep smacking into is if I go ahead and configure everything the way that I should according to best practices and the rest, I wind up with a firehose torrent of information in terms of CloudTrail logs, et cetera. And it’s expensive in its own right. But then to sort through it or to do a lot of things in security, there are basically two options. I can either buy a vendor’s product, which generally tends to start around $12,000 a year and goes up rapidly from there on my current $6,000 a year bill, so okay, twice as much as the infrastructure for security monitoring. Okay.Or alternately, find a bunch of different random scripts and tools on GitHub of wildly diverging quality and sort of hope for the best on that. It feels like there’s nothing in between. And the reason I care about this is not because I’m cheap but because when you have an individual learner who is either a student or a career switcher or someone just trying to experiment with this, you want them to begin as you want them to go on, and things that are no money for an enterprise are all the money to them. They’re going to learn to work with the tools that they can afford. That feels like it’s a big security swing and a miss. Do you agree or disagree? What’s the nuance I’m missing here?Kelly: No, I don’t think there’s nuance you’re missing. I think security observability, for one, isn’t a buzzword that particularly exists. I’ve been trying to make it a thing, but I’m solely one individual screaming into the void. But observability just hasn’t been a thing. We haven’t really focused on, okay, so what, like, we get data and what do we do with it?And I think, again, from a software engineering perspective, I think there’s a lot we can do. One, we can just avoid duplicating efforts. We can treat observability, again, of any sort of issue as similar, whether that’s an attack or a performance issue. I think this is another place where security, or any sort of chaos experiment, shines though because if you have an idea of here’s an adverse scenario we care about, you can actually see how does it manifest in the logs and you can start to figure out, like, what signals do we actually need to be looking for, what signals mattered to be able to narrow it down. Which again, it involves time and effort, but also, I can attest when you’re buying the security vendor tool and, in theory, absolving some of that time and effort, it’s maybe, maybe not, because it can be hard to understand what the outcomes are or what the outputs are from the tool and it can also be very difficult to tune it and to be able to explain some of the outputs. It’s kind of like trading upfront effort versus long-term overall overhead if that makes sense.Corey: It does. On that note, the title of your book includes the magic key phrase ‘sustaining resilience.’ I have found that security effort and investment tends to resemble a fire drill in—Kelly: [laugh].Corey: —an awful lot of places, where, “We care very much about security,” says the company, right after they very clearly failed to care about security, and I know this because I’m reading getting an email about a breach that they’ve just sent me. And then there’s a whole bunch of running around and hair-on-fire moments. But then there’s a new shiny that always comes up, a new strategic priority, and it falls to the wayside again. What do you see the drives that sustained effort and focus on resilience in a security context?Kelly: I think it’s really making sure you have a learning culture, which sounds very [unintelligible 00:17:30], but things again, like, experiments can help just because when you do simulate those adverse scenarios and you see how your system behaves, it’s almost like running an incident and you can use that as very fresh, kind of, like collective memory. And I even strongly recommend starting off with prior incidents in simulating those, just to see like, hey, did the improvements we make actually help? If they didn’t, that can be kind of another fire under the butt, so to speak, to continue investing. So, definitely in practice—and there’s some case studies in the book—it can be really helpful just to kind of like sustain that memory and sustain that learning and keep things feeling a bit fresh. It’s almost like prodding the nervous system a little, just so it doesn’t go back to that complacent and convenient feeling.Corey: It’s one of the hard problems because—I’m sure I’m going to get castigated for this by some of the listeners—but computers are easy, particularly compared to the people. There are deterministic ways to solve almost any computer problem, but people are always going to be a little bit different, and getting them to perform the same way today that they did yesterday is an exercise in frustration. Changing the culture, changing the approach and the attitude that people take toward a lot of these things feels, from my perspective, like, something of an impossible job. Cultural transformations are things that everyone talks about, but it’s rare to see them succeed.Kelly: Yes, and that’s actually something that I very strongly weaved throughout the book is that if your security solutions rely on human behavior, they’re going to fail. We want to either reduce hazards or eliminate hazards by design as much as possible. So, my view is very much again, like, can you make processes more repeatable? That’s going to help security. I definitely do not think that if anyone takes away from my book that they need to have, like, a thousand hours of training to change hearts and minds, then they have completely misunderstood most of the book.The idea is very much like, what are practices that we want for other outcomes anyway—again, reliability or faster time to market—and how can we harness those to also be improving resilience or security at the same time? It’s very much trying to think about those opportunities rather than, you know, trying to drill into people’s heads, like, “Thou shalt not,” or, “Thou shall.”Corey: Way back in 2018, you gave a keynote at some conference or another and you built the entire thing on the story of Jurassic Park, specifically Ian Malcolm as one of your favorite fictional heroes, and you tied it into security in a bunch of different ways. You hadn’t written this book then unless the authorship process is way longer than I think it is. So, I’m curious to get your take on what Jurassic Park can teach us about software security.Kelly: Yes, so I talk about Jurassic Park as a reference throughout the book, frequently. I’ve loved that book since I was a very young child. Jurassic Park is a great example of a complex system gone wrong because you can’t point to any one thing. Like there’s Dennis Nedry, you know, messing up the power system, but then there’s also the software was looking for a very specific count of dinosaurs and they didn’t anticipate there could be more in the count. Like, there are so many different factors that influenced it, you can’t actually blame just, like, human error or point fingers at one thing.That’s a beautiful example of how things go wrong in our software systems because like you said, there’s this human element and then there’s also how the humans interact and how the software components interact. But with Jurassic Park, too, I think the great thing is dinosaurs are going to do dinosaur things like eating people, and there are also equivalents in software, like C code. C code is going to do C code things, right? It’s not a memory safe language, so we shouldn’t be surprised when something goes wrong. We need to prepare accordingly.Corey: “How could this happen? Again?” Yeah.Kelly: Right. At a certain point, it’s like, there’s probably no way to sufficiently introduce isolation for dinosaurs unless you put them in a bunker where no one can see them, and it’s the same thing sometimes with things like C code. There’s just no amount of effort you can invest, and you’re just kind of investing for a really unclear and generally not fortuitous outcome. So, I like it as kind of this analogy to think about, okay, where do our effort investments make sense and where is it sometimes like, we really just do need to refactor because we’re dealing with dinosaurs here.Corey: When I was a kid, that was one of my favorite books, too. The problem is, I didn’t realize I was getting a glimpse of my future at a number of crappy startups that I worked at. Because you have John Hammond, who was the owner of the park talking constantly about how, “We spared no expense,” but then you look at what actually happened and he spared every frickin expense. You have one IT person who is so criminally underpaid that smuggling dinosaur embryos off the island becomes a viable strategy for this. He wound up, “Oh, we couldn’t find the right DNA, so we’re just going to, like, splice some other random stuff in there. It’ll be fine.”Then you have the massive overconfidence because it sounds very much like he had this almost Muskian desire to fire anyone who disagreed with him, and yeah, there was a certain lack of investment that could have been made, despite loud protestations to the contrary. I’d say that he is the root cause, he is the proximate reason for the entire failure of the park. But I’m willing to entertain disagreement on that point.Kelly: I think there are other individuals, like Dr. Wu, if you recall, like, deciding to do the frog DNA and not thinking that maybe something could go wrong. I think there was a lot of overconfidence, which you’re right, we do see a lot in software. So, I think that’s actually another very important lesson is that incentives matter and incentives are very hard to change, kind of like what you talked about earlier. It doesn’t mean that we shouldn’t include incentives in our threat model.So like, in the book I talked about, our threat models should include things like maybe yeah, people are underpaid or there is a ton of pressure to deliver things quickly or, you know, do things as cheaply as possible. That should be just as much of our threat models as all of the technical stuff too.Corey: I think that there’s a lot that was in that movie that was flat-out wrong. For example, one of the kids—I forget her name; it’s been a long time—was logging in and said, “Oh, this is Unix. I know Unix.” And having learned Unix as my first basically professional operating system, “No, you don’t. No one knows Unix. They get very confused at some point, the question is, just how far down what rabbit hole it is.”I feel so sorry for that kid. I hope she wound up seeking therapy when she was older to realize that, no, you don’t actually know Unix. It’s not that you’re bad at computers, it’s that Unix is user-hostile, actively so. Like, the raptors, like, that’s the better metaphor when everything winds up shaking out.Kelly: Yeah. I don’t disagree with that. The movie definitely takes many liberties. I think what’s interesting, though, is that Michael Creighton, specifically, when he talks about writing the book—I don’t know how many people know this—dinosaurs were just a mechanism. He knew people would want to read it in airport.What he cared about was communicating really the danger of complex systems and how if you don’t respect them and respect that interactivity and that it can baffle and surprise us, like, things will go wrong. So, I actually find it kind of beautiful in a way that the dinosaurs were almost like an afterthought. What he really cared about was exactly what we deal with all the time in software, is when things go wrong with complexity.Corey: Like one of his other books, Airframe, talked about an air disaster. There’s a bunch of contributing factors in the rest, and for some reason, that did not receive the wild acclaim that Jurassic Park did to become a cultural phenomenon that we’re still talking about, what, 30 years later.Kelly: Right. Dinosaurs are very compelling.Corey: They really are. I have to ask though—this is the joy of having a kid who is almost six—what is your favorite dinosaur? Not a question most people get asked very often, but I am going to trot that one out.Kelly: No. Oh, that is such a good question. Maybe a Deinonychus.Corey: Oh, because they get so angry they spit and kill people? That’s amazing.Kelly: Yeah. And I like that, kind of like, nimble, smarter one, and also the fact that most of the smaller ones allegedly had feathers, which I just love this idea of, like, feather-ful murder machines. I have the classic, like, nerd kid syndrome, though, where I read all these dinosaur names as a kid and I’ve never pronounced them out loud. So, I’m sure there are others—Corey: Yep.Kelly: —that I would just word salad. But honestly, it’s hard to go wrong with choosing a favorite dinosaur.Corey: Oh, yeah. I’m sure some paleontologist is sitting out there in the field on the dig somewhere listening to this podcast, just getting very angry at our pronunciation and things. But for God’s sake, I call the database Postgres-squeal. Get in line. There’s a lot of that out there where looking at a complex system failures and different contributing factors and the rest makes stuff—that’s what makes things interesting.I think that there’s this the idea of a root cause is almost always incorrect. It’s not, “Okay, who tripped over the buried landmine,” is not the interesting question. It’s, “Who buried the thing?” What were all the things that wound up contributing to this? And you can’t even frame it that way in the blaming context, just because you start doing that and people clam up, and good luck figuring out what really happened.Kelly: Exactly. That’s so much of what the cybersecurity industry is focused on is how do we assign blame? And it’s, you know, the marketing person clicked on a link. And it’s like, they do that thousands of times, like a month, and the one time, suddenly, they were stupid for doing it? That doesn’t sound right.So, I’m a big fan of, yes, vanquishing root cause, thinking about contributing factors, and in particular, in any sort of incident review, you have to think about, was there a designer process problem? You can’t just think about the human behavior; you have to think about where are the opportunities for us to design things better, to make this secure way more of the default way.Corey: When you talk about resilience and reliability and big, notable outages, most forward-thinking companies are going to go and do a variety of incident reviews and disclosures around everything that happened to it, depending upon levels of trust and whether your NDA’ed or not, and how much gets public is going to vary from place to place. But from a security perspective, that feels like the sort of thing that companies will clam up about and never say a word.Kelly: Yes.Corey: Because I can wind up pouring a couple of drinks into people and get the real story of outages, or the AWS bill, but security stuff, they start to wonder if I’m a state actor, on some level. When you were building all of this, how did you wind up getting people to talk candidly and forthrightly about issues that if it became tied to them that they were talking to this in public would almost certainly have negative career impact for them?Kelly: Yes, so that’s almost like a trade secret, I feel like. A lot of it is yes, over the years talking with people over, generally at a conference where you know, things are tipsy. I never want to betray confidentiality, to be clear, but certainly pattern-matching across people’s stories.Corey: Yeah, we’re both in positions where if even the hint of they can’t be trusted enters the ecosystem, I think both of our careers explode and never recover. Like it’s—Kelly: Exactly.Corey: —yeah. Oh, yeah. They play fast and loose with secrets is never the reputation you want as a professional.Kelly: No. No, definitely not. So, it’s much more pattern matching and trying to generalize. But again, a lot of what can go wrong is not that different when you think about a developer being really tired and making a bunch of mistakes versus an attacker. A lot of times they’re very much the same, so luckily there’s commonality there.I do wish the security industry was more forthright and less clandestine because frankly, all of the public postmortems that are out there about performance issues are just such, such a boon for everyone else to improve what they’re doing. So, that’s a change I wish would happen.Corey: So, I have to ask, given that you talk about security, chaos engineering, and resilience-and of course, software and systems—all in the title of the O’Reilly book, who is the target audience for this? Is it folks who have the word security featured three times in their job title? Is it folks who are new to the space? What is your target audience start and stop?Kelly: Yes, so I have kept it pretty broad and it’s anyone who works with software, but I’ll talk about the software engineering audience because that is, honestly, probably out of anyone who I would love to read the book the most because I firmly believe that there’s so much that software engineering teams can do to sustain resilience and security and they don’t have to be security experts. So, I’ve tried to demystify security, make it much less arcane, even down to, like, how attackers, you know, they have their own development lifecycle. I try to demystify that, too. So, it’s very much for any team, especially, like, platform engineering teams, SREs, to think about, hey, what are some of the things maybe I’m already doing that I can extend to cover, you know, the security cases as well? So, I would love for every software engineer to check it out to see, like, hey, what are the opportunities for me to just do things slightly differently and have these great security outcomes?Corey: I really want to thank you for taking the time to talk with me about how you view these things. If people want to learn more, where’s the best place for them to find you?Kelly: Yes, I have all of the social media which is increasingly fragmented, [laugh] I feel like, but I also have my personal site, kellyshortridge.com. The official book site is securitychaoseng.com as well. But otherwise, find me on LinkedIn, Twitter, [Mastodon 00:30:22], Bluesky. I’m probably blanking on the others. There’s probably already a new one while we’ve spoken.Corey: Blue-ski is how I insist on pronouncing it as well, while we’re talking about—Kelly: Blue-ski?Corey: Funhouse pronunciation on things.Kelly: I like it.Corey: Excellent. And we will, of course, put links to all of those things in the [show notes 00:30:37]. Thank you so much for being so generous with your time. I really appreciate it.Kelly: Thank you for having me and being a fellow dinosaur nerd.Corey: [laugh]. Kelly Shortridge, Senior Principal Engineer at Fastly. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment about how our choice of dinosaurs is incorrect, then put the computer away and struggle to figure out how to open a door.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Brooke Sargent, Software Engineer at Honeycomb, joins Corey on Screaming in the Cloud to discuss how she fell into the world of observability by adopting Honeycomb. Brooke explains how observability was new to her in her former role, but she quickly found it to enable faster learning and even a form of self care for herself as a developer. Corey and Brooke discuss the differences of working at a large company where observability is a new idea, versus an observability company like Honeycomb. Brooke also reveals the importance of helping people reach a personal understanding of what observability can do for them when trying to introduce it to a company for the first time. About BrookeBrooke Sargent is a Software Engineer at Honeycomb, working on APIs and integrations in the developer ecosystem. She previously worked on IoT devices at Procter and Gamble in both engineering and engineering management roles, which is where she discovered an interest in observability and the impact it can have on engineering teams.Links Referenced: Honeycomb: https://www.honeycomb.io/ Twitter: https://twitter.com/codegirlbrooke TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. This promoted guest episode—which is another way of saying sponsored episode—is brought to us by our friends at Honeycomb. And today’s guest is new to me. Brooke Sargent is a software engineer at Honeycomb. Welcome to the show, Brooke.Brooke: Hey, Corey, thanks so much for having me.Corey: So, you were part of I guess I would call it the new wave of Honeycomb employees, which is no slight to you, but I remember when Honeycomb was just getting launched right around the same time that I was starting my own company and I still think of it as basically a six-person company versus, you know, a couple of new people floating around. Yeah, turns out, last I checked, you were, what, north of 100 employees and doing an awful lot of really interesting stuff.Brooke: Yeah, we regularly have, I think, upwards of 100 in our all-hands meeting, so definitely growing in size. I started about a year ago and at that point, we had multiple new people joining pretty much every week. So yeah, a lot of new people.Corey: What was it that drove you to Honeycomb? Before this, you spent a bit of time over Procter and Gamble. You were an engineering manager and now you’re going—you went from IC to management and now you’re IC again. There’s a school of thought that I vehemently disagree with, that that’s a demotion. I think they are orthogonal skill sets to my mind, but I’m curious to hear your journey through your story.Brooke: Yeah, absolutely. So yeah, I worked at Procter and Gamble, which is a big Cincinnati company. That’s where I live and I was there for around four years. And I worked in both engineering and engineering management roles there. I enjoy both types of roles.What really drove me to Honeycomb is, my time at Procter and Gamble, I spent probably about a year-and-a-half, really diving into observability and setting up an observability practice on the team that I was on, which was working on connected devices, connected toothbrushes, that sort of thing. So, I set up an observability practice there and I just saw so much benefit to the engineering team culture and the way that junior and apprentice engineers on the team were able to learn from it, that it really caught my attention. And Honeycomb is what we were using and I kind of just wanted to spend all of my time working on observability-type of stuff.Corey: When you say software engineer, my mind immediately shortcuts to a somewhat outdated definition of what that term means. It usually means application developer, to my mind, whereas I come from the world of operations, historically sysadmins, which it still is except now, with better titles, you get more money. But that’s functionally what SRE and DevOps and all the rest of the terms still currently are, which is, if it plugs into the wall, congratulations. It’s your problem now to go ahead and fix that thing immediately. Were you on the application development side of the fence? Were you focusing on the SRE side of the world or something else entirely?Brooke: Yeah, so I was writing Go code in that role at P&G, but also doing what I call it, like, AWS pipe-connecting, so a little bit of both writing application code but also definitely thinking about the architecture aspects and lining those up appropriately using a lot of AWS serverless and managed services. At Honeycomb, I definitely find myself—I’m on the APIs and partnerships team—I find myself definitely writing a lot more code and focusing a lot more on code because we have a separate platform team that is focusing on the AWS aspects.Corey: One thing that I find interesting is that it is odd, in many cases, to see, first, a strong focus on observability coming from the software engineer side of the world. And again, this might be a legacy of where I was spending a lot of my career, but it always felt like getting the application developers to instrument whatever it was that they were building felt in many ways like it was pulling teeth. And in many further cases, it seemed that you didn’t really understand the value of having that visibility or that perspective into what’s going on in your environment, until immediately after. You really wished you had that perspective into what was going on in your environment, but didn’t. It’s similar to, no one is as zealous about backups as someone who’s just suffered a data loss. Same operating theory. What was it that you came from the software engineering side to give a toss about the idea of observability?Brooke: Yeah, so working on the IoT—I was working on, like, the cloud side of things, so in Internet of Things, you’re keeping a mobile application, firmware, and cloud synced up. So, I was focused on the cloud aspect of that triangle. And we got pretty close to launching this greenfield IoT cloud that we were working on for P&G, like, we were probably a few months from the initial go-live date, as they like to call it, and we didn’t have any observability. We were just kind of sending things to CloudWatch logs. And it was pretty painful to figure out when something went wrong, from, like, you know, hearing from a peer on a mobile app team or the firmware team that they sent us some data and they’re not seeing it reflected in the cloud that is, like, syncing it up.Figuring out where that went wrong, just using CloudWatch logs was pretty difficult and syncing up the requests that they were talking about to the specific CloudWatch log that had the information that we needed, if we had even logged the right thing. And I was getting a little worried about the fact that people were going to be going into stores and buying these toothbrushes and we might not have visibility into what could be going wrong or even being able to be proactive about what is going wrong. So, then I started researching observability. I had seen people talking about it as a best practice thing that you should think about when you’re building a system, but I just hadn’t had the experience with it yet. So, I experimented with Honeycomb a bit and ended up really liking their approach to observability. It fit my mental model and made a lot of sense. And so, I went full-steam ahead with implementing it.Corey: I feel what you just said is very key: the idea of finding an observability solution that keys into the mental model that someone’s operating with. I found that a lot of observability talk sailed right past me because it did not align with that, until someone said, “Oh yeah, and then here’s events.” “Well, what do you mean by event?” It distills down to logs. And oh, if you start viewing everything as a log event, then yeah, that suddenly makes a lot of sense, and that made it click for me in a way that, honestly, is a little embarrassing that it didn’t before then.But I come from a world before containers and immutable infrastructure and certainly before the black boxes that are managed serverless products, where I’m used to, oh, something’s not working on this Linux box. Well, I have root, so let’s go ahead and fix that and see what’s going on. A lot of those tools don’t work, either at scale or in ephemeral environments or in scenarios where you just don’t have the access to do the environment. So, there’s this idea that if you’re trying to diagnose something that happened and the container that it happened on stopped existing 20 minutes ago, your telemetry game has got to be on point or you’re just guessing at that point. That is something that I think I did myself a bit of a disservice by getting out of hands-on keyboard operations roles before that type of architecture really became widespread.Brooke: Yeah, that makes a lot of sense. On the team that I was on, we were using a lot of AWS Lambda and similarly, tracking things down could be a little bit challenging. And emitting telemetry data also has some quirks [laugh] with Lambda.Corey: There certainly is. It’s also one of those areas that, on some level, being stubborn to adopt it works to your benefit. Because when Lambda first came out, it was a platform that was almost entirely identified by its constraints. And Amazon didn’t do a terrific job, at least in the way that I tend to learn, of articulating what those constraints are. So, you learn by experimenting and smacking face first into a lot of those things.What the hell do you mean you can’t write to the file? Oh, it’s a read-only file system. [except slash tap 00:08:39]. What do you mean, it’s only half a gigabyte? Oh, that’s the constraint there. Well, what do you mean, it automatically stops after—I think back in that point it was five or ten minutes; it’s 15 these days. But—Brooke: Right.Corey: —I guess it’s their own creative approach to solving the halting problem from computer science classes, where after 15 minutes, your code will stop executing, whether you want it to or not. They’re effectively evolving these things as we go and once you break your understanding in a few key ways, at least from where I was coming from, it made a lot more sense. But ugh, that was a rough couple of weeks for me.Brooke: Yeah [laugh]. Agreed.Corey: So, a topic that you have found personally inspiring is that observability empowers junior engineers in a bunch of ways. And I do want to get into that, but beforehand, I am curious as to the modern-day path for SREs because it doesn’t feel to me like there is a good answer for, “What does a junior SRE look like?” Because the answer is, “Oh, they don’t.” It goes back to the old sysadmin school of thought, which is that, oh, you basically learn by having experience. I’ve lost count a number of startups I’ve encountered where you have a bunch of early-20-something engineers but the SRE folks are all generally a decade into what they’re what they’ve been doing because the number-one thing you want to hear from someone in that role is, “Oh, the last time I saw it, here’s what it was.” What is the observability story these days for junior engineers?Brooke: So, with SRE I agr—like, that’s a conversation that I’ve had a lot of times on different teams that I’ve been on, is just can a junior SRE exist? And I think that they can.Corey: I mean, they have to because otherwise, it’s well, where does it SRE come from? Oh, they spring—Brooke: [laugh].Corey: —fully formed from the forehead of some God out of mythology. It doesn’t usually work that way.Brooke: Right. But you definitely need a team that is ready to support a junior SRE. You need a robust team that is interested in teaching and mentoring. And not all teams are like that, so making sure that you have a team culture that is receptive to taking on a junior SRE is step number one. And then I think that the act of having an observability practice on a team is very empowering to somebody who is new to the industry.Myself, I came from a self-taught background, learning to code. I actually have a music degree; I didn’t go to school for computer science. And when I finally found my way to observability, it made so many, kind of, light bulbs go off of just giving me more visuals to go from, “I think this is happening,” to, “I know this is happening.” And then when I started mentoring juniors and apprentices and putting that same observability data in front of them, I noticed them learning so much faster.Corey: I am curious in that you went from implementing a lot of these things and being in a management role of mentoring folks on observability concepts to working for an observability vendor, which is… I guess I would call Honeycomb the observability vendor. They were the first to really reframe a lot of how we considered what used to be called monitoring and now it’s called observability, or as I think of it, hipster monitoring.Brooke: [laugh].Corey: But I am curious as to when you look at this, my business partner wrote a book for O’Reilly, Practical Monitoring, and he loved it so much that by the end of that book, he got out of the observability monitoring space entirely and came to work on AWS bills with me. Did you find that going to Honeycomb has changed your perspective on observability drastically?Brooke: I had definitely consumed a lot of Honeycomb’s blog posts, like, that’s one of the things that I had loved about the company is they put out a lot of interesting stuff, not just about observability but about operating healthy teams, and like you mentioned, like, a pendulum between engineering management and being an IC and just interesting concepts within our industry overall as, like, software engineers and SREs. So, I knew a lot of the thought leadership that the company put out, and that was very helpful. It was a big change going from an enterprise like Procter and Gamble to a startup observability company like Honeycomb, just—and also, going from a company that very much believes in in-person work to remote-first work at Honeycomb, now. So, there were a lot of, like, cultural changes, but I think I kind of knew what I was getting myself into as far as the perspective that the company takes on observability.Corey: That is always the big, somewhat awkward question because of the answer goes a certain way, it becomes a real embarrassment, but I’m confident enough, having worked with Honeycomb as a recurring sponsor and having helped out on the AWS bill side of the world since you were a reference client on both sides of that business, I want to be very clear that I don’t think I’m throwing you under a bus on this one. But do you find that the reality, now that you’ve been there for a year, has matched the external advertising and the ethos of the story they tell about Honeycomb from the outside?Brooke: I definitely think it matches up. One thing that is just different about working inside of a company like Honeycomb versus working at a company that doesn’t have any observability at all yet, is that there are a lot of abstraction layers in our codebase and things like that. So, me being a software engineer and writing code Honeycomb compared to P&G, I don’t have to think about observability as much because everybody in the company is thinking about observability and had thought about it before I joined and had put in a lot of thought to how to make sure that we consistently have telemetry data that we need to solve problems versus I was thinking about this stuff on the daily at P&G.Corey: Something I’ve heard from former employees of a bunch of different observability companies has a recurring theme to it, and that it’s hard to leave. Because when you’re at an observability company, everything is built with an eye toward observability. And there’s always the dogfooding story of, we instrument absolutely everything we have with everything that we sell the customers. Now, in practice, you leave and go to a different company, that is almost never going to be true, if for no other reason than based on simple economics. Turning on every facet of every observability tool that a given company sells becomes extraordinarily expensive and is an investment decision, so companies say yes to some, no to others. Do you think you’re going to have that problem if and when you decide it’s time to move on to your next role, assuming of course, that it’s not at a competing observability company?Brooke: I’m sure there will be some challenges if I decide to move on from working for observability platforms in the future. The one that I think would be the most challenging is joining a team where people just don’t understand the value of observability and don’t want to invest, like, the time and effort into actually instrumenting their code, and don’t see why they need to do it, versus just, like, they haven’t gotten there yet or they haven’t had enough people hired to do it just yet. But if people are actively, like, kind of against the idea of instrumenting your code, I think that would be really challenging to kind of shift to especially after, over the last two-and-a-half years or so, being so used to having this, like, extra sense when I’m debugging problems and dealing with outages.Corey: I will say, it was a little surreal the first time I wound up taking a look at Honeycomb’s environment—because I do believe that cost and architecture are fundamentally the same thing when it comes to cloud—and you had clear lines of visibility into what was going on in your AWS bill by way of Honeycomb as a product. And that’s awesome. I haven’t seen anyone else do that yet and I don’t know that it would necessarily work as well because, as you said, there, everyone’s thinking about it through this same shared vision, whereas in a number of other companies, it flat out does not work that way. There are certain unknowns and questions. And from the outside, and when you first start down this path, it feels like a ridiculous thing to do, until you get to a point of seeing the payoff, and yeah, this makes an awful lot of sense.I don’t know that it would, for example, work as a generic solution for us to roll out to our various clients and say, oh, let’s instrument your environment with this and see what’s going on because first, we don’t have that level of ability to make change in customer environments. We are read-only for some very good reasons. And further, it also seems like it’s a, “Step one: change your entire philosophy around these sorts of things so we can help you spend less on AWS,” seems like a bit of a tall order.Brooke: Yeah, agreed. And yeah, on previous teams that I’ve been on, I definitely—and I think it’s fair, absolutely fair, that there were things where, especially using AWS serverless services, I was trying to get as much insight as possible into adding some of these services to our traces, like, AppSync was one example where I could not for the life of me figure out how to get AppSync API requests onto my Honeycomb trace. And I spent a lot of time trying to figure it out. And I had team members that would just be, like, you know, “Let’s timebox this; let’s not, like, sink all of our time into it.” And so, I think as observability evolves, hopefully, carving out those patterns continues to get easier so that engineers don’t have to spend all of their time, kind of, carving out those patterns.Corey: It feels like that’s the hard part, is the shift in perspective. Instrumenting a given tool into an environment is not the heavy lift compared to appreciating the value of it. Do you find that that was an easy thing for you to overcome, back when you were at Procter and Gamble, as far as people already have bought in, on some level, to observability from having seen it in some kind of scenarios where it absolutely save folks’ bacon? Or was it the problem of, first you have to educate people about the painful problem that they have before they realize it is in fact, A, painful, and B, a problem, and then C, that you have something to sell them that will solve that? Because that pattern is a very hard sales motion to execute in most spaces. But you were doing it at it, from the customer side first.Brooke: Yeah. Yeah, doing it from the customer side, I was able to get buy-in on the team that I was on, and I should also say, like, the team that I was on was considered an innovation team. We were in a separate building from, like, the corporate building and things like that, which I’m sure played into some of those cultural aspects and dynamics. But trying to educate people outside of our team and trying to build an observability practice within this big enterprise company was definitely very challenging, and it was a lot of spending time sharing information and talking to people about their stack and what languages and tools that they’re using and how this could help them. I think until people have had that, kind of, magical moment of using observability data to solve a problem for themselves, it’s very hard, it can be very hard to really make them understand the value.Corey: That was is always my approach because it feels like observability is a significant and sizable investment in infrastructure alone, let alone mental overhead, the teams to manage these things, et cetera, et cetera. And until you have a challenge that observability can solve, it feels like it is pure cost, similar to backups, where it’s just a whole bunch of expense for no benefit until suddenly, one day, you’re very glad you had it. Now, the world is littered with stories that are very clear about what happens when you don’t have backups. Most people have a personal story around that, but it feels like it’s less straightforward to point at a visceral story where not having observability really hobbled someone or something.It feels like—because in the benefit of perfect hindsight, oh yeah, like a disk filled up and we didn’t know about that. Like, “Ah, if we just had the right check, we would have caught that early on.” Yeah, coulda, woulda shoulda, but it was a cascading failure that wasn’t picked up until seven levels downstream. Do you think that that's the situation these days or am I misunderstanding how people are starting to conceive about this stuff?Brooke: Yeah. I mean, I definitely have a couple of stories of even once I was on the journey to observability adoption—which I call it a journey because you don’t just—kind of—snap your fingers and have observability—I started with one service, instrumenting that and just, like, gradually, over sprint’s would instrument more services and pull more team members in to do that as well. But when we were in that process of instrumenting services, there was one service which was our auth service—which maybe should have been the first one that we instrumented—that a code change was made and it was erroring every time somebody tried to sign up in the app. And if we had observability instrumentation in place for that service, it wouldn’t have taken us, like, the four or five hours to find the problem of the one line of code that we had changed; we would have been able to see more clearly what error was happening and what line of code it was happening on and probably fix it within an hour.And we had a similar issue with a Redshift database that we were running more on the metrics side of things. We were using it to send analytics data to other people in the company and that Redshift database just got maxed out at a certain point. The CPU utilization was at, like, 98% and people in the company were very upset and [laugh] having a very bad time querying their analytics data.Corey: It’s a terrific sales pitch for Snowflake, to be very direct, because you hear that story kind of a lot.Brooke: Yeah, it was not a fun time. But at that point, we started sending Redshift metrics data over to Honeycomb as well, so that we could keep a better pulse on what exactly was happening with that database.Corey: So, here’s sort of the acid test: people tend to build software when they’re starting out greenfield, in ways that emphasize their perspective on the world. For example, when I’m building something new, doesn’t matter if it’s tiny or for just a one-off shitposting approach, and it touches anything involving AWS, first thing I do out of the gate is I wind up setting tags so that I can do cost allocation work on it; someday, I’m going to wonder how much this thing cost. That is, I guess my own level of brokenness.Brooke: [laugh].Corey: When you start building something at work from scratch, I guess this is part ‘you,’ part ‘all of Honeycomb,’ do you begin from that greenfield approach of Hello World of instrumenting it for observability, even if it’s not explicitly an observability-focused workload? Or is it something that you wind up retrofitting with observability insights later, once it hits a certain point of viability?Brooke: Yeah. So, if I’m at the stage of just kind of trying things out locally on my laptop, kind of outside of, like, the central repo for the company, I might not do observability data because I’m just kind of learning and trying things out on my laptop. Once I pull it into our central repo, there is some observability data that I am going to get, just in the way that we kind of have our services set up. And as I’m going through writing code to do this whatever new feature I’m trying to do, I’m thinking about what things, when this breaks—not if it breaks; when it breaks [laugh]—am I going to want to know about in the future. And I’ll add those things, kind of, on the fly just to make things easier on myself, and that’s just kind of how my brain works at this point of thinking about my future self, which is, kind of like, the same definition of self-care. So, I think of observability as self-care for developers.But later on, when we’re closer to actually launching a thing, I might take another pass at just, like, okay, let’s once again take a look at the error paths and how this thing can break and make sure that we have enough information at those points of error to know what is happening within a trace view of this request.Corey: My two programming languages that I rely on the most are enthusiasm and brute force, and I understand this is not a traditional software engineering approach. But I’ve always found that having to get observability in involved a retrofit, on some level. And it always was frustrating to me just because it felt like it was so much effort in various ways that I’ve just always kicked myself: I should have done this early on. But I’ve been on the other side of that, and it’s like, should I instrument this with good observability? No, that sounds like work. I want to see if this thing actually works at all, or not first.And I don’t know what side of the fence is the correct one to be on, but I always find that I’m on the wrong one. Like, I don’t know if it’s, like, one of those, there’s two approaches and neither one works. I do see in client environments where observability is always, always, always something that has to be retrofit into what it is that they’re doing. Does it stay that way once companies get past a certain point? Does observability of adoption among teams just become something that is ingrained into them or do people have to consistently relearn that same lesson, in your experience?Brooke: I think it depends, kind of, on the size of your company. If you are a small company with a, you know, smaller engineering organization where it’s not, I won’t say easy, but easier to get kind of full team buy-in on points of view and decisions and things like that, it becomes more built-in. If you’re in a really big company like the one that I came from, I think it is continuously, like, educating people and trying to show the value of, like, why we are doing this—coming back to that why—and like, the magical moment of, like, stories of problems that have been solved because of the instrumentation that was in place. So, I guess, like most things, it’s an, ‘it depends.’ But the larger that your company becomes, I think the harder it gets to keep everybody on the same page.Corey: I am curious, in that I tend to see the world through AWS bills, which is a sad, pathetic way to live that I don’t recommend to basically anyone, but I do see the industry, or at least my client base, forming a bit of a bimodal distribution. On one side, you have companies like Honeycomb, including, you know, Honeycomb, where the majority of your AWS spend is driven by the application that is Honeycomb, you know, the SaaS thing you sell to people to solve their problems. The other side of the world are companies that look a lot more like Procter and Gamble, presumably, where—because I think of oh, what does Procter and Gamble do? And the answer is, a lot. They’re basically the definition of conglomerate in some ways.So, you look at that, a bill at a big company like that and it might be hundreds of millions of dollars, but the largest individual workload is going to be a couple million at best. So, it feels very much like it’s this incredibly diffuse selection of applications. And in those environments, you have to start thinking a lot more about centralization things you can do, for example, for savings plan purchases and whatnot, whereas at Honeycomb-like companies, you can start looking at, oh, well, you have this single application that’s the lion’s share of everything. We can go very deep into architecture and start looking at micro-optimizations here that will have a larger impact. Having been an engineer at both types of companies, do you find that there’s a different internal philosophy, or is it that when you’re working in a larger company on a specific project, that specific project becomes your entire professional universe?Brooke: Yeah, definitely at P&G, for the most part, IoT was kind of the center of my universe. But one philosophy that I noticed as being different—and I think this is from being an enterprise in a startup—is just the way that thinking about cost and architecture choices, kind of, happened. So, at P&G, like I said, we were using a lot of Lambda, and pretty much any chance we got, we used a serverless or managed offering from AWS. And I think a big part of that reasoning was because, like I said earlier, P&G is very interested in in-person work. So, everybody that we hired her to be located in Cincinnati.And it became hard to hire for people who had Go and Terraform experience because a lot of people in the Midwest are much more comfortable in .NET and Java; there’s just a lot more jobs using those technologies. So, we had a lot of trouble hiring and would choose—because P&G had a lot of money to spend—to give AWS that money because we had trouble finding engineers to hire, whereas Honeycomb really does not seem to have trouble hiring engineers. They hire remote employees and lots of people are interested in working at Honeycomb and they also do not have the bank account [laugh] that Procter and Gamble has, so just thinking about cost and architecture is kind of a different beast. So, at Honeycomb, we are building a lot more services versus just always choosing a serverless or easy, like, AWS managed option to think about it less.Corey: Yeah, at some level, it’s an unfair question, just because it comes down to, in the fullness of time, even Honeycomb turns into something that looks a lot more like Procter and Gamble. Because, okay, you have the Honeycomb application. That’s great, but as the company continues to grow and offer different things to different styles of customers, you start seeing a diffusion where, yeah, everything stills observability focused, but I can see a future where it becomes a bunch of different subcomponents. You make acquisitions of other companies that wind up being treated as separate environments and the rest. And in the fullness of time, I can absolutely see that that is the path that a lot of companies go down.So, it might also just be that I’m looking at this through a perspective lens of… just company stage, as opposed to what the internal story of the company is. I mean, Procter and Gamble’s, what, a century old give or take? Whereas Honeycomb is an ancient tech company, by which I mean it’s over 18 months old.Brooke: Yeah, P&G was founded in 1837. So that’s—Corey: Almost 200 years old. Wonderful.Brooke: —quite old [laugh]. Yeah [laugh].Corey: And for some reason, they did not choose to build their first technical backbone on top of AWS back then. I don’t understand why, for the life of me.Brooke: [laugh]. Yeah, but totally agree on your point that the kind of difference of thinking about cost and architecture definitely comes from company’s stage rather than necessarily the industry.Corey: I really want to thank you for taking the time out of your day to talk with me about what you’re up to and how you view these things. If people want to learn more, what’s the best place for them to find you?Brooke: Yeah, so I think the main place that I still sometimes am, is Twitter: @codegirlbrooke is my username. But I’m only there sometimes, now [laugh].Corey: I feel like that’s a problem a lot of us are facing right now. Like, I’m more active on Bluesky these days, but it’s still invite only and it feels like it’s too much of a weird flex to wind up moving people to just yet. I’m hoping that changes soon, but we’ll see how it plays. We’ll, of course, put links to that in the [show notes 00:31:53]. I really want to thank you for taking the time out of your day to talk with me.Brooke: Yeah, thanks so much for chatting with me. It was a good time.If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.