Amrit Williams, BigFix CTO, investigates the new RUGGED Software Manifesto with its authors by Joshua Corman and David Rice.

Subscribe in iTunes:

Subscribe with XML:

FULL TRANSCRIPT

Amrit Williams: Welcome! This is Amrit Williams, I’m your host on “Beyond the Perimeter”, and today I am joined by Joshua Corman, Enterprise Security Practice Research Director at the 451 Group, and David Rice, Executive Director at the Monterey Group.

Guys, thanks for joining me today.

What I wanted to talk to you guys about today is the Rugged Software Manifesto you and David and Jeff put together. Basically it is an awareness campaign with some tenets around how developers should be looking at developing software in regards to making it more survivable, more available, more secure. I hope I’m not misstating that too much, so why don’t I turn over to you guys, and just take the audience through what is Rugged Software, what is the Rugged Software Manifesto, what are you guys trying to achieve?

Joshua Corman: I think the genesis of this idea was we were at the Grayloc party at RSA last year — and I think you’ve agreed with me on this before, Amrit — but each year at RSA a lot of us kind of feel like we should quit security. And I say that tongue-in-cheek; but we tend to be fairly frustrated, because we hear a very similar message year to year to year, and it seems like a very stagnant and static security approach to a very dynamic and evolutionary problem.

So I met David through the Institute for Applied Network Security, and I know you got Jack Phillips on the podcast in the past; but David’s book on economics really made me look at the economic impact and the economic incentives and disincentives in the true cost of weak software. And obviously he can speak to his book better that I can; but he and I had some fairly, intellectually honest and vigorous debates over the years.

So we were speaking about some of the disdain for the Security community at this party, and he introduced me to Jeff Williams from OWASP; he is the Chairman of OWASP right now. And we were kind of say, “You know, it’s nice that we have web-app firewalls required by PCI. It’s nice that we have static and dynamic analysis. We have lots of tools and technologies — and, no offense, Jeff: OWASP is doing a fantastic job for the people who know about security — but until we get to the hearts and minds of all developers, until we can let them know that there’s a security context and that their software has become modern infrastructure, we’re fighting the heads of the Hydra and not the heart. We’re at the end of the lifecycle instead of the beginning, and part of the fix here to drive down the cost and complexity of security and make sure we don’t have 70 security products and 80 or 90 next year is, we need to inherently more secure infrastructure.

So we were talking about Agile, we are talking about different frameworks and what we said is, “You know, we don’t really have our own manifesto; we don’t have our own meme, our M-E-M-E. And we need a contagious philosophy or values that people can sink their teeth into”. So after a couple of hours, we came up with Rugged, R-U-G-G-E-D, and it was to have this concept or notion that you will be tested and that you’re tough enough to survive it or that you’ll have a mission to do and you’ll be attacked, you’ll live out longer than you were intended to; and we decided to come up with our own Manifesto that night back at RSA.

You want to pick it up from there, David?

David Rice: So in many ways, Rugged is simply an awareness campaign. I had to preach outside the choir of cyber security.

Unknown Male: We get it.

David Rice: And everyone in cyber security thinks, gets that there is a need; but the perceived need outside of the Security community is nonexistent. And so there is a notion that we create software to live in this utopian environment filled with unicorns and butterflies, and we’ll be all right on the Internet. What we know now from all of the news stories and data that we do have is that the Internet is not this utopian environment that it was originally envisioned to be; it is truly a UFC ring match, and unless your software can survive in that brutal, hostile environment, the very value that we’re trying to create through software can be called into question — not that we’re destroying all value, but that the notion that these attackers are spending a tremendous amount of effort across the board to get into software that, by one infrastructure report, recognizes that software is one of most effective products on the planet.

Well, that’s a really devastating comment; but it also has this notion that they say, “Well, okay. If this is an awareness campaign, we need to get outside of the choir and let people see that, in Rugged mentality, that is a different mindset is required than what was before”. Beyond that, it’s all detail. But it’s really that fundamental, simple but monumental task of just doing a bit flip, that little switch in the head that says, “Well, I have to think about the environment that this code is going to be running in, and for all intents and purposes we know that it’s hostile now”. And that’s fundamentally, I think, what we’re trying to do. It’s a very simple message, a very direct message, but monumental in its impact.

Joshua Corman: Yeah. And we did do this in San Francisco, and so just being in that environment I remember when I was a kid with all the earthquakes we’d get, my dad told me that — and I now know this as an adult — but if you’re an architect and you build a skyscraper in San Francisco, I mean, you can’t just build a building there the same way you can somewhere else. You need to factor in the earthquakes and design for it.

(00:05:09)

And I guess that was the same kind of sphere we were communicating at the SANS Conference was, “Look, this becomes a design parameter now that the environment your code is going to face, your code is going to live on well beyond its intended lifespan and is going to be used in ways you couldn’t anticipate for longer than it was ever supposed to, and there are talented adversaries in every series who seek to undermine it”. So the Manifesto that comes out of that: just trying to let problem-solvers — I mean, my first five years of my career were in a software-development organization, and engineers have an incredibly high standard for their work.

And when we first learned the memory leaks, they resisted at first; but eventually they were very, very proud to make sure that their code never had leaks in it when we ran it through the leak tests. And I realized that instead of all these other campaigns trying to call them stupid or lazy or ignorant, honestly if we can get an increased or enhanced worldview in front of them, programmers are very good problem-solvers, and in my experience even if 5% more people say, “Oh, I didn’t realize that”, it affects their choices on programming languages or avoids some of the common mistakes, drives them to their first OWASP meeting, gets them maybe to evaluate the differences between different frameworks. It’s not a silver bullet, but we simply can’t keep preaching to the choir that already knows about this.

Amrit Williams: Yeah, I would guess that anybody you talk to in Security if you said, “We would really like to create more awareness around developers taking security development of their software and improving the security of it more seriously”, there probably isn’t a single person in Security that would disagree that that’s a good thing — or, a bad thing.

What I wanted to focus on a little bit, though, is that I did read the Rugged Software Manifesto, I reviewed the presentation that you guys put together, I tried to view it from the perspective of a developer and what I would think about it, and the one thought that I walked away with is when put in a context or against the backdrop of something like the Agile Manifesto is the way that the Rugged Software Manifesto is laid out — and for those folks on the phone who’d like to find out more about it, they can go to ruggedsoftware.org and they can read the Manifesto there — is it’s presented in a way that I felt was a touch condescending, and I didn’t feel like it was empowering me. I mean, there had been the same struggle that has occurred with quality, and if you look at software development in the early ‘90s, no one really took QA or Quality Assurance seriously. It was really difficult to get folks to use even tools that were well-known like Purify and Quantify, and then there was a lot of folks who looked at memory leakage, who looked at performance technology tools and they did get excited about being the ones who could develop less-buggy software.

And I felt that some of the language here is not as empowering. Am I reading this wrong? Is that a fair comment? I mean, how do you guys address the developer mindset, because it’s pretty much from a perspective of a Security Analyst reading this, say, would go, “I completely agree, I’m on board, I’ll do whatever I can to support you”. From a development perspective, I felt like it was worded in such a way that I took exception with it.

David Rice: It listed a notion of a competition. And so the first perception is very important, and so I agree with you that the first impression of the taste-test sales, there is a difficulty there.

And then it’s also balancing against the fact that “Well, we have some very real data out there now, as soft as it may be; but we know that software is a critical aspect that’s enabling many of the attackers to come in”.

So part of what the Manifesto is doing is eliciting a notion of competition. And that is, developers typically live in a meritocracy; that is, it’s very much based on skills and capability. And what the Manifesto, in one of the lines of the Manifesto that we’re bringing out, is that this is a competition not between software companies, but between those individuals that wish to take away or undermine the value that these companies are trying to provide. So it’s not a competition between software companies; it is now a competition between attackers and the developers.

And one aspect of this is to elicit the notion that, “Well, who’s better here?” These hackers are talented; but we know that developers are far more talented. And certainly there’s pressures on their development: timelines, et cetera. Mainly it is a notion of trying to elicit this notion that there is a competition here between people on the outside wanting to undermine the software to get to the crown jewels of these companies, and the notion that, “Well, who’s really better here?”

And our vote is a very positive vote. It is that as Josh said before: developers are smart folks. These individuals are dedicated to their craft. And so that craft needs to reflect the mindset of the developers and not reflect the abilities of the attackers, which we’re seeing here getting far more press that the actual good software developers that are out there.

Josh, you have more to add on that?

Joshua Corman: Yeah, I mean, I took the criticisms on the chin, and we did a lot of testing of the Manifesto language to make sure it was positive so that at least from a design perspective it was never our intention to be condescending; and I still take that criticism on the chin, and this is our 1.0 Manifesto and now that it’s out there we’re getting lots of great feedback on it.

I think when we presented this, someone said, “How is this different, Josh?” and I said, “Well, it’s different than other initiatives in this space in three ways”, and this touches on why I was disappointed to see that it didn’t pass the taste test universally.

But I said, “It’s different in three ways. Number one, we’re trying to get outside of the choir, because if we keep preaching to the same people who care about it, we’re only going to be solving a single-digit percentage of the community.

“Number two, it has to get to the hearts and minds, everything, the technology solution and the technology fix. This is going to be explicitly a hearts-and-minds’ thing, because want to tap into a value set someone already has versus talking down to them. So we’re looking to just light a spark.”

I know you had Michael Santarcangelo on your program, and it’s the catalyst kind of idea: does this value set we’re putting forth resonate with you, and if it does then we want you to opt into it and light a fire and put you in touch with more resources.

And the third is that often in this space when we want to improve things we force people to do it, and this was intentionally not telling you “Thou shalt, thou shalt; you must. Here is a compliance mandate and a fine”. I mean, you and I, Amrit, have talked many times about the dangers of just using compliance regulatory and the stick instead of the carrot. This one was meant to be aspirational and look for the best in people, instead of assuming that they’re all incompetent jerks.

So I think you know me well enough to know that I have the respect level for these folks, and if we didn’t hit a homerun in the first set of language, this is a conversation that we’ve started. We believe the concept or the attribute of being Rugged as an individual, as a chunk, as an organization, as a chunk of code, as a website. We are now trying to give the mainstream a word that works better or a concept that works better than Security, because clearly we’ve been telling people to care about security and they don’t, whereas we do feel that this somehow has a stickiness in it, a more contagious concept that is more attractive to a business owner. Rugged Cloud or Rugged Datacenter or Rugged Website, something that might have tipped the scales in the economic conversation in a way that previous dialogue hasn’t.

So I think this intends to be aspirational and seek the best in people, and if we have got some rough edges, it’s this kind of feedback that’s going to help us fix that.

Amrit Williams: Well, I completely agree, by the way. I’m very glad you didn’t call it The Secure Software Manifesto, because that would have been lambasted to no end, and what you’re really talking about in Rugged is survivability of the software itself in a very hostile environment, and I do appreciate that. I don’t think you guys wrote it to be condescending; but I was just making an observation as a developer, it almost says to me, “You’re doing it wrong; here, you have to do it right, and here is a set of values you should adopt”.

Developers, as you know, as you stated — which I really appreciated the way that David and you have stated that — is that they are very competitive by nature, and I think that there is a really nice thread for you guys to pull on here around driving competition for survivability and Rugged in terms of how folks develop software, as opposed to what is normally done, which is “You’re the root of all evil, you’re the reason we have these problems, you need to fix yourself”, which instantly puts people in a defensive type of stance, which is what some of what you saw on the mailing list that we’re on, which was I think a natural reaction of folks that really work really hard — I mean, developers are some of the hardest-working people in the industry, and as David mentioned earlier, they take great pride in their craft, and more than most people in Security, they went to school and got degrees in the sector that they’re actually in right now. There’s a lot of people in Security don’t have that background. So I appreciate that.

I wanted to switch gears a little bit from this, because I don’t think you guys are again purposely trying to be condescending. I really like that you guys didn’t include the word “secure” in here and you went after something that has and represents something that is about survivability. It’s about Ruggedness.

One of the other things that struck me was, I think there’s a lot of frameworks, there’s a lot of methodologies out there that people can adopt. I mean, I wrote a paper I sent to you when I was at Gartner around adding security and enveloping securities as part of the software-development lifecycle: everything from threat modeling to code reviews and design reviews and blah, blah, blah that included a security perspective.

I don’t think you actually will have that hard of a time convincing developers that there’s things that they can do to better enable survivability and Ruggedness in their software. I think the challenge that a lot of developers have, and if you sit down with them — and I’m sure you have — and talk to them, the response is “I would love to do this; but Management is telling me I have two days to do something, and they won’t let me adopt the tools and the processes that I need to adopt to better enable this inside of my environment”.

So I wanted to ask you guys a little bit about how you have a conversation with organizations that do want to adopt this, but are struggling with the perception that there will be increased cost, longer times to market, all the other things — and by the way, we’ve been through this with QA and Quality Assurance already, and it’s already been proven, right, that finding bugs earlier in the cycle are obviously more economically attractive than finding them in the general public. So what is the type of conversation you have with an organization that’s struggling with the economic incentives of moving to survivable Rugged software?

Male Speaker: There’s large systemic issues that come with this. One of them is just market pressure. And again, you know, Security is very good of defining doctrines, creeds, practices. I mean, we have a bevy of them. If you look all the way back to Watts Humphrey through the Software Engineering Institute, I mean, the man was given a medal by the President for his work in software quality; and yet a majority of the market really still ignores stuff that he’s really through fact proven that, yes, you can improve quality and security of software.

But I think what we have to realize from a Security perspective is that people don’t buy facts; they buy feelings. And part of that is a recognition of when I am pressured by my Managers to develop in a certain amount of time, I’ve got to make certain economic decisions; that is, my time is an economy, and there’s only so much time I can spend. Well, what Security comes in and does is throws in all these frameworks and all these things, and the immediate reaction is push-back: “There’s just too much for me to do”.

One of the aspects in terms of success is starting from where you are with what you have and do the best you can with it. And Rugged in its simplicity is simply a recognition of that bit flip in the brain that says, “Well, okay, I might not be able to do all this; but, I mean, heck, it’s there. It’s just that notion that “Well, gee, I need to do this bounce check” or “I need to check this input”.

That’s different than having to do all these huge external-compliance frameworks, and really that’s what they come down to be is: you must to do this to get secure software.

Well, what Josh, Jeff and I are really trying to look at is a value-driven, an internal mechanism where the hands on the keyboard have a mind behind it that says, “Well, wait a minute, what about this?” And if we get that one extra question in terms of “How do I make this software more resilient without having to go through all these frameworks, although they’re available”, then we’ve already started to change the direction of the ship. It’s a huge ship, and it’ll take years to change direction of; but once you get the hands on the keyboard and the minds that control those hands thinking just a little bit differently, that’s a huge success, because right now the value argument around secure software has not been made well. And so you can argue that, yes, fixing a bug after production or in production is 100 times more expensive than in the design time. Well, why do we still have bugs, then? I mean, we all know the economics. Why are we not paying attention to the economics? And it basically comes down to the value argument.

So what we’re trying to do here is make it an onramp; that is, the value argument is that it’s hard for organizations, because “people get it”. They know that “Okay, well, we might not be able to do everything in a purist mindset to develop secure software, but at least we can do something”. And if we can move towards Ruggedness, as opposed to achieving security, completely different level of expectations and there’s actually hope in that message. Like “Well, actually, if we do A, B and C, we might be able to get a little bit more Rugged”, as opposed to in a typical Security mindset you need Thous: thou shalt do ABCDEFGHIJKLMNOP — you know, you lost people at D. They’re already checked out at that point. But it’s conceptual to get people at ABC, wow, that’s a totally different ballgame.

Male Speaker: Yeah. And there is no silver bullet. I mean, there are strong economic disincentives to doing this with the current mindsets, and we recognize that part of the Rugged movement here that started with a personal pledge, very personal language for the software developers; but it is our immediate intention to start working groups in business cases on successful ways in which if you’re an engineer you’ve sold security to your employer; if you are an employer, you’ve sold a new framework or a new methodology down to your employees. What are the business drivers?

I was talking to Weiss Opel at VeriCode, and he and I had both started our careers in QA, it turns out; so we have similar heritage, I guess. But there are the traditional arguments, but there’s also arguments now and what he’s finding in the market is customers of the software can drive an economic action. So if someone is going to buy between your software package or a competitor’s software package, they’re putting in their RPs now to have the static and dynamic testing done and the analysis done by either Jeremiah’s company or one of the other product suites or Weiss Opel’s at VeriCode, and there are other value lovers that are going to be outside the range of what an individual developer could do.

But I also can’t forget for a second that my first pragmatic marketing course or product-management course said something like, “He or she who owns the compiler wins”, right? So don’t ever forget that the hands on the keyboard ultimately are the biggest tipping point on what the outcome is.

(00:20:04)

So despite deadlines, if you have a program where who now knows not to use a risky system call or in the language selection on which programming language they’re going to use or which influence in their methodology, a tremendous amount of power and purview and influence comes from each individual developer at the end of the day.

So this is how you change the world: just kind of one person at a time, one project at a time. And it’s not going to be just in the hands of  developers. It’s going to be a multi-altitude, multi-point of attack, and now we at least have a sticky concept that’s hopefully more effective than just calling things secure.

Amrit Williams: Yeah. And I actually agree with that in a paper that I wrote — god, I think I wrote that back in 2005 or 2004. I actually put a prediction in there that by 2008, organizations would start adding the security of the software itself as a critical evaluation factor. So it’s very encouraging to hear that you’re seeing the folks at VeriCode and White Hat hearing that back from the market. That’s definitely an encouraging thing.

I wanted to talk a little bit about real quick, you’ve mentioned a couple of times the economic disincentives for adopting this, and I want to switch the bid a little bit because I think there’s actually a lot of economic incentives for people to make their software more secure — “more secure”, I’ll even stay away from that  — but just more survivable and more Rugged, because of the impact that it could have in market adoption. But the problem is that it’s very difficult to prove that, and companies are very slow to respond to those market dynamics.

I mean, everyone always points to Microsoft as a great example of a company that does this well. They didn’t do it by choice, and they didn’t do it quickly. They did it very slowly over time, because the market really pressured them to do it.

You look at Adobe, and I don’t imagine Adobe is going to do anything quickly, either. They’re trying; but it’ll probably be a while before they actually get a response that’s acceptable by the market. It almost seems like this is so much market-driven.

But how do you raise that conversation up to an executive so that they understand how critically important it is, that it’s on the same level as software quality, that they do invest in it and that they do understand now?

Male Speaker: I think we’re really soft on the business cases, yeah. And I know that Jim Routh, who was formerly CISO at DTCC, Depository Trust & Clearing Corporation, in the program that he built up, over time he could show about a 10% to 12% savings over the lifetime of any given project when they embedded security into the actual project, as opposed to waiting until afterwards. If you walk into any executive office and say, “Listen, I can save you 10% to 12% on this project”, certainly that’ll raise the eyebrows.

But Gary McGraw has a great counterpoint, not directly to Jim; but the problem with metrics is that they’re like body organs: all of us have them, but I can’t take my liver and put it in Josh’s body without some consternation from Josh’s body.

So we can take certain metrics and try to apply them to other organization, but we’re still at a very nascent stage in terms of the business concept or the business drivers of software assurance. And so we can make some fairly good statements; whether or not those statements apply across the board will always be open to debate.

But I think a lot of Security executives are out there — and maybe in lines of business executive in some instances — actually see that value that is in a recession-pressured company to actually start realizing savings. I mean, if you look at cost-avoidance mechanisms, if we put in a CFL light bulb, we can show you a payback period in a two- to three-year timeframe. So we know it might be a large upfront expense, but we get a payback period of three years.

In software assurance, we’re still not quite there yet; but we’re getting closer. So between Jim’s work and other executives out there, I think people are starting to get their heads around how we can actually show the payback period for the business case of software assurance. We still have a long way to go.

Amrit Williams: It’s going to be critically important, because even when you do win over the hearts and minds of the developers, just as you would in the example that Josh gave about the memory problems or the boundary problems that they experienced in the ’90s, it was difficult for the developers to adopt. I imagine that even when they said “Completely agree”, there still had to be an economic case made to the executives that “I do need to invest in software, I do need to change processes and policies, and there will be a near-term impact hopefully to sustain a long-term gain”.

And so this is going to be really important to drive this message out is to wrap it around an ecosystem that can support those values and those ideals. So it’s really encouraging to hear that you guys are also pursuing those things, because there is, as you state, David, a real lack of that information as it pertains to this area.

David Rice: Yeah, you know, we’ve encountered quite a bit of like passionate and enthusiastic support from some places we never even anticipated. Like when I was talking to Joe Jarzombek from the Department of Homeland Security, and he was just a really enthusiastic supporter. And the enthusiasm’s been there and so has the criticism, and what I noticed after I got some of the criticisms, which was pretty much the minority; but people are looking like “You’re not going to have a quick fix, Josh”. And I said, “I know we’re not going to have a quick fix. This is a long view; we’re taking a long view at this”.

In fact, I actually expect that some of more fruitful progress we’re going to make has been with the conversations I’ve been having with universities that have an undergrad or postgrad program. And I’m not saying that we can’t teach an old dog new tricks. I mean, the people whose kids play or my kids, they’re still programmers, and just simply talking to them about this, they’ve already started looking into things, they’re going to go to their first OWASP meeting in the area, and I think we’re going to have some impact on the existing population. But we kind of have to have a long view. This is going to take time. Maybe it’s five years, maybe it’s ten years; but we were kidding in San Francisco and said, “What’s security going to be like in 100 years?” And I think there’s a tendency in the Security market to want a quick fix. Well, guess what? A bunch of quick fixes and instant Band-Aids, we’ve got that and we have 70 different product markets, and we’ve got firewalls and IPSs and the number of pizza boxes you could install in your network perimeter is staggering, and those quick fixes aren’t really getting at the systematic issues.

So this is not going to be a quick thing, it’s not a silver bullet; but I think the hearts and minds giving some sort of sticky concept that can be used at universities and the purchasers of software, the developers of software — I mean, heck, I didn’t even intend this initially; but as the Cloud adoption happens, there’s going to be datacenters or Cloud services that will have some failures, and how can they articulate in business terms that their Cloud is more Rugged or survivable than someone else’s Cloud? So this could become an economic token or totem that can be used to essentially slowly crank up the awareness, the design, the education; so it’s not going to be quick overnight, but it is going to hopefully permeate the way we approach these offerings.

David Rice: And not to spend too much time on it, but at the highest level in economics there’s really two driving forces: one, people will do whatever they can to make themselves better off; and, two, which is probably more important than one, is that people will not consciously do anything what they feel they’ll be worse off for doing. And right now we can simply answer the question of why don’t people or why don’t software developers do security a lot in their code? Well, because they don’t believe they’re going to be better off for doing it. I mean, at the highest level it’s the simplest question that they’re asking yourselves: will I be better off for doing this? And the answer is probably not. I know I’ll be worse off, because I’ve got more frameworks, I’ve got more work to do, I’ve got time pressures.

So at the highest level, economics really isn’t about numbers; it’s about people. And so what we’re trying to address is a core incentive, and that is again that bit flip that goes from “Oh, my gosh, I’ve got to do all this stuff” to being right to “Well, gee, if I can aspire to Ruggedness, can do what I can where I am to the best of my ability and actually make some progress” — and that’s hugely important, because then all of a sudden, like I said, people don’t buy facts, they buy feelings. Well, Rugged is a feeling, and that’s really important because what that does is drive different behaviors. And ultimately if they feel they’re better off and not worse off for becoming Rugged, well, then that’s a huge win for us and that’s a key aspect that “No, you can’t buy a pizza box for this; there’s just no way to do it”. And so we really want to get to the hands on the keyboards, because that’s where both the solution is and the aspiration.

Amrit Williams: And I appreciate that. I wish you guys the best of luck.

For those out there who want to get more information, they can visit www.ruggedsoftware.org. This is Joshua Corman, Enterprise Security Practice Research Director from the 451 Group who joined us, and David Rice, Executive Director at The Monterey Group.

Guys, I really appreciate you guys joining us today. If folks want to hear more from you directly, how can they contact you guys? Josh?

Joshua Corman: I’m on Twitter at Josh Corman, J-O-S-H, C-O-R-M-A-N; or email me at jcorman@the451group.com.

David Rice: And you can contact me at david@montereygrp.com, M-O-N-T-E-R-E-Y, G-R-P, dot com.

Amrit Williams: Really appreciate having you guys on, wish you the best of luck; I’ll see you guys at RSA, correct?

Joshua Corman: Yes.

David Rice: Wonderful. See you there.

Announcer: You have just listened to “Beyond the Perimeter”, sponsored by BigFix, Inc. Views expressed on this podcast are the personal opinions of podcast participants and do not reflect official positions of their employers or BigFix. Thanks for listening.