I’ve performed hundreds of security reviews over a decade. Here’s how to stack the deck in your favor.
One of the most critical phases of the secure software development cycle is the security audit: an independent assessment of one’s code or systems that’s designed to flag possible flaws. I’ve performed hundreds of assessments over a wide range of systems during the last decade including applications, networks, devices, wallets, and smart contracts. During that time, I’ve observed how most successful companies get their systems assessed by external reviewers. These processes largely follow what decent providers will tell you to do, mixed with some tactics that will help you to negotiate for better terms.
In this post, I’ll share what I’ve learned that can help you prepare for and follow best practices throughout — before, during, and after — a security review.
As a quick primer, let’s examine what makes a security review or audit successful. First, the reviewer needs to be an expert in your type of system with some degree of experience that’s directly applicable. Second, the reviewer needs to know where and how to look at the right things. And third, the reviewer must have enough time (and focus) to comb through the things that need looking at.
Beyond these three main criteria, there are other factors to consider. These include the knowledge of other team members supporting the reviewer, the proprietary tools available to the reviewer, and the approach of the reviewer. These other elements contribute less significantly to the audit’s success, but they are still worth considering.
First, think about what you want out of your review. If your answer is something lofty, like “find all the vulnerabilities in our system”, then you’re out of luck. Instead, set a more reasonable goal, like “get a good idea of the state of security of our key management infrastructure” or “look for opportunities for an attacker to exploit our smart contracts to siphon funds.” Keep these goals in mind as you go into an audit and stick to them as the success criteria for the assessment.
Going through a quick “threat modeling” session using existing frameworks like STRIDE or PASTA can be helpful to get an idea about what you want out of the review. If you’re less of a fan of formal threat models, you can simply have your engineers consider the following question: “What aspects of the system are we concerned about malfunctioning?” This exercise can help you understand which areas you want included in your review.
Once you have the goals and concerns recorded, you’re ready to understand what parts of the system should be scrutinized. This should include all systems reasonably required to accomplish the goals and address the areas of concern. (Good audit firms will typically help you through this process as part of project scoping.)
“Which vendor should I choose?” is a question I hear fairly often. Unfortunately, there isn’t a foolproof way to know who will perform the best. However, word of mouth recommendations can point you in the right direction. Asking around and comparing notes will make the selection process much easier.
A common mistake when getting started is to assume an assessment can be booked on demand. If you rush the engagement, it will likely lower the overall quality. Security vendors are busy, and specialized vendors may be more so. If you book an audit last minute, the vendor is going to end up choosing the best available reviewer — that is, someone who is typically either not in demand or who just happened to be on a project that was canceled or delayed. Either way, you’re depending on luck and hoping to get a good fit. This strategy is not advisable.
Set yourself up for success by engaging early with as many vendors as is reasonable (typically 2 or 3). This will give you a good chance at getting a good personnel fit for the assessment without bogging you down in too many meetings. With more lead time, you can wait for better personnel to be available and have a bigger pool to choose from. Think of it like hiring: The more candidates the better (preferably from vetted sources).
How early you should engage depends on the type of assessment and the time of year. Factors that increase the required lead time:
Take subject matter expertise like cryptography, for example. Audit firms may have only a select few PhD cryptographers on staff who are capable of performing an in-depth cryptographic audit. If those experts are booked, you’re going to have to wait until their other assessments are complete. Depending on market demand, you may need to push out your timeline by at least 2–3 months.
Essentially, it comes down to the following factors: “How many people can do this?” (specialization, pickiness) and “When can they do this?” (longer assessments make it more difficult to find time). For fairly run-of-the-mill assessments (e.g., web applications, network assessments, solidity smart contracts), you can expect to need a few weeks lead time. If you’re attempting to book auditors who work with large companies, you may need to schedule months in advance to get engagements between October and December. If you’re in an especially active and new technology sector, timelines can balloon. Always think about scheduling months in advance.
During this process, be as straightforward with the vendors as you can be. Share information, assumptions, and documentation proactively. This will help them help you. Make sure you’re answering scoping questions to the best of your ability. These questions could look like: “What are the services involved in management of cryptographic materials?” Respond promptly to requests for information. The more openly you communicate, the better chance you’ll have of getting the right staffing for the project.
This is where the companies experienced with getting assessments start to shine. Basically, you can be choosy about who you hire to look at your systems. You may be a small company, but you can still ask about available resources, including highlights about the person and examples of projects they’ve done.
Now, there are limits here. If you’re a smaller company, you may have a harder time getting people to answer your questions; professional organizations will engage you as long as you’re professional and respectful. But don’t waste people’s time by being overly choosy, or by not being prepared to hire a good reviewer. The goal is to establish the best fit for both parties to make sure the project is a success. Rarely are firms going to try to stick you with someone who is not going to be a good fit (unless they have no other choice).
If you’re able to choose someone, look for a person who has assessed projects similar to yours in the past and who has stated interest in topics that are related to your project. If you’re able to have a conversation or communicate with the potential auditor beforehand, ask some questions to make sure you’ve found a good fit. The specific auditor may not always be available to talk to you, since doing so takes time out of their day and comes with context switching costs, but if the project is important then you may be able to request this level of service.
Bear in mind that there are diminishing returns on experience, especially when it comes to simple systems. If your project is similar to many other systems out there and does nothing out of the ordinary, then consider biasing more towards interest rather than extensive experience. An example could be a system that forks an existing well-reviewed protocol or a set of libraries with little added custom business logic. Seeing the 20th version of a similar project may cause a reviewer to be less engaged in the project than someone who has only seen similar projects a handful times.
Beyond what we’ve already discussed, getting an auditor is effectively like hiring a contractor. You can do your best to make sure they’re a good fit and are likely to do a good job, but there is still an element of chance that influences how everything shakes out.
All right, now that you have the best fit you could get, what else can you do? Don’t just hope things are going to work out; prepare.
First, get all your ducks in a row. Clean up any code or systems that you know need attention. Make sure any test environments or instructions on how to run your environment are available and up-to-date. At the end of this process you should be reasonably sure that when the reviewers are going to test your system that they can interact with a running version.
Next, get your code or systems to a place where you’re comfortable having them looked at. Hopefully this doesn’t take very long, but if you’re stressing to get the last feature in for testing, then you’re setting yourself up for pain. Dedicate a week or two to go over your systems, codebase, and documentation to make sure that you have everything in place to start the audit in the best possible conditions.
Lastly, time permitting, take a look at the code or systems and try to spot issues on your own. This doesn’t need to involve conducting a full assessment yourself, but it should involve looking for anything strange or that might have been previously overlooked. Running through a set of tests or even some ad-hoc usage can help resolve potential issues before they happen. Even if what you see during this type of review will not be fixed prior to the audit commencing, knowing about issues earlier gives you more time to fix them fully.
All right, you’re as ready as you’re going to be. Now, you need to make sure all the work you did earlier is communicated to the people doing the assessment. Typically you will do this in a meeting before the assessment starts. Introduce yourself to everyone, let them know your goals and areas of concern, tell them about the system and your documentation, and try to get everyone on the same page. Do your best to walk the auditors through the scenarios your team is worried about along with a ranking of your concerns.
Set expectations for communication during this meeting. Use a medium that makes sense for both parties and that both sides can take prompt action on. Appoint someone from your team to support the effort by answering any questions the testers have. Have a knowledgeable point of contact for the auditors as you’ll no doubt need to address some clarifying questions during the assessment.
Along with the communication strategy, establish a cadence for updates; more frequent may seem better, but this can pose challenges. Many assessors are results-oriented and they can be self-conscious about their work when it’s in progress. Encourage your reviewer to be process-oriented and to provide updates about how things are going along the way. Hopefully the goals you provided make clear that you care less about the specific “results” (or vulnerabilities in the report) and more about fully executing on the goals you provided.
Lastly, take the notes you wrote down at the beginning of the process — including the goals, concerns, and notes from the developers — and send it to them to re-enforce your priorities. Make sure that they acknowledge and understand the priorities. You may feel like you’re being difficult here, but in my experience this is the single most important factor for influencing the quality of the assessment.
Now that the audit has started, make sure your appointee answers questions in a timely manner. This is important, so consider freeing up that person’s schedule. You don’t want your reviewer wasting time spending days wading through low-priority components of the system because they weren’t aware it wasn’t related to the stated goals. They’re brand new to your systems, so they may get lost sometimes.
Read and follow-up on the updates you asked for at the beginning of the assessment. If you’re not reading them, then you’re wasting everyone’s time. If the priorities seem off, ask about them and make sure they meet your expectations.
All of this comes down to making sure the assessment isn’t going off the rails. This doesn’t happen often, but being prepared for the worst case can help avoid bad outcomes. If you see that progress seems to have stalled — and the reviewer can’t tell you what they’ve been doing or what they have left to do — then it’s time to start playing hardball. Escalate the situation to the best of your ability and try to ensure the issue gets resolved. A good firm will step in to resolve the situation promptly, especially if it’s clear things are going awry. The best escalation path tends to involve voicing your concerns to the project or account manager who will then relay them to project leads or management.
If escalations go nowhere and the assessment continues to go off the rails, then there isn’t much more to be done to rescue the project. You’ll probably be writing off the work as insufficient and looking at ways to either shore up the audit with auxiliary activities or find another firm to perform another audit.
Now that the audit is over, hopefully you have a good idea of what was done and a reasonable expectation of what you’re going to see in the report. Once you get the report, read it. Make sure the issues make sense, respond where necessary, and ask questions about anything that you think is odd or incorrect. Do this promptly so the reviewer still has context for what you’re asking. If you ask two weeks later, then they’re probably already in the middle of another project and won’t be able to answer as quickly.
After you have the report sorted, you’ll want to address the issues you’re most concerned about. Do whatever is most pressing immediately so the system can be re-tested sooner. Otherwise, take your time to get most of the issues resolved and then ask for the re-test, which can take anywhere from a day to a couple of weeks depending on the complexity of the testing and availability of the auditor. Once you have the OK that your stuff has been fixed, then you should be good to go.
This isn’t so much part of the process, but it is something to keep in mind. As alluded to earlier, finding all vulnerabilities in a system is an impossible task. Even methods like formal verification can’t make determinations on relatively simple aspects of systems with certainty in a reasonable amount of time. Audits are “best effort,” meaning someone took a look, used the most advanced tools and techniques at their disposal like fuzzing, symbolic execution, and concolic execution, and then gave you their interpretation of the results of that testing. That is as good a “guarantee” as you’re going to get, at least for the time being.
Additionally, assessments only apply for the specific point in time in which they were conducted. If your system is changing rapidly, then that review can become quickly out-of-date. If you need your system to be secure, slow down. You need to stop making so many rapid changes and instead space out releases (or else conscript an army of auditors). Technically you could put an audit firm on retainer, implement robust security practices, or both, but you can expect to lose out on some “big picture” items. If you’re auditing deltas — the differences between two releases — then you’re typically not looking at everything and so you’ll be more likely to miss side effects (as happened with projects like Euler when they got hacked after their code was audited).
Unfortunately, this can feel dissatisfying, especially given the price tag that comes with some of these assessments. (Specialized smart contract audits with two auditors can cost up to almost $100,000 per week.) But that’s how it works. Bearing this in mind, keep those “threat models” up to date and have your own people — those most familiar with the systems — look for issues as well.
No audit is perfect and there will probably still be issues with your system. Given this, consider introducing other mechanisms that can help you get the peace of mind you want. This is where vulnerability disclosure programs like bug bounties come in. By creating a bug bounty, you establish that you’ll immediately pay for vulnerabilities if they’re presented to you. Ideally, this means you’ll receive them before they’re used for an exploit. Programs like HackerOne, BugCrowd, and ImmuneFi can assist in creating and managing these types of bug bounties.
Lastly, create mechanisms for monitoring, detection, and potentially response. Put safeguards in place that can help you catch strange behavior so you at least have a chance to pause or react to hacks rather than just sit and watch your system get exploited into oblivion. If you need assistance creating or managing these types of systems, take a look at organizations that specialize in these types of efforts, like Forta does for onchain visibility. Additionally, organizations like the Security Alliance (SEAL) have created safe harbor agreements that crypto protocols can adopt to allow external actors to help rescue funds on their behalf.
And that’s the full lifecycle. You’ve crafted an assessment that focuses on specific goals and concerns related to your project. You engaged with vendors early to get a better fit for the auditors looking at your project and made it as seamless as possible for them to do their jobs. You understand that at no point are you guaranteed a good outcome, but you’ve done everything available to you to make sure success is more likely.
So you can rest easy (but stay ever vigilant).
***
See more:
by Matt Gleason
***
Matt Gleason is a security engineer for a16z crypto, helping portfolio companies with their application security, incident response, and other audit or security needs. He has conducted audits, and found and helped fix critical vulnerabilities in code prior to project deployment on many different projects.
***
Editor: Robert Hackett
***
The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the current or enduring accuracy of the information or its appropriateness for a given situation. In addition, this content may include third-party advertisements; a16z has not reviewed such advertisements and does not endorse any advertising content contained therein.
This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets) is available at https://a16z.com/investments/.
The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information.
