Wasting Money on Penetration Tests

Penetration testing may be one of the worst things you can allocate funds for in a cyber security budget, and this is coming from someone who has worked as a professional penetration tester for over a decade. Before you blow up my comments with invectives designed for shock and awe, let me explain and maybe, just maybe, you will agree.

I did a quick Google search for “cost of a penetration test” and found results ranging from around $2,500 to over $100,000 depending on several factors. The cost of the test, however, is not the problem. The problem is how these tests are positioned. This is the text from one web site that showed up early in the search results.

“A pen test can measure your system’s strengths and weaknesses in a controlled environment before you have to pay the cost of an extremely damaging data breach. In IBM’s 2019 Cost of a Data Breach Study, they indicate that the average cost of a data breach is 3.92 million dollars with an average loss of 25,575 records. This might be basic math, but 3.92 million dollars spent repairing losses from a data breach is a lot more than the average $10,000-$30,000 bill from a professional, rigorous pen test.”

I did not link to the site because my intent is not to call out any given organization. Multiple sites contained similar concepts and I want to highlight the core problems. The above quote clearly implies that if you do “measure your system’s strengths and weaknesses in a controlled environment” you won’t need to “pay the cost of an extremely damaging data breach.” It also implies that if you spend $10,000-$30,000 on a professional, rigorous pen test, you can avoid $3.92 million in average breach. Spending $30,000 to generate $3.92 million is savings is an annual return on investment of almost 13,000% and that assumes a single breach. It is possible that you suffer more than one breach increasing that ROI to astronomical levels. Unfortunately, these metrics are largely “smoke and mirrors”. They are a subtly crafted message that, if believed, is a lie.

At this point, you may be thinking about this from a statistical perspective. The average breach cost is just that, an average. There is no guarantee that you will suffer a breach. The specifics of your organization and the specifics of a breach mean that your cost could differ wildly from the stated averages. While these factors are true, they are not the real problem. The real problem is the implication that a penetration test will prevent a data breach. This is not only untrue, it is completely impossible.

The next time you plan on performing a penetration test, ask the tester if they are willing to guarantee that you will not suffer a breach, assuming you implement 100% of their recommendations. Their answer will be, without question, no. No ethical penetration tester would make such a guarantee for a few reasons:

• You might implement the recommendations incorrectly

• The penetration testers may not have found every vulnerability

• New vulnerabilities affecting your environment could be discovered

• Users could take action harmful to the environment

• New technology could be added to the environment

The fact is that organizations are in a constant state of flux. Business or mission requirements change requiring changes in personal and in technology. These changes impact not only the presence or lack of vulnerabilities, but also the ability of the organization to detect and respond to threats. The state of flux is not just internal. New vulnerabilities are discovered on a regular basis, and threats change and emerge. Your organization could be completely vulnerability free one day and wide open to compromise the next.

Penetration testing costs anywhere from a few thousand dollars to $100,000 depending on the size and complexity of the target organization and the scope of the penetration test itself and, as when purchasing a new car, the value of the test begins to depreciate the instant it is completed. Let’s assume you contracted with a mythical world’s best penetration tester. They are virtually omniscient when it comes to available attack techniques and exploits, and virtually omnipotent when it comes to executing their attacks. They can discover and exploit every vulnerability in your environment perfectly. Let’s also assume that the scope of the penetration test allowed the perfect pen tester to operate in any way necessary to execute their attack. Finally, let us assume that the penetration tester can provide actionable recommendations that will, if implemented, eliminate every discovered vulnerability. At the end of the penetration test, you will have a 100% complete understanding of your vulnerabilities and know exactly what remediation needs to be done. You perfectly implement all the remediation steps and correct every vulnerability. Is your environment now secure? Absolutely not. Why? Because at any time after the penetration tester completes their testing, a new, exploitable vulnerability affecting critical assets could be discovered.

The previous scenario assumes a “perfect” penetration test executed by a “perfect” penetration tester but in reality, there is no “perfect”. Penetration testers are human, each with their own strengths, weaknesses, skills, and deficiencies. Furthermore, even the best penetration tester can have a bad day, forget something, or spend too much time going down the wrong path during a test. Therefore, all you know at the end of the test is what that specific tester found on that specific day. A different penetration tester might have discovered an entirely different set of issues. Worse, if the tester finds no significant problems, does that mean you, in fact, have no exploitable vulnerabilities or that your specific tester found none. When you combine the fact that the value of a perfect penetration test begins to decrease the second the tester completes the test, and that there is no such thing as a perfect test, regular penetration testing makes little sense even though it is one of the most popular areas of focus in cyber security.

So, what role does penetration testing play in today’s computing environment? First and foremost, it may be required to comply is various industry standards and/or regulations. Healthcare environments (HIPAA), retail (PCI DSS), SOC 2 testing, and financial institutions (FINRA) all require some type of penetration testing to be performed. Similarly, industry standards such as ISO27000 and the Critical Security Controls also require penetration testing. If your goal is to comply with a standard or regulation and that, in turn, requires penetration testing, then penetration testing provides value.

Penetration testing (and/or vulnerability scanning) can also be useful for identifying “low hanging fruit” or the obvious vulnerabilities that can be quickly and easily identified and quickly and easily corrected. If penetration is viewed as a method to identify areas where improvement can be made, and not as validation of existing security controls, it can also be very effective. The key, in this case, is to make sure the cost of the testing is commensurate with the value received. At the end of the penetration test, you will have a list of some of the exploitable vulnerabilities present in your environment along with an understanding of what could happen if those vulnerabilities are exploited. There is value in that and, as long as the cost of the test is in line with that value, it is a worthwhile activity.

Penetration testing can be useful for identifying vulnerabilities in a specific piece of technology. This could include testing new applications (e.g., web app, mobile app, or similar) or a recent implementation of security controls (e.g., a new firewall, IDS, EDR, etc.). This is narrowly focused testing designed to determine if there are any unknown security issues with the tested technology. The advantage of this type of testing, as opposed to broadly targeted testing, is that it allows for better vetting of the tester resulting in more predictable results. A broadly targeted test may include web applications, wireless, Windows, Linux, Mac OS X, social engineering, APIs, cloud environments, and even zero-day vulnerability discovery and exploitation. Few, if any, penetration testers are equally skilled in all these areas. Finding a penetration tester who specializes in network attacks, or evaluation web applications is much easier.

Lastly, penetration testing can serve the function of a checkup performed by a medical professional. When getting a checkup, the medical professional takes a variety of measurements such as your height, weight, temperature, and blood pressure. They collect blood samples, examine your reflexes, look in your eyes, ears, and throat. While not testing for any specific conditions, the checkup can identify indicators that more significant problems may exist, and more detailed testing can be performed. Many penetration tests focus on discovered vulnerabilities and the subsequent impact of a breach. A skilled penetration tester can, however, add significant value by attempting to understand the root cause of the identified problems. By noticing that the majority of vulnerabilities are the result of patches missing from non-Microsoft software installed on Windows computers, the penetration tester can provide guidance on how that specific aspect of an organizations patch management process can be improved. By focusing on improving root, operational causes of security problems, organizations can meaningfully improve their security capabilities over a prolonged period of time.

Having outlined the weaknesses of penetration tests, and then outlined the value they can play despite those weaknesses, why did I start out by saying that penetration testing may be one of the worst things on which to allocated cybersecurity budget? It boils down to opportunity cost. I have personally done penetration tests for organizations who have no vulnerability scanning capabilities, no hardening standards, virtually no monitoring capabilities, and no real incident response capabilities. The $10,000 (or more) spend on a penetration test could have been used to:

• Acquire a vulnerability scanning tool and implement regular internal and external scans

• Implement central logging with the ElasticStack, Wazuh, or similar

• Provided internal security personnel with training to increase organizational capabilities

• Generate high fidelity alerts using network modifications or deception technologyDrafted a detailed incident response plan

I do understand that for a given organization, $10,000 may not be sufficient to achieve any of the above goals. That said, the costs of a penetration test tend to be proportionate to the size and complexity of the environment. An organization for which a penetration test would cost $10,000 would likely be able to implement at least one of the above for that same $10,000. An organization for which $10,000 would be insufficient to implement at least one of the above would likely incur penetration testing costs that exceed $10,000. The focus, however, should not be on the exact cost of a penetration test, nor should it be on what specific alternatives can be accomplished for the same price. Instead, organizations should focus on answering the following questions:

• Am I compelled by regulation or other compliance requirement to conduct a penetration test?

• If not, how much will the test cost and what is the lifespan of the value of that test?

• Are there other things that would provide increased value or value that persists for a longer duration?

A penetration test starts losing value immediately after it is completed and has little to no value after a matter of months. Increasing the capabilities of your personnel or expanding the capabilities of your security infrastructure will typically increase in value over time. It is not whether penetration testing has value. Instead, it is matter of whether that value should be prioritized over other security measures. In most cases, there are better ways to spend your money.