Getting the Best Value From Penetration Testing

Penetration testing is a critical cybersecurity tool for any organization that has an Internet accessible component to their business. Spoiler alert! That's every business! A good pentest can tell you about the technical deficiencies that make you vulnerable. A bad pentest can lie to you and make you believe you're more secure than you really are. Overconfidence is a serious issue in cybersecurity, and a penetration test is a great opportunity to ensure you don't suffer from that issue. During my career I've helped hundreds of customers scope their penetration tests, and guided (most) of them to getting the testing that supports their maturity, needs, and wants. The rest of this document will be focused on helping you do that for yourself, and avoiding three key mistakes that can lead to a poor penetration test experience.

Begin With the End in Mind - Outcomes and Results

What are you looking for this penetration test to do? After all, these things are not inexpensive (if they're any good anyway), and companies don't generally like spending money on things just for the hell of it. Understanding what you need from a pentest is going to have a lot of impact on what your pentest should look like. This will help determine what actions the tester should take, what systems are in and out of scope, and even what sort of reporting you want. Below are some typical examples of why companies perform pentests.

Compliance and Reporting

As I've said before, compliance is the reason to spend the least possible amount of time, effort, and money on a task. However, compliance and reporting based pentests generally have well defined scopes. Either the compliance framework or the compliance auditor should be able to help you understand exactly what needs to be tested, and exactly what reporting/evidence needs to be created and collected for that. That said, if you're unsure of your scope meeting the compliance requirements, you should really consult experts on that compliance requirement to identify them.

Generally compliance based penetration testing is focused on "making you look good." A controversial statement to be sure, but at the end of the day you're paying for testing that you can pass, which means the testers should be offering you a "pretest" and a "final test" option, giving you the chance to address findings before final testing completes and final deliverables are generated. This assumes, of course, that you have the capacity to resolve such issues in an appropriate way within the time-frame available. If you can't please remember that isn't the tester's fault.

Planning Program Improvements

Pentesting can be a fantastic tool to use prior to budgeting time. If you are already aware of deficiencies in your security program, pentest results can help create the "objective, third party validation" of your concerns that support your buying plans. Pentesters love knowing that you want special attention paid to, and they're happy to comply. Given that many pentesting organizations are also technology integrators and resellers, you can often ask them to work with their peers to provide some improvement recommendations to go along with their finding reports. (Please make them aware of this during the sales process, don't spring it on them after)

That said, you need to be very prescriptive in your scope for these sorts of tests. If you want to focus on a few particular areas that need improvement you don't want to be surprised by findings that identify some other part of the program as being in more dire need of improvement. There are two primary ways of avoiding this:

1. Have a very mature program with regard to those other areas, and let the pentest confirm that.
2. Focus the scope and activities in the pentest to avoid other parts of your program and just focus on your priorities.

If the second option smells bad to you, I understand. It isn't necessarily what I'd recommend, but there are times when it may be appropriate and every organization's situation is different. (Think M&A for example - you may only have the time and budget to focus on that new company you just merged with) There is always a compromise however, to perform some base level of testing across the entire environment and focus more strenuous testing against your priorities.

Overall Discovery

This was always my favorite pentest focus to help customers with. This is the test whose outcome is most focused on answering the question "how technically secure is my environment against a variety of attack vectors." These were the companies who were really focused on overall improvement and on bringing their entire environment up to scratch. Sometimes that would mean we just did basic work across the widest possible scope, other times that meant we threw everything including the kitchen sink at them - budget and maturity were generally the two factors that impacted those decisions. "Inch deep and mile wide" output was often the output, and from that security leadership could plan improvements and shift focus as needed.

Three Common Mistakes

The scoping conversations that concerned me were the ones with a customer who didn't really know what they needed from a pentest, or who had a misunderstanding of what a pentest could do for them. So I offer the following three examples to you of common mistakes I saw customers make when defining their pentests in hopes that you won't make those yourselves.

Underscoping

This covers a lot of ground, but I can tell you that every time someone said "that is out of scope" as we were working with them to define their test I cringed a bit - unless they had already shared with me their focus and rationale for doing so.

Underscoping is absolutely devastating for compliance focused penetration tests. For PCI for example, I would always recommend to the customer that they consult with their QSA before we finalized the scope of the test. I can't tell you how many customers came back to us upset because while their test validated all the right security controls were in place, it didn't validate the entire scope of the environment.

Customers who wanted to skip computers because they were "sensitive" and might hang or lock-up during a penetration test was always a scope warning to me as well. If you have such systems in your environment they have no business running business-necessary applications.

Finally, customers who said "don't include test or dev" always made me cringe.

Overconfidence

Overconfidence usually came from the buyer who didn't want us mucking about with all that pedestrian stuff like vulnerability scanning and the like, and just wanted a "capture the flag" style penetration test, where we all agreed on a target (usually domain admin) and we had a set number of hours to achieve that. I would try my hardest to talk the buyer out of that - capture the flag was best for extremely mature environments who really had their entire program in order. This was my number-one customer mistake for re-scoping and change orders.

Invariably what would happen is that we'd agree on 40 or 80 person-hours of work to capture the flag + a fixed-fee for the reporting. Within the first 8 hours of effort the flag would have been captured by our pentesters. (Sometimes it happened during the kickoff call to introduce the customer to the testers) Then a conversation like this would ensure:

Customer: "That's it? But what about testing x, y, and z? You aren't telling me anything about my controls around w! This test doesn't tell me anything!"

Us: "Yes, we did try to warn you about that. How about we do a change order and use the remaining 78 hours to do a, b, and c? And if you want us to really cover all the bases, we can re-scope for all the other things you just asked about too."

Customer: "<grudging agreement>"

Secrecy

I'm not talking about trying to make pentesting "legal work product" by CC'ing outside counsel on every conversation¹, I'm referring to keeping key information from people who need to know it during a penetration test.

The first secrecy mistake I saw was demanding that penetration testers have limited knowledge about the environment - sometimes referred to as a "black box" testing methodology. That meant that the tester had to waste valuable hours doing reconnaissance on the environment, and was testing "blind." While this sounds "realistic," I always recommend giving your tester all the data they ask for. This means they'll be able to spend more time and effort on testing your controls and systems and less time on less valuable activities. Further, it is productive to assume the "worst case" scenario that your real attacker already knows all of this information.

The second secrecy mistake I saw was keeping the SOC out of the loop in an effort to "test their responses." It can be valuable to perform a "ring the bell" test against an uninformed SOC, absolutely, but for the overall test it is far more valuable to cooperate with your tester and your defenders. It is all but impossible to hand your SOC team a report that says "on one afternoon 3 weeks ago the tester did this" and expect them to be able to go back and identify what failed in the monitoring. Working together in a "can you see when I do this" way is much more fruitful.

Putting a Bow on Pentesting

As with nearly everything else, what planning and effort you put into choosing the right pentesting will come back to you in results. Be sure you know what you want and need from a pentest, and don't fall into the trap of common mistakes. Work with your testing partner to pick testing that is as useful it can be within your budget, and heed their input in the scoping process. You'll thank yourself when you get the final report.

^{1 IANAL, but I really have a hard time believing that actually provides any meaningful value/protection.}