Between various speaking gigs, running online events, writing for the Circuit, work, and some side projects, it’s been a while since I posted. Given that, I want to reactivate this with something that might be controversial and talk about pen testing.
Pen testing, or penetration testing, is the (usually) manual assessment of a system by experts in technical security. It can encompass, or be part of, red teaming exercises where physical vectors are also used, phishing testing, and similar activities, but the USP is usually that it’s done manually by experts (for a given definition of manually – any commercial pen tester who doesn’t script 90% of what they do is doing it wrong and wasting their time and your money).
There’s no denying the value of a well-scoped pen test performed at the right time in a product lifecycle. In the perfect scenario this pen test is comprehensive, with enough time to thoroughly delve into the system and provide proof of exploitation for all discoveries. Mostly, there isn’t enough time given for a fully comprehensive and thorough test, but there’s still value in this kind of deep delve.
Even if the test is short, and poorly scoped, it’s better than nothing so long as it isn’t taken to give a false sense of security around a product.
The findings of a good pen test can provide an organisation with a final check before going live to decide whether to accept, mitigate, or transfer any risks which are made visible. What a pen test cannot provide is assurance that the outcome of a project is secure, any more than any other testing. A pen test with no findings is like a house that hasn’t burned down – it’s no reason to skip having a fire alarm.
The pen test should be the last step of your security testing stack. It’s the most effortful, and most expensive, form of testing you can undergo and so wasting it on false positives and easy-to-fix vulnerabilities is not a good way to spend your money. Ideally you should pen test when you have a production-ready candidate, in parallel with your last round of release testing.
On top of that findings picked up by a pen test will be far, far more expensive to fix (both in terms of effort, delay, and financial outlay), so you want to minimise the chances of things being picked up this late. Earlier development lifecycle stages should progress from concept testing (threat modelling is great for this, and making sure requirements incorporate security), code analysis tools and secure coding practices, vulnerability scanning, fuzz testing, and/or any other testing that fits your model.
Where is it going?
This is the bit that might be controversial, and I want to be absolutely clear that I am in no way denigrating the value of pen testing, or the expertise of pen testers. This is speculation based on what I’ve seen in the security industry, and what I’m expecting to come up.
Pen testing benefits hugely from being the sexy, hyped-up side of security. Red teaming and pen testing tracks get far, far more attention and excitement than the more reactive incident response side, and both get more hype than the fully defensive planning and design stages.
I can understand this. I’m a big fan of heist movies myself, when I want to relax and do something professional I’d rather do a bit of CTF or play around with something from vulnhub than review architecture diagrams and project requirements, and I know from experience that I can get far more attendees and interest at a workshop or presentation on lockpicking than I can one on threat modelling (caveat: based on personal experience and anecdotal evidence rather than comprehensive data collection).
This is a problem. The attacking side may be the sexy side, but it only has limited effectiveness when you’re trying to defend a system. Pen testing done well is a simulated attack, and a simulated attack is not a comprehensive, long-term, defensive strategy. A pen test is, at its basic level, running through a series of tactics and seeing if they work. The reason a lot of it can be automated is because the vast majority of pen test findings are known vulnerabilities already, and the pen test is just discovering them and showing they can be exploited.
So I genuinely believe that as (or if) we mature the blue side more, pen testing is going to become both less important, and much, much more frustrating as a career. In fact security overall is (hopefully) going to become less of a thrill-a-minute experience if we do it right. In my ideal world, pen tests will become more and more a case of continuous security research into discovering new attack vectors and vulnerabilities, and pen testing for existing vulnerabilities will disappear.
That’s a utopian world and will never be fully achieved, but the automation of pen testing as we mostly know it today is here and that poses a threat to pen testing companies. While it’s an uncomfortable truth for both attack and defense sides, penetration tests are mostly compliance driven rather than arranged out of a genuine awareness of security needs. With the growth of automated testing, I am reasonably certain that it won’t be long before automated testing platforms and frameworks are accepted for compliance purposes.
Forward-thinking pen testing companies should doing two things if they want to survive and thrive. Preparing to shift into a different way of working (and honestly, a more interesting one than running through a list of the OWASP Top 10 for a few days – this is hyperbole but does have a grain of truth) is the first. Educating customers that they can add value beyond achieving ISO27k, PCI-DSS, SOC2, CBEST, etc is the second.