It seems quite a few businesses are resorting to using 3rd parties to implement continuous pen testing for not only their own products but also for online services they also consume – this can be a very bad idea and lead to a false sense of security.

In the face of it regularly scanning an online product for vulnerabilities sounds like a good idea, it gives you some assurance that the product is secure. A bit like having a security guard periodically going around the outside of a property checking the locks. Although its a bit of a false sense of security, as continuous penetration tests are not without their flaws, namely:

• They can only test what they can ‘see’, this doesn’t mean to say that there aren’t vulnerabilities in locations or places they cannot get to or ‘enact’ – often such continuous pen tests do a ‘lite’ scan and do not go deep into a site.
• They can only test what they have been coded to test for – and given the above point – this will be a very limited set of tests compared to a full pen test. So the ‘confidence’ you can have in such a scan discovering a serious fault is actually quite low (nobody scanned for the Php Zero-Day until it was discovered..).

To reuse the Security Guard analogy – they will be unable to check how good the locks actually are and will only quickly scan for things of interest to them.

There is also the not inconsiderable issue of performing a Pen Test against what is a production instance of service – how do you know 100% that such a scan will not impact the Quality of Service of the product under test? There is also the capacity problem to think of; if all the SaaS providers customers did this, when would the product actually experience a normal load? Pen Tests can quite easily generate 1000’s of requests a minute, you don’t need many of those overlapping to cause problems. Cloud providers have been known to disable product provider accounts due to someone running an unauthorized Pen Test, thinking it was an actual attack – in effect the SaaS product goes down for everyone. Not good for the SaaS product provider and certainly not good for those running the unauthorised Pen Test (this can be considered legally a denial of service attack).

This also leads onto a larger point, and that’s one of maintaining a signal to noise ratio in the security analysis of the Production instances – if the logs are full of scans from Continuous Pen Tests – how are the security team meant to focus on the real bad guys – their trail will be lost in all the noise… So Continuous Pen Testing could have a real negative impact on the security of the product!

There is also the fact that SaaS products do not operate ‘alone’, they pull in resources from all over the Internet to display the product you see, this could include libraries from Google, Facebook, Twitter, etc plus a whole load of other libraries from open source projects – a Pen Test is never going to be able to fully assess the security of such 3rd party dependencies, that is down to the quality of the internal development process and associated oversight within the Service provider itself.

Is such testing worth it?

Given how interconnected online services are now, even a good Pen Test will only ever give you a fractional degree of coverage of the total surface area of a complex system. Remember most online services not only offer web-based applications but provide:

• API’s and SDK’s to tie into all sorts of systems and services.
• Integrations out the box with other cloud services, for instance, payment processing, email, calendar, chat, video, etc.
• File upload and file download services.
• Application plug-ins and extensions you buy from their own online store – which can run easily into the 100’s if not 1000’s.
• Mobile and tablet-based applications.
• Different tiers of service offering, with their own security controls.

Of which the web application is just one small part, an important part none the less, but just one part. Nearly everything else is well beyond the reach of the Pen Test to access.

So what should I do?

if you still want to do your own Pen Test, you want to be running your Pen Tests against an instance that is NOT the production service but equivalent to – this way the production instance traffic is kept clean of Pen Test traffic and hence will not be impacted by it. Also, the Cloud Provider will see this and not overreact.

Also remember that external Pen Tests, if the service provider is any good, will be more a test of their WAF (Web Application Firewall) then the actual core application itself. Only the Service Provider can test the core application directly without the firewall. Which they should be doing, as this provides Defense in Depth confidence that multiple layers in the service are known to have strong security qualities.

Also, given how ‘lite’ such continuous scans are, I would check if the SaaS provider does their own regular in house scanning – many do, and a lot of cloud service platform providers (like AWS and Azure) have a lot of smart monitoring and shared SOC services that can be utilized. Well gone are the days when boxes ran ‘dark’ and it was hard to monitor properly. Check instead for the proper use of in platform security tooling and that they have an in-house security team and appropriate certifications. At the end of the day, security and its maintenance is still very much a human driven endeavour.