Web site scanning without prior permission is very likely illegal, here’s why

It appears that there is a growing wave of SaaS utilities that will either scan websites and internet-based services for you or provide you with access to historical information on sites and services they have already scanned, often for a fee. Unfortunately, using such services can result in your seriously falling foul of the law in a major way, especially if permission has not been obtained by either the company performing the scans or the client who asks them to perform a scan on their behalf. Let me explain below.

Ask for permission first, always…

Simply put, regardless of the safety or intent of the scan being performed; you need to get permission prior. This is for several reasons:

#1 There is no such thing as a Passive Scan

Quite often such companies will say they are only doing a passive scan and that it cannot possibly cause any harm to the service being scanned. Wrong! The act of scanning itself is an information exchange between two computer programs, both of which know nothing about each other when they start communication. This means that if there are bugs or bad code in either party, this could result in the scanned service either having its data integrity impacted at the least or at the worst, the availability of the service could be impacted. Also, Passive Scanning is not an exactly defined technique, where is the line between what is truly Passive or Active? We all know the extremes, but where is the line where you cross over from Passive to Active? Remember everything is an interaction.

If you have no awareness whatsoever of the implementation of what you are scanning, how can you have 100% confidence you will not be negatively impacting them?

#2 You are consuming someone else’s resources

Such scans can range from something light to something intense, either way, your act of scanning somebody else’s service is consuming their resources without prior consent. What if that scan was to trigger an autoscale event or cause a database to be upscaled? This is a serious cost to those running the service and it’s to no benefit of the service operator.

Plus if you take this type of service to the limit and assume there will be many providers of such scanning facilities, it will end up with internet services being scanned all the time by multiple scanners – which is a serious waste of resources that will probably have a negative impact on the environment as a result.

#3 Who is using the results of the scan?

Now you may think it’s only you as the customer who has access to the reports, but who else has access to such reports? How is it guaranteed that a bad actor does not gain access to such reports and use the treasure trove of security information to their benefit? In effect, such scanning could be weakening the security posture of the very sites and services they are trying to protect!

#4 Who are the bad guys?

If you are running an online site or service and you have all these random scans against you, how do you know who is the bad actor in all of this? Plus it’s likely the scans will trigger security alerts, if only for the ‘sniffing’ and oddly timed traffic, this again is consuming resources and in this case consuming human resources to boot!

#5 Where is the data being stored?

If the scans record fragments from the websites or provide snippets from what the services returned, there is a risk that PII might be revealed or otherwise sensitive information. This could fall foul of privacy regulations in a big way. This also raises the question of whether this vast treasure trove of information is being stored securely or not.

The legals…

In most countries (including, the USA, UK, AU, etc), there is specific legislation to make it illegal to scan or probe online services to discover security vulnerabilities without prior permission. This is done for good reason, as it provides a clear separation between those obtaining such information for bad reasons and those doing it for good. The intent is not that strongly valued, if someone has not given permission, you are aware of this lack of permission, and you are scanning regardless; that is considered illegal.

  • For instance in Australia, you have a complex interplay between Federal Legislation (see here, which covers everything from Hacking through to malware and the sale of hardware, software, or other tools that can be used to commit cybercrime) and State legislation (see here, for instance, Western Australia has ‘unlawful operation of a computer system’, plus has a discussion of what ‘unauthorized’ really means and the interplay with policy awareness, notification and enforcement).

Now you may say that Google and all the other search engines have been doing this for years, so what’s the problem? The thing is they are enacting those sites for the primary purpose for which they were intended. They do not disclose to all parties any security issues they have found indexing the sites. Likewise, the usage of standalone security scanners (like those in some browsers or security toolkits) is an individual action taken by a person, they have direct legal responsibility. Plus Google and other search engines have established mechanisms in place to disable scanning of sites that don’t want to be scanned, they can decline in advance.

Whereas if you are a business scanning sites looking for security vulnerabilities, without a client’s indication of permission to do so, you are harvesting (illegally) a treasure trove of information that could be used by bad actors. In several legislations, this could be considered possession or use of services that are knowingly aiding in the committing of cybercrime (and if you operate such services by reading this article, you are now fully aware that this could well be going on). What have those running such services done to ‘vet’ and screen those accessing such report data sets?

Sometimes to get a report, you just reach out to one of their sales staff and they will happily hand it over, no questions asked. What does this really say about their approach to security and hence the value of what they are selling?

This is why those who perform Penetration tests obtain what is known as a ‘get out of jail free’ letter, to show to the authorities they have got prior permission to scan. The fact they are doing a possibly invasive scan is NOT the reason for this, it’s the fact they are doing a scan AT ALL is the reason.

Also, at this point I usually check their Terms and Conditions for the service as provided, often you will find the standard provided ‘as is’ without warranty or fit-for-purpose disclaimers and blanket limitations of liability capped to a few hundred dollars if you are lucky. Now, you may think this is something everybody does, and most businesses do this, but if you are going to base your security on such reports, wouldn’t you want to know the business supplying them puts their money where their mouth is? Such businesses should buy insurance and have terms that allow you to at least recover the fees you paid them if their reports cause you harm – I think that’s fair and a good sign of a basic commitment to the quality and integrity of their product.

The false attraction of a vendor score

Something else I want to touch on is that a lot of these tools take pride in displaying a global score of security fitness, often displayed as a grade (A to F typically). This is complete security theatre due to 4 things:

  1. The scan performed to derive this score is usually based upon a single web request and its analysis, plus an analysis of other publicly available data sets (be that DNS, whois records, etc). The single web request is the weak link in this, one page does not make a security assessment. There could be a serious security weakness on another page they never visit, but a hacker would easily find (take for instance the recent Optus breach).
  2. As mentioned, the scan cannot ‘go deep’ into the site and go beyond login forms. So the vast majority of the service is beyond being assessed.
  3. The scanner, given the previous point, therefore has no understanding of how the various things it has scanned relate to each other in a security sense. For instance a static marketing site, that has no dependency on the live product, would be ‘bulked in’ together to produce a total score…
  4. The scoring is based on the risk as perceived by the scan developers, it is not based on your risk exposure as a customer of the vendor being scanned. This may be completely out of alignment.

So, I would be taking the results and scoring of such scans with a planet-sized pinch of salt and if you are purely depending on this scoring to assess vendors, you are doing it wrong in my book.


The only right and safe way to do this

First off, the results of such scans and reports should not be made freely available to anyone who wants to know, which I would consider seriously irresponsible and reckless at best. It’s a bit like running a free candy store for hackers if you make this information public. Now the argument may be that everybody else is doing this, therefore it’s okay; I’ve got news for you, that will not stand in court. Ignorance of the law is never a defense.

Secondly, all clients must sign a waiver and agreement that they have the authority to utilize such scans and reports in a way that does not negatively impact any sites or services so scanned, and that they have sought prior approval from the site or service operators to have such scans performed.

Thirdly, site or service operators and owners need a way to either ban such 3rd party scanning or have an approval mechanism in place to give consent to allow each scan to occur. The wholesale security scanning of sites or services for security weaknesses must stop.

Fourthly, site or service operators should update their terms of service to specifically exclude the security scanning of websites or services without prior written permission. This clearly puts those doing so without prior permission on the wrong side of the law. Sites can also signal to scan services what they allow and when using the robots.txt extension here.

This is the only way to do this, prior permission and agreement to be scanned MUST be a critical gate in the process.

The lack of Security Significance of such scans

Something else to keep in mind when using such scanning services for websites and vendors is they are literally basing their analysis and scoring on just what they can see. For most online services, the vast majority of the data and systems are behind some form of login system, which such scanners cannot go beyond. It is quite likely the security controls are more focussed around the data and systems ‘within’ the online service being offered than outside of it – this is a very pragmatic way to approach security, in that you are focussing on the things of value first.

It could also be what is outside forms some part of a honeypot (a security-weakened service used to detect and trap hackers), so such a scanning service would be negatively scoring a business that is being proactive in trapping hackers…

Another aspect overlooked by such scans is they really say nothing about the security posture, professionalism, training, policies, and procedures enacted by a business. You will only find this out by talking to the business directly, who will probably then provide you with summary results for their own penetration tests across the whole service as offered, both inside and out, so why bother with such pointless scans in the first place? As always it comes down to people and how security is enacted by those people within the business in question.

Conclusion

What is currently occurring is the online equivalent of going around checking everyone’s homes to see if the doors or windows are locked, then making such information available to anyone willing to pay, or even for nothing. In the real world, you would get arrested for doing this, and with good reason. The penny needs to drop that the same rules & consequences also apply online.

Those providing online security scanning services need to grow up, it’s not a free-for-all and it has never been a free-for-all, there are consequences and responsibilities to providing such services, which need to be enacted. Those businesses that respect the law and its intent will rise to the top and those that don’t will sink. The other thing they need to understand is that their actions and the position they take sets the legal response and its evolution and interpretation over time.

There is also the question of the true real value of such reports, as unlike a proper penetration test or ISO27001 standard, they by their very nature can only look at the surface of the website or service and anybody who has implemented websites at scale knows that is often the tip of a very big iceberg. I also get the sense, from the reports I’ve seen, that they are mostly ‘security theatre’, in other words, they overplay risks and possible impacts to make people feel they have derived value from such reports. They can also give a false impression that what they have scanned is meaningfully secure (or not); to use the physical analogy, not all locks and alarms are created equal. I feel such reports offer a very weak evaluation of overall security, especially when compared to a comprehensive Pen Test, they pale into insignificance.

Remember, these scanning services can cost up to several thousand dollars a month, what are you really paying for and does it actually move the needle in a useful way on your security position? Would this money be better spent on training and well-targeted tools and utilities instead?