Online Security Questionnaires – do they work?

When tendering for a third-party service it’s usual to send the service provider a security questionnaire to ascertain their security ‘stance’ and if their policies, procedures and controls meet what’s required, in a security sense, for the services on offer and the data being processed. You want to know if their approach to security is fit for purpose.

In the past the security questionnaire was a short document or spreadsheet which asked a number of key security questions related to the service under consideration. Of late, with the proliferation of online providers that offer ‘questionnaires for rent’ services, online security questionnaires contain hundreds of questions. So, what used to take an hour or two to answer instead has turned into a marathon session often running into days and involving many people.

Security Questionnaire Blinkers…

Now I understand that an increased focus on obtaining ‘proof’ and ‘evidence’ of security is driving this, but I would like to put forward that it’s a rather ineffective way of assessing the true security stance of an organisation. Think of it like assessing a future life partner by giving them a 200-page questionnaire and basing your decision to commit purely on what is put in that questionnaire alone? People have been known to exaggerate or put their answers in a favourable light, especially when a positive outcome is determined by the answers, that’s just human nature, nobody wants to document their faults if given a choice.

Having been on the receiving end of dealing with these online security questionnaires, it’s clear that those who write them often have little real understanding of the interplay between security policies, procedures, certifications, people and culture. They focus purely on what is documented or enacted in a very ‘tick box’ mindset, which makes no sense, as security is implemented by people acting together. In other words, you can have plenty of policies and procedures, but if the business is not invested in implementing and enacting them, both in intent and spirit, they are meaningless.

Insecure Security Questionnaires…

There is also the question of the security of the online security questionnaire service, both from the prospective of the customer and the service provider completing it. The service provider has to answer highly sensitive security questions that relate to all sorts of security controls and policies, and often documented evidence is required. These documents provide internally sensitive information extremely useful to a hacker (or has immense value to a competitor). So, by requesting a service provider to offer up such sensitive evidence you are putting them at real risk of a data breach if the security of the questionnaire service provider is an unknown..

The risk of the online security questionnaire provider suffering a breach will reflect back on their customer in terms of the risk that the service provider was subjected to. Is there a legal risk here due to the service provider having no option other than to fill in the security questionnaire?

Illogical Security Questionnaires…

Another area where these online questionnaires fail is the weak logic used in working out whether a particular subject or area has already been covered in sufficient depth. You encounter the same sort of question asked from a slightly different perspective, some language is imprecise and confusing and it’s a recipe for inconsistent answers that then defeats the whole purpose of the security questionnaire.

There also is a certain amount of ‘pre-cloud’ cut and pasting going on – a question that made perfect sense in the on-premise world gets mangled to see if it applies to cloud-based services. Almost as if the author is unwilling to let a good question go to waste, not understanding that it no longer makes sense. I also get a strong impression that what were several distinct questionnaires get ‘merged’ into one behemoth in an attempt to cover all situations (and maybe reduce a subscription fee as well). I’ve lost count of the number illogical questions that result from this approach. Some questions require you to be a mind reader to work out what they are really after.

I also suspect there is a form of ‘cold-war-race’ going on between the various online security questionnaire providers as to whom can provide the most complex and all-encompassing security questionnaire – almost as if the sheer size and weight of the questionnaire has a bearing on how good it is and therefore justifies its cost. This couldn’t be further from the truth, given the above. Long, complex and illogical questionnaires suffer from diminishing returns like everything else, and we are well beyond the sweet spot.

Also, the online security questionnaire providers seem to have little understanding of how security certifications interplay with security risks and prove covering security controls and procedures. I think only one questionnaire out of the many hundreds I’ve seen has resulted in the collapse of the majority of questions once evidence of ISO 27001:2013 certification was provided.

What to do?

There’s a lot wrong with the usage of online security questionnaires, that is clear! The process has degraded into something almost farcical, if it wasn’t for the seriousness of what’s trying to be achieved. Although I do have a few suggestions as to what can be done to make them work better and actually get a true understanding on how secure the service provider really is.

  • If you have a questionnaire that consists of more than 100 questions – you are doing it wrong! Beyond that number and the questionnaire becomes a true beast to manage, especially given the need to consult with others to find answers and ensure coverage. Also remember if you are reviewing a SaaS application, this will only be at best a moment in time snapshot of their security posture, applications often undergo constant improvement and so does the IaaS environments they are hosted on. Things move quickly in the cloud.
  • Give security certifications their due. Businesses that have and maintain security certifications like ISO 27001:2013 need to be acknowledged as such and not put through such torture. Reward security professionalism!
  • If the service provider is going to be offering an important or sensitive service for you – I suggest you actually have a meeting with their in-house security team. At the end of the day security is implemented and operated by people, so how can you expect to assess security without seeing who is enacting it? This also means you have a business relationship with those providing security in the provider, which can be very helpful for both parties.
  • Assess the security of the online security questionnaire service provider and require them to prove at least the following before you use them:
    • All data entered is cryptographically strong encrypted in transit and at rest (both in production and in backups);
    • Access to a questionnaire requires an account, direct in email linkage to the questionnaire is not considered secure;
    • All passwords are stored cryptographically strong one way hashed;
    • Failed password account lock out is implemented (either as an absolute lock out or a timed lockout);
    • Questionnaire data when deleted or no longer required is wiped and then deleted to ensure there is no residual data. This includes backups (which implies per questionnaire encryption keying);
    • External Pen Test summary report enacted in the last 12 months in which no critical or high issues are unresolved.

Remember, at the end of the day, security is implemented by people working together to maintain and improve controls, procedures and processes over time. I have yet to see any security questionnaire that is able to fully assess day to day security professionalism within an organisation, that always requires talking to people. To me this is by far the most important way to assess security.