The Fatal Flaw in IT Risk Management

Data breaches are not all bad news. Sure, a data breach can be disastrous for a company.  However, breaches can also provide valuable insight into the broken IT risk management and security practices of different organizations. The infamous Target breach is a prominent example of some of the ways a large, well-funded security program can fail to protect business. One of these insights is how blind organizations are to some fundamental threats.

Failures In IT Risk Management

Target’s breach has all the hallmarks of an organization that cannot and does not manage security operations effectively.  When you dissect the breach information, there are numerous interlocking moments of laziness on the part of IT security staff and vendors:

• Networking groups failed to isolate the access of a third party vendor, which provided access for the initial attack.
• Internal staff failed to detect the installation of a command and control system inside their own network.
• And, security teams overlooked or ignored alerts from detection technologies.

However, a question emerges from these failures: how did the risk managers miss all these weaknesses?

In all fairness, Target is not unique here.  Most organizations would miss threats of this nature because the data from their risk assessment efforts does not accurately reflect the reality of the organization.  What is the culprit for this flawed data? The use of surveys and questionnaires to gather risk data.

Questionnaires Are Everywhere

The use of questionnaires has a long history in IT risk management.  As organizations have become more diverse, and threats more sophisticated, it is increasingly difficult to assess risk.  Furthermore, risk management teams are under pressure to build processes where the organization can plug in any qualified risk auditor to manage the processes.

Accentuating this issue are GRC (governance risk and compliance) platforms like RSA Archer.  These platforms almost exclusively rely on questionnaires to gather risk data from people.  The reasoning is that questionnaires are efficient and promote reusability.  Once those questionnaires are built, they can be reused indefinitely with almost no effort on the part of risk assessors.  They also standardize the questions across all respondents.

Even some compliance programs rely on questionnaires.  The PCI-DSS uses Self Assessment Questionnaires to gather information from compliant entities.  Some security firms have also begun marketing “point and click” style HIPAA and NERC-CIP portals which use similar questionnaire programs.

So what is the problem with this widely used technique?  Succinctly, people lie.

Questionnaires Yield Bad Data

Ask yourself this question: do you sincerely believe that an incompetent person is going to respond to a questionnaire in a manner that highlights their incompetence? For example, imagine an incompetent or lazy system administrator.  His work is poor, his attention to detail weak, perhaps he is distracted with personal or financial problems.  On a questionnaire, it asks this system administrator to explain how often he checks systems for updated patches.  He knows that company policy mandates that every system is checked monthly.  However, he has not checked them in months.  Maybe he does not like his boss or was passed over for a promotion.

What are the chances that this very weak link in the security chain will answer a standardized questionnaire honestly?  What motivation does this person have to be truthful?

He has almost no motivation. His poor performance actually motivates him to lie even more to cover up his incompetence. In a recent Harvard Business Review post, a statitician discussed this exact problem: people are not objective about their behaviors.

The real problem is that this incompetent employee is the most important threat to the business and IT security.  His incompetence, laziness, or lack of training is a gigantic threat to the organization. Yet, any questionnaire that person responds to will completely hide that fact.  Moreover, his lack of understanding of risk concepts, terms, or practices, could also hide a wide assortment of weaknesses.

Questionnaires yield untrustworthy data.  When a person’s job is on the line, they are not going to highlight their weaknesses or struggles.  Quite the contrary, incompetent people often overstate and inflate their skill set where as highly competent people tend to understate their skills (See Dunning-Kruger Effect) . This skews the vulnerability data even more.

Moreover, standardization actually works against threats. Threats are evolving so rapidly, that what was important to the organization 12 months ago could be radically different now. As such, any questions written 12 months ago, are not as relevant now. Standardization of questions assumes the threat landscape never changes.

In order for risk assessment to work, assessors must be able to trust that the data is meaningful.  The data must provide a truthful picture of the environment, such that threats and vulnerabilities can be ranked and assessed.  Moreover, it must adjust to the shifting threat landscape naturally. If the data gathered from staff does not paint a representational picture of the environment, then whatever risk analysis comes from that data is faulty. This is merely a variant on the “garbage in, garbage out” cliché.

Much of the risk assessment happening in large organizations these days is derived from this flawed data.  When questionnaires are sent out, people check off whatever they perceive as the “right answer.”  This results give the appearance of a well-managed environment. Likewise, threats that are much more serious to an organization are incorrectly skewed to being lower risk than they actually are.

This entire process erodes the whole point of risk management. When risk assessment teams have faulty data, they have faulty risk assessments, which in turn causes faulty decision making among executives.

Benefits of Interviewing

If surveys and questionnaires produce flawed data, then the alternative is rather simple: conduct in-person interviews.  In-person interviews can yield significantly more useful risk data as well as having numerous ancillary benefits as well.

The key to making this process work, is to make the interviews as informal as possible.  You want to get people babbling and venting.  When people vent and babble, they reveal truth.  A skilled interviewer can draw that truth out of the person and corroborate that with what other people say.

This is exactly the reason why the police never use questionnaires on suspects.  Can you imagine the police sitting down a suspect and asking him to fill out a questionnaire?  People are gullible for the truth if they feel like it is safe to share the truth.  If you put a guilty person in a room, and make them feel comfortable and safe, they will usually confess.

Furthermore, employees are not guilty of some heinous crime.  Most people know exactly what is wrong in the organization and they desperately want somebody to listen to them.  In this case, even a modicum of sympathy and concern will get them venting about all matter of vulnerabilities.

How to Conduct a Risk Interview

IT risk assessment might not be looking for bank robbers, but we can leverage some of the skills of interrogators to conduct meaningful interviews.  Below are some best practices for conducting risk interviews:

• Meet one on one.  In groups people are more likely to just follow along with what others say or what they think they are supposed to say.  Without a group, people are more likely to be honest and share their real thoughts.  You can also wear down a person’s walls and get them to reveal information.
• Establish a safe-zone.  The interview must be presented as a completely safe environment to share openly.  State your intention in the beginning of the meeting: to learn about the environment and to help improve security.  Also, be very clear that they can share anything anonymously.  The interviewee must not feel like there will be repercussions to their honesty.
• Ask open ended, broad questions.  Questions should be designed to get people sharing and, if necessary, venting.  Mostly, they should not be about risk.  Ask about their role, responsibilities, and what they think works and does not work.  Having a person simply walk through a business process will often uncover commentary about how broken the process is.
• Appeal to Ethics.  Most people have a desire to do the right things.  This is a strong sentiment, especially among IT professionals.  Appealing to this will reassure the interviewee and help create an atmosphere where they can freely share their experience and insights.
• Evaluate the credibility of the person. Interviews allow the risk assessor to improve the accuracy of the data through excluding obviously faulty data.  If a resource lacks knowledge or experience about a process or control, their input can be discounted or even ignored.  This allows the assessor to evaluate accuracy as the data is being collected.  Questionnaires provide none of this, and therefore all input has to be accepted as true.
• Build rapport.  Risk is human. The process should emphasize the importance, role, and input of individuals.  Questionnaires impose an unnecessary “formality” which few people can relate to.

Suggested Questions

Questions for Business Process Owners

• Explain the business functions you oversee.
• How critical are these to the business?
• Have those processes failed before? How?
• Where are the weak points? What is broken here?
• What would be the damage to the business if these processes failed? +

Questions for Technical Custodians (IT Staff)

• Walk me through the infrastructure (or systems, applications, etc.) you manage.
• What do you think is broken here? What is not working? Who is not doing their job?
• How do people learn about security controls here?
• How bad would it be if _____ (threat) happened?
• Has _____ (threat) happened before?
• What do you have in place to prevent these threats from happening?
• What controls does the environment need?

Make sure you take ample notes about these answers.  You might consider recording these interviews, if allowed.

The Painted Picture

Risk interviews are all about aggregating the data into a picture.  That picture tells a story of risk.  Mostly, the interviews often uncover threats that do not appear on vulnerability scans or penetration tests.  When analyzing interview data, look for the following.

• Poor leadership, direction and vision
• Broken, inefficient, or unproductive practices
• Bottlenecks in reporting or decision making
• Inexperienced or demoralized staff handling sensitive data or systems
• Overly permissive access
• Lack of controls around granting, revoking, or monitoring access
• Lack of change and configuration management
• Frustration among security or IT staff
• Lack of security awareness or training
• Lack of respect for security controls and policies
• Lack of knowledge of policies or procedures
• A “high-blame” culture.

Ancillary Benefits of Interviewing

The primary complaint with interviews is their inefficiency.  It is a true that interviewing people consumes more time than sending out questionnaires.  However, this extra effort is not wasted.  On the contrary, the ancillary benefits of interviews far outweigh the increase in effort.  Interviews are positive organization engagement, with long term benefits.

Interviews provide risk managers or IT security staff time to engage with the organization at a very human level.  Every interview is both a chance to assess risk as well as an opportunity to discuss security best practices.  Interviews give IT security a face. They are more human and tangible.  Each interview is a chance to strengthen a relationship with stakeholders in the organization and demonstrate value.  These intangibles can have a profound impact on the ability of information security or risk management to carry out important remediation efforts in the future.  If people in the organization trust you, they will follow you.

Conversely, questionnaires are a negative engagement model.  When staff receive requests (which are more often seen as demands) to fill out questionnaires, it quickly becomes an annoyance.  “This is just a stupid paper work exercise,” is a common complaint we hear from people faced with completing questionnaires.  This further erodes the honesty of the respondents as well as their trust in IT security people.  Since they do not respect the process, they do not respect the people who execute that process.  In the long term, this makes it significantly more difficult to get buy-in on future remediation efforts.

Conclusion

It is time to fix risk assessment.  The reliance on questionnaires and other impersonal techniques does not deliver useful risk data.  Every breach is another example of organizations are overlooking critical threats to their business.

There is a way to avoid the next data breach.  However, it will not come from a GRC portal.  These portals deliver only the illusion of control and insight.  It is time logoff the GRC portals, and get out into the organization and interact with people.  Within the babbling, venting, and pondering of the employees lurks honest risk data.  A reality of what is really going on, and where the real threats are.

Risk management is human.  To effectively assess risk, you need more than data, you need intelligence.  Questionnaires, surveys and portals full of data are not intelligence.  Intelligence comes from the analytical mind of a skilled risk assessor who can digest both hard and soft data to paint a complete picture of risk.