Recent events have caused people all over the information security community to question the efficacy of the PCI-DSS. The Target breach has become a lightning rod for debate as to how well the PCI-DSS protects organizations.
In a recent blog entry, Avivah Litan from Gartner said: “the PCI security standard has largely been a failure when you consider its initial purpose and history.” This entry goes on to discuss how the PCI-DSS has nothing in it that would have detected and stopped the latest malware in the Target breach. The full blog entry is here.
Yesterday, Bob Russo, head of the PCI Council issued a statement countering Litan’s assessment. In this statement, Russo claims that the majority of breaches have come from a breakdown in security basics such as poor implementation or maintenance of controls. Russo defends the standard as being an “excellent line of defense.”
They are both correct and they are both avoiding the elephant in the room: the failure of the PCI standard is not the standard itself; rather it is the deplorable state of Qualified Security Assessors (QSAs) who are passing companies that are nowhere near compliant. The PCI-DSS has ample controls that would catch malware like that used in the Target breach. What the PCI-DSS lacks are assessors who can effectively analyze an organization to determine if those controls actually work.
Appalling Assessments
The current state of QSAs is appalling. While there are plenty of capable and competent QSAs, there are also plenty of truly awful ones. Those awful ones seem to stay very busy. We have observed organizations that are profoundly non-compliant get passing grades from other QSAs. We have conducted assessments where clients were significantly out of compliance, which enraged the business leaders. These leaders then shopped around for another QSA who willingly passed the company in a non-compliant state. Furthermore, clients have outright solicited us to ignore their non-compliant configurations and let them pass, citing the fact that other QSAs were willing to pass them and therefore we should as well.
In one incident, about two years ago, I met with the compliance team at a large, national company. The director of this team said to me, in very clear terms, that they did not care what the PCI-DSS said, they were compliant, and if we wanted their business we would pass them without question. They told me they fired their previous QSA because they “took too long performing quality assurance.” When I told them this was not how Anitian worked, they dismissed us immediately and made it clear that our competitors had no problem with these assurances and that we clearly did not know how PCI worked. It was beyond demoralizing to hear this, especially coming from a company that has a very respectable name (which I will not mention for obvious legal reasons.)
Moreover, there are assessment firms who aggressively promote checkbox style assessments as a cost-effective way to compliance. They use euphemistic terms like “compliance as a service” to tip off business leaders that PCI is nothing more than checking off some boxes on a form. They also make it clear that they will pass a company regardless of their actual compliance state.
Furthermore, these are not small-time QSA companies operating out of dingy offices. These are widely recognized names with thousands of PCI clients. They have become extremely successful at selling a type of compliance that is appealing to business leaders who do not comprehend the scope of the risks, and just want PCI to go away.
Weak QSA companies ultimately breed weak assessors. Many of these firms hire people who only have basic audit skills and no technical acumen. These people supposedly understand the PCI standard and process but lack the requisite experience to effectively evaluate if technical controls are implemented correctly on systems which they don’t understand. Their technical inexperience becomes an insecurity, and rather than admit they do not know something, they just ignore the complex problems. Compliant entities have seized upon this and purposefully created ridiculously complex descriptions of their security controls for the sole purpose of intimidating inexperienced auditors.
For example, Anitian has worked with a number of clients who had been compliant for years. Each year the previous QSA would come in, spend a few days locked in a conference room, talk with nobody, look at no systems, review no configurations, and then deliver a signed Report on Compliance. It was shocking for both us and the client to discover how little the QSA understood the environment. Yet, they took no time to actually look at anything or confirm that systems were configured in the manner the company had claimed.
Exacerbating this problem are companies who are both QSAs and product resellers (for full disclosure, Anitian does both as well.) In these cases, there is an endemic bias in the assessment process since the assessor not only is approving the control, they also sold the control to the company. At Anitian, we keep these two teams entirely separate and enforce strict separation of duties. However, many QSA companies have no such separation and will use their role as the auditor to push whatever products they are selling.
The Council Responds
The PCI Standards Council has tried to clean up these weak QSAs, but with middling success. A few years ago, they introduced a quality assurance process for QSA companies. QSA companies are regularly audited to ensure they are following the assessment rules. Moreover, they introduced a process for removing companies that do not comply with quality assurance rules. However, in the years since this was introduced, a tiny fraction of companies have been removed from the list of QSA companies. Occasionally a QSA company will show up as “in remediation” indicating there has been a QA problem, but these are fleeting and quickly disappear.
The QA process was a step in the right direction, and frankly, the Council deserves credit for implementing it. It goes without saying that some QSA firms were unhappy to see those QA programs implemented since it meant more overhead for their assessments.
It would be logical to attribute our position in this blog entry to mere sour grapes for losing opportunities. There is truth to that assertion. We are frustrated that some people portray our honesty and integrity as a weakness. It is discouraging when the management of a respected brand-name company tells you that the only way you can earn their business is to be unethical. It is equally discouraging to hear about breaches, which were the result in basic lapses in security, while simultaneously QSA companies are promoting weak assessments as a competitive advantage.
Yes, we are frustrated, but it is because we agree with Russo that the PCI-DSS is a good standard for security – if companies actually followed it and QSAs actually enforced it.
Improve the QSA Certification Process
The PCI industry needs to grow up. This begins with some improvements to the PCI process of certifying and educating assessors. If we want to stop the breaches, the entire PCI process needs some improvements. As such, our recommendations for Mr. Russo and the PCI Council include:
1. Significantly Increase the Technical Expertise Necessary to be a QSA
The current QSA testing and certification process is focused exclusively on explaining the PCI standard itself. The training does almost no evaluation of an assessor’s technical skills. The certification process should have a more rigorous testing of technical concepts, particularly around network segmentation and system security. All potential QSAs should be required to pass a technical test, or attend a remedial security controls class. A QSA should be able to expertly analyze complex network and system controls.
2. Reformulate Audits of QSA Companies
The PCI Council currently conducts audits of QSA companies on a quarterly basis. The audits are predominantly a paperwork exercise. The audit effort should focus on interviewing the actual QSAs themselves and less on paperwork. This should be an oral test of sorts, where a QSA has to respond to questions about scoping, technical controls, and the intent of the standard. QSAs that do not meet minimum qualifications should be required to attend remedial technical training or face removal from the QSA list.
3. Forbid QSAs from Performing Integration Work or Vice-Versa
There is too much opportunity for corruption and bias when a QSA is both the auditor and the integrator of compliance solutions. QSA companies should be limited to one or the other, but not both. A company that sells remediation technologies to an assessed entity should automatically be forbidden from also conducting their annual QSA audit. Likewise, a company that is conducting an annual assessment should be forbidden from selling security products to those same clients.
4. Forbid Long Term Assessment Agreements
Companies should be required to change their QSA assessor every two years. This will cut down on long, multi-year contracts which promote complacency.
It should be noted that many of these improvements would hurt our own business. However, we believe that if the PCI-DSS is to remain viable, it must improve the integrity of the assessment process.
The Council has already shown a willingness to start demanding higher quality work. In the PCI-DSS 3.0, for example, the rules around penetration testing were significantly augmented. This was in direct response to companies hiring low-cost, low-effort, and low-quality “penetration testers” to fulfill the requirements. These tests were nothing more than glorified vulnerability scans and did not meet the purpose of the exercise. The new standard requires more rigorous testing process and validation of scope. Clearly, the Council is not afraid of going after weak testing. They just need to now extend that to the QSAs they certify.
Conclusion
The PCI-DSS is only as good as the people and companies that conduct assessments. Russo is correct; most breaches come from organizations missing basic security controls. If the PCI-DSS is going to remain relevant, it needs to be something that companies, consumers, and governments can trust to ensure those basic security controls are in place and functioning correctly. While weak QSAs might not be the reason for all breaches, they are eroding public trust in the PCI-DSS and facilitating conditions where breaches are more likely.
It is time to make the PCI-DSS be something that companies can really trust. This will means that the PCI Council is going to have to stand up to some of the largest names in compliance and demand they behave in a more responsible manner.
23 thoughts on “The Failure of the PCI-DSS?”
I think you get most of it right. I have personally seen companies that have absolutely no chance in Hell getting a passing grade, yet there are QSAs that are perfectly willing to give them a clean RoC. Really nothing prevents companies from going RoC-shopping as long as there are QSAs out there that are selling. Why? Because QSAs inherit absolutely no responsibility or liability for wrongly giving an assessed company a passing grade.
Second, yet- QSAs need to be evaluated on their technical skill. When I went to QSA training, over half of the class were financial auditors. How can they look at a firewall ACL list and determine what is appropriate? How can they explain to a C-level executive why Telnet is bad? How can a CPA effectively explain what SDLC is, secure handling of encryption keys, or even give a 10 minute talk on OWASP Top 10?
Both of these flaws in the system dilutes the QSA value- it’s basically a license to print money for some people. As long as there is no liability on the QSA side for issuing clean bills of health when one is not earned, and there is no demonstrable competency in security and technical audits, then yes, it is a joke.
This does not, however, relieve Target one iota of their responsibility or liability of what happened to them. Even with the worst QSA in the world, they have enough manpower and expertise to know what is correct and proper.
Yes, you are correct that Target does not get off the hook for a bad audit. I think asking QSAs to sholder the financial liability is not the answer. The assessed entitity is ultimately responsible for their compliance. And if they hire a weak QSA and get breached, then its the assessed entities fault. What would be nice is if the QSA companies who assessed these breached companeis was made public. Then companies could see which QSAs are giving out bad ROCs. For example, I suspect one of the larger names in PCI would have a looooooong list of companies they have passed that went on to be breached. If that was made public, consumers could then decide if they wanted to use that QSA. Also forcing entities to change QSAs every few years would mean there would be more turn-over in assessors, and therefore a more diverse set of opinions.
Andrew — I agree 100% with your diagnosis of the problem. The QSA market is setup to reward firms that do poor quality assessments quickly and cheaply; especially small firms that roll the dice that their shoddy work won’t end up being front-page news (still quite unlikely; after all, we still don’t even know who did the Target ROC!). One more place to start might be to require that QSA firms have peer reviews or something similar to assess their own internal QA controls, and have some sort of limited individual QSA/partner liability to get some “skin in the game” (probably not enough to bankrupt someone, but some type of fine).
A bigger idea that I’m a big fan of is to re-align the incentives by using insurance. Back in the early 2000s (around the Enron/Anderson collapse) there was an idea floating around called “financial statement insurance”. Basically, instead of the government requiring publicly-traded companies be audited, FSI would require that they carry insurance which would pay out to investors if there were a lawsuit that concluded there was a material misstatement in the financial reports. The insurers would price their premiums based on inherent risk factors (how big is the company, what industry is it in, are its finances simple or complex), as well as a specific assessment. E.g. the insurers would be the ones who would select auditors, and they would use the results of those audits to decide whether to continue providing insurance, and how expensive the premiums would be for that insurance. Incidentally, this idea never happened in the U.S. because the Big 4 hate it — they like the status quo where they control the market for big company audits, and FSI would have broken them up and made the market look more like law firms — lots of smaller and mid-sized regional players.
I think an FSI-like insurance model would be perfect for the payment card marketplace. Merchants would basically be required to get insurance, and the insurance companies would be empowered to do as much or as little they want in determining risk, likely using something like PCI-DSS as a tool to standardize their decisions. The insurance would pay out upon a data breach. The insurance companies would hire auditors based on their ability to cost-effectively do *quality* assessments, that were based on *real risk*. Poor quality audits would result in bad underwriting, and would be unprofitable for the insurance companies. This would improve the quality of audits when necessary, reduce the burden on smaller companies that have lower inherent risk and higher relative compliance burden, and ultimately help sort out the question of responsibility.
One final comment is that by going down this path, card brands could easily start including entities who generally do not get a ROC today, but do have lots of cardholder data (namely: issuing banks). This would be a cost effective way to balance risk versus compliance cost for all of the entities with cardholder data.
I just ran across a blog from one of my favorite law firms (InfoLawGroup) where they explore this topic with a bit more depth to how it could realistically work in this context:
InfoLaw Group: Payment Card Breaches: Time to Spread the Risk with Mandatory Cyber Insurance
This is a good idea. On the surface I am skeptical, since it may lead to just another layer of abstraction. However, I like the general idea. I suppose the one complaint I would have is that insurance companies would essentially take over the assessment process…leaving us assessors out in the cold. However, the core idea is sound.
I don’t believe there is cure for weak assessors, there will never be enough good ones on market. I spent few years in security audit but returned back to network engineering after having too much exposure to auditors and managers with no technical background. Even Security engineers are not good enough to understand i.e. network basics so how auditors could be ?
Better a few good ones that a lot of bad ones. The bar definitely needs to be raised, but so as long as the PCI council has a vested interest in cranking out QSAs of any caliber, then the Council will continue to do so.
It is posts like this that make me really respect Anitian. It takes a lot of cajones to say these things. I do not see this kind of stuff from any other security firm. You are 100% correct, Mr. Plato. I too have seen this stuff. I worked with a QSA a few years ago who did not even understand what a VLAN was. I had to explain to him the whole idea of tagging.
I too like what Mr. Dyk suggested. But, do you really think the payment brands and banks would allow this? It seems like the whole PCI system is setup to diffuse risk and provide only the appearance of security.
As long as the one being audited are able to choose who shall audit them the problem will still be there, even if they are forced to find a new auditor every few years that will pass them without having all the controls in place.
There must be put more control on the QSA, both as mentioned during training and certification of each QSA but also of the QSA company itself by follow up on processes and procedures. I have met QSA that hardly grasp the basics of encryption and key handling.
One of the QSA company that I have been audited by always demands that the evidence presented during the audit shall be kept by the QSA for 3 years if the QSA get audited. This includes documentation of processes, standards and logs the evidence stored also includes screen shots from system settings.
If all QSAs do store such detailed evidence it would be easy for the PCI SSC to audit the audits in detail. If possible each QSA would have to have one of its audits reviewed each year, if the audit fails all of the QSA audits for the last year is reviewed. For each failed audit there would be a fine and if needed possibility to revoke the QSA status for the reviewed QSA, the clients of the QSA would also need to be revoked of their PCI DSS status if their is major security concerns.
Above would set pressure on the QSA and also the possibility for audited companies to be revoked of their PCI DSS status, it will not solve the entire problem.
Another issue in regards of security is that many entities do see the PCI DSS as a quality stamp for having high security, where it in fact should be seen as a bare minimum for keeping thieves away, having such believe many do relax and expect all to be fine when they instead need to monitor the area of security more closely.
“For example, I suspect one of the larger names in PCI would have a looooooong list of companies they have passed that went on to be breached. ”
Care to enlighten us on that larger name in PCI? 🙂
Unfortunately, I cannot. That would be a breach of confidentiality with our clients, not to mention expose our firm to legal risk. However, I can say they are well known names. I will say that its the low-cost providers who are really bad. Anybody who is the lowest bidder on a PCI deal is probably doing poor work. Also, watch out for any place that relies heavily on some kind of portal or cloud service to deliver their compliance work. While those are very useful tools, they are not a replacemnt for sound assessment practices.
plato; This is an interesting blog, I am in full support with the point that the council should ensure that QSAs are technically sound. Secondly the rotation of QSAs by entities will definitely save a lot of entities. what I have also observed is that entities do not have a sustainability strategy in place that can assist them to keep their time based controls in check. Penetration testing is just one aspect, but if processes are not effective and properly reviewed an entity can easily be breached.
The council should also ensure that no QSA company sells security solutions to the entity and it should be stated clearly as part of QSAs code of professional ethics.
Thank you for the feedback. I agree that sustainability is a common problem we see. As I have mentioned before, “checkbox” style compliance portals greatly exacerbate this problem. People think “oh, all I have to do is fill out these forms, get the right stuff to the QSA via their portal, and that PCI thing is all taken care of.” Once everything is signed off, the company just unwinds everything they did to get compliant (or just lied about it in the first place.)
This is why on-site assessments from highly-technical people are so important. The QSA must validate that controls are not merely in-place, but actually working correctly. Many assessors neither understand the controls nor how to properly analyze if they work correctly. As such, they just take the client’s word that they are doing things right, check them off, and send over a ROC.
El estandard PCI DSS es realmente solido, sin embargo estoy de acuerdo en decir que las QSA requieren una preparación mucho mas tecnica para sus auditores y consultores, pues estamos llevando el PCI DSS al juego de la norma ISO 27001 en la que cada empresa selecciona simplemente algo de lo que deberia cumplir y el auditor basa su criterio en esa intención (SOA), pero PCI es algo mas robusto y que no deberia dar lugar a puntos medios mas alla de lo que se considera un control compensatorio, es decir que mientras el estandard sea implementado con coherencia y responsabilidad, este sin duda proporciona un esquema de defensa y prevención de alto nivel.
TRANSLATION (provided by Google): The PCI DSS standard is really solid, though I agree in saying that the QSA require much more technical preparation for auditors and consultants, as we are PCI DSS bringing the game to the ISO 27001 standard in which each company selects just something that should be met and the auditor based his judgment on that intention (SOA), but PCI is something more robust and should not cause average points beyond what is considered a compensating control, ie that while the standard is implemented with consistency and accountability, this certainly provides a scheme of defense and prevention of high level.
Not all QSAs are technical, but QSAs ARE NOT THE PROBLEM.
I am a QSA and have had several clients that have switched me out for another QSA in my company. It does not take long for upper management to notice. Within one or two clients a QSA in the large firms will get blackballed by their own management. In the big sweatshop firms you “are a team player” or you will be blackballed. These firms have lots of deals pending and don’t want a QSA jacking up their client relations. A QSA has no chance to do a good job.
No extra test questions or classes will fix this. I consider myself very technical and it does not help one bit. My solution has been to find the right QSA company to work for, and then deal with the previous rubber stamp ROCs (yes sometimes they don’t even get right the OSs of the servers they “assessed”), but they would if their company had let them look.
QSAs are not the entire problem, but they are part of the problem. Yes, there is a huge problem with “audit mills” that pump out meaningless audits. There is equally a problem with QSAs who simply do not understand what they are looking at. This is a problem that needs to be addressed at both levels. We need technically compentent QSAs, and harder certification requirements would help. We also need to start holding QSA companies accountable when they put out bad audits.
Another issue is that the council themselves are not really interested in security. I worked for a company that the lead qsa was so bad he got us put in remediation. Then started his own company and took many of our clients. There was no recourse. Earlier I had questioned how he got away with his practices. His answer, when I am in Boston I bring the girls flowers. But don’t try to whistle blow or you will get a nasty email that ONLY the lead QSA dare approach and kiss the ring of the glorious council. I once tried to turn in a retailer I discovered clearly was and I had evidence was complicate in cc fraud and was given the nasty gram. As long as the council has the power that is all they really care about. I am a libertarian but in this instance unless someone else checks the council’s hand it won’t improve.
I feel PCI-DSS is just the wrong approach altogether, at least with regard to online / web commerce. Even were it adopted 100%, the cardholder still has no idea what the compliance level is of the vendor their giving their card to. It is a simple fact that the truth of an agency’s compliance is always uncertain. The strain and financial burden it imposes on businesses is immense and incredibly costly, driving down adoption. The burden should be carried by the banks.
Why the industry has not gotten together to implement a “pay pal” like payment solution for cardholders is just incredible. A consortium within the banking and finance industry could accomplish this very quickly. It doesn’t matter what level of compliance with PCI-DSS we have, putting credit card info into a website it still a very bad idea.
Were this the case, there would of course still be some standards compliance to address, but it would be simple and easily verifiable. And it would have an astounding rate of adoption. I don’t understand why no one is talking about this. The current solution, PCI-DSS, is unworkable and is in fact not working at all.
I don’t think there is anything fundamentally wrong with the PCI DSS. The standard is fine. The problem is really that companies ignore security and checkbox auditors endorse that. The burden of compliance is not that bad when you architect security into a solution from day one. Unfortunately, many companies refuse to deal with security and then it causes pain later on.
Also, remember that it was the cardbrands that developed, built, and enforce the PCI DSS. They did so primarily to avoid government regulation. Getting all the banks and cardbrands on the same controlled system would post numerous problems, the least of which being collusion between financial organizations to jack up rates. The last thing we need is financial organizations have even more power than they already do.
Oh I don’t think there’s anything wrong with the standard itself or its recommendations at all. But even granted your system is pretty secure, doing all the scoping and nitpicking with documentation and trying to organize it all is actually the biggest nightmare to me.
I don’t mean to suggest financial organizations should be on or sharing the same running system, but rather use a standard API from say, an open sourced server product. I do believe this burden should be on the banks. We can talk about how people’s systems should be secure and meet the standard until we’re blue in the face, but it will never happen. It’s like preaching abstinence to teenagers.
I just want to reiterate that the entire problem is using credit cards for ecommerce in the first place. It’s an outdated payment system and requires replacement. Even if the PCI-DSS standard was successful and systems were widely compliant, it’s still a poor solution, because we’re still using cards. There are simply way too many points of failure using payment cards for ecommerce. I’d never trust it and neither should anyone else.
Hey sorry to hop into the conversation so late. But have you ever actually seen or heard of a company that was banned from taking credit/debit cards because of their PCI audit? I’m working for a company now that probably shouldn’t have passed it’s PCI Audit but honestly seeing as how fast the company providing our QSA passed us I don’t think there’s anything we could have done/not done that would have caused us to actually lose the ability to take payments.
Maybe not because of their PCI audit directly, but we have seen companies get heavily fined or take on huge lawsuits because of a breach…the result of which was because the previous QSA did not validate any of the required controls.