by Adrian Schoch
During several months, the content delivery network service provider Cloudflare leaked sensitive information like passwords from millions of websites. This is the story behind it and what your reaction should be.
On February 17, Tavis Ormandy, a security researcher at Google’s Project Zero team, discovered chunks of uninitialized memory interspersed with valid data originating from a reverse proxy operated by Cloudflare – a major content delivery network service.
The team around Ormandy was able to reproduce the problem and fetched some live samples. The samples contained passwords, cookies, encryption keys, parts of POST data and even HTTPS requests.
The findings, later referred to as Cloudbleed, were reported to Cloudflare, where appropriate action was taken immediately. Within minutes a cross-functional team was assembled and in less than 7 hours, thanks to a global team, the problem was resolved. The industry standard time to fix such bugs is usually multiple months.
The events surrounding this recent incident shows once more how important a highly-qualified and fast incident response team is to mitigate immediate threats effectively.
Lessons learned
Apart from having a capable incident response team at hand, organizations need to step up their forensic readiness, define timely update mechanisms and implement the tools required to do so. Furthermore, some flexibility within the organization is required to quickly adapt to emerging threats and develop countermeasures.
Recommended measures
If your organization is affected by Cloudbleed, you should immediately contact your customers that this incident happened and during the past few months sensitive information including passwords and login tokens may have leaked to adversaries. Customers are advised to change their login credentials and revoke single-sign-on tokens. If possible, 2-factor authentication should be activated for important accounts.
Detailed information
The following events took place during the Cloudbleed incident (information provided by Cloudflare):
2017-02-18 0011: Tweet from Tavis Ormandy asking for Cloudflare contact information
2017-02-18 0032: Cloudflare receives details of bug from Google
2017-02-18 0040: Cross functional team assembles in San Francisco
2017-02-18 0119: Email obfuscation disabled worldwide
2017-02-18 0122: London team joins
2017-02-18 0424: Automatic HTTPS rewrites disabled worldwide
2017-02-18 0722: Patch implementing kill switch for cf-html parser deployed worldwide
2017-02-20 2159: SAFE_CHAR fix deployed globally
2017-02-21 1803: Automatic HTTPS rewrites, server-side excludes and email obfuscation re-enabled worldwide
Further information can be found on:
https://bugs.chromium.org/p/project-zero/issues/detail?id=1139
https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
https://github.com/pirate/sites-using-cloudflare
About Oneconsult
Adrian Schoch is Head of Digital Forensics at Oneconsult AG.
Oneconsult AG offers incident response services and will gladly assist you with a team of highly trained, Swiss-based security professionals to reduce business-critical information security risks (such as Cloudbleed).
Oneconsult AG is a renowned Swiss cybersecurity consulting company with approx. 30 employees, offices in Switzerland and Germany, a customer base of 300+ organizations and 1200+ completed security projects worldwide. We are your trustworthy partner for a holistic cybersecurity approach against external and internal threats such as APT, hacker attacks, malware infection, digital fraud and data leakage. Our core services are penetration tests, ISO 27001 security audits and IT forensics. To protect your organization and mitigate specific information security risks, Oneconsult also offers practical security consulting, security training and virtual security officer services. Dedicated IT security researchers and a large team of certified penetration testers (OPST, OSCP, etc.), digital forensics experts (GCFE, GREM) and ISO security auditors (ISO 27001 Lead Auditor) are at your service.
