A letter from Valtix CEO Douglas Murray
Last week was an exhausting one for many in the tech world. CVE-2021-44228 (Log4Shell) provided a shot across the bow in what seems like a long-term war rather than a short-term battle. As we head into the holidays, I hope everyone can get some much-needed rest.
We see Log4shell as the most impactful and pervasive vulnerability of the last ten-plus years, with additional Log4j vulnerabilities now emerging just to make matters worse. I don’t believe I’m an alarmist. It’s a harsh reality how slow the software industry’s response has been – I’ll expand on this later. Our Security Research team started seeing evidence of active exploitation and reconnaissance related to CVE-2021-44228 on December 10, 2021. A noticeable increase in exploitation attempts began on 12/12/21, as more community tools became publicly available. Over the last week, we have provided regular updates to the community and daily updates of our security feeds.
We’ve also had great success in helping customers put in place mitigations while they patch or wait for vendor fixes. As a result, we have decided to offer up, for free, the full enterprise edition of the Valtix platform to anyone across the industry who needs help for 90 days. We also won’t hound you with sales. It’s not the time for that. Response to CVE-2021-44228 is an industry-defining moment. I believe it is essential to do our part to assist security teams around the world. If interested, please sign up here.
If one thing’s certain from Log4shell, the impact of having vulnerabilities so deeply embedded in so many products will require additional levels of vigilance over the coming days, months, and possibly years. I hope this is a wake up call that all software vendors, especially security vendors, need to approach things differently. But this is also an excellent opportunity to ensure every organization has defense-in-depth and zero trust strategies, which are essential to mitigate supply chain issues when they get through.
A New Wrinkle in the Supply Chain
Even after over a week, it’s still uncertain about the extent of this vulnerability in commercial software.
Fortunately, the Cybersecurity & Infrastructure Security Agency (CISA) compiled a list of software products. In addition to being a useful reference, the list starts to reveal an interesting picture of the current situation.
As of the time of writing, there were 1,638 products listed. Of these, about 30% had no status listed, which we’ll ignore for now. Of the remaining, 338 were listed as affected and 207 under investigation. That’s about 50% of total software with a potential impact from Log4shell. If we were to extrapolate that to the likely thousands of applications not on this list, it’s incredible to think of the extent and number of person-hours it will take to remediate when all is said and done.
It doesn’t stop there; of the 338 listed as ‘affected’, only 109 show as fixed. That’s a whopping 67% of the log4j ‘affected’ applications without a fix after a week. If nothing else, that’s a lot of uncertainty for organizations that might have many of these applications running critical functions.
And that’s just the availability of a fix. It takes effort to test updates for compatibility and roll them out at scale while also dealing with incomplete initial responses. I call this mean time to secure (MTTS). From the time the zero-day was announced, how quickly can organizations deploy protections, even before they have time to scan their environments to discover vulnerable systems and patch them. It will be a marathon, not a sprint.
Those are the numbers, and now the other interesting aspects are the patterns that emerge in the CISA list:
Software across the spectrum was impacted. New products, old products, on-prem, cloud; Log4j2 shows up in some unexpected places (and some places that didn’t surprise me in the least bit).
Most cloud services received fixes quickly. Many cloud services show up, but most already list fixes. How quickly cloud-delivered services received an update is an excellent illustration of one of the many well-known benefits of cloud and why I believe cloud-first, as-a-service is the way to build software going forward.
Sometimes cybersecurity becomes the weak link. Many cybersecurity products are on the list, and many fall into the unfixed list from vendors large and small. A notable example includes PAN-OS for Panorama, which is responsible for managing network security at many organizations.
Palo Alto Networks lists the Panorama vuln as a 9.8 on a scale of 10, but no fix was available more than a week after the vuln was known. Their estimated availability? December 22nd. That’s almost two weeks after the issue was revealed – just in time for the holidays for already weary IT teams.
(Update: PAN has now updated their website to say that released patches on 12/20, which is still not a great turnaround for such a serious issue)
That Panorama was impacted, and the response has been so slow, points out that there is still a lot of legacy security that requires too much management. Our point of view: security should require zero management.
As an industry, we must do better. Two weeks to provide a fix is too long for a code red vulnerability such as CVE-2021-44228 in such an essential security management product.
Beyond the obvious need to find, fix, and patch, what does this all mean?
The only mitigation for this issue comes back to defense-in-depth. Web sites should be protected. Intrusion Prevention policies should mitigate exploit attempts – often called virtual patching. Least privilege access should be maintained. Outbound traffic should be monitored with an allowed list of sites (domains) and URLs. And you can’t secure what you can’t see – you need visibility.
All of these strategies are applicable whether you’re in the cloud or on-premises. What, in the end, matters is the speed of enabling without a lot of heavy lifting or maintenance if the next zero-day targets the underlying technology. The advantage of cloud-native security is that you can do this in minutes, not days or weeks, with zero maintenance.
As a security vendor, we always feel the tension between being helpful in these situations versus the perception that we’re just trying to sell our solution. Again, to help those who want it, we’ve opened up full use of our commercial product to mitigate Log4Shell or any further derivatives in AWS, Azure, GCP, and OCI – free with no sales pitch. Organizations can self-service sign up in minutes here.