Why the World Needs a Software Bill Of Materials Now
“Sunburst” was the most sophisticated hack the world has ever seen. One of the most urgent lessons so far learned from the attack: we need to enforce global software supply chain management now. ¹
The first details on the “Sunburst” attack were released in December 2020: a highly evasive attacker leveraged the supply chain of the U.S. based software company SolarWinds in order to compromise an administrative I.T. management tool which is used by tens of thousands of private and government organizations. The backdoors and malware that were installed on the victims’ infrastructures have been called by Microsoft “the most sophisticated and protracted intrusion attacks of the decade”.² (That was a week before hundred of thousands of Microsoft exchange Servers were hacked, but alas).
It is still too early to draw definitive conclusions from the first forensic analyses, but this much is clear: supply chain attacks are not just devastating for individual organizations but they fundamentally undermine trust in information technology.
One of the underlying reasons the attack was so successful is that our society, at large, has a fundamental software problem. The issue is systemic, and it is on the verge of turning into an apocalypse.
We are increasingly dependent on computers and the software they run on. In the financial sector, in healthcare, in the military, in the manufacturing industry, and in the service domain, we rely on software we believe can be trusted.
And software is no longer confined merely to computers. It now controls the operation of phones, complex power generators, medical hardware, cars, doorbells, cameras, and kitchen appliances. As software engineer and venture capitalist Marc Andreessen put it already ten years ago, “Software is eating the world.”
The world runs on software. It runs on information technology. But it can’t run with confidence if major governments are disrupting and attacking the software supply chain in this way.
Brad Smith, President of Microsoft.³
Software Supply Chain Attack
The attack on SolarWinds has made it more apparent than ever before that software supply chain attacks are incredibly destructive. Damage done in a tiny module delivered by a small software vendor can break a global software empire’s security. This can lead to a loss of sensitive customer information, disruption of manufacturing processes or reputational damage.
However, there is an even more worrying effect: if attacks are possible at such a scale that their effect is felt across whole sectors, countries or even globally, as is the case with the “SolarWinds” attack, they have the potential to fundamentally undermine our trust in information technology.
Attacking the global software supply chain is easy. It does not take the coordinated effort of thousands of engineers in a nation-state to stage a global supply chain attack. It is straightforward to insert malicious code in open-source libraries. As recently as February 2021, a young hacker described how he set up a successful supply chain attack against companies such as Apple and PayPal by attacking the way common programming languages, such as Python and Ruby, load external modules. The mechanism can be simply tricked into installing compromised modules from public code repositories. ⁴ On march 29th, the official PHP Git repository was hacked and the code base tampered with.The incident is alarming considering PHP remains the server-side programming language to power over 79% of the websites on the Internet.⁴ᴮ
Virtual Private Networks (VPN) are also a vulnerable part of the supply chain. Cisco’s VPN equipment is most widely affected, but also the products of smaller vendors for enterprise VPN equipment are actively being targeted, as was shown at the beginning of Mai, with Fortinet and Pulse Secure having serious security flaws.⁴ᶜ
Supply chain attacks are not new, and they can occur in any industry. Yet, a few risks unique to the software domain make it especially vulnerable to tampering.
First, all software comes to market with flaws and vulnerabilities that are gradually discovered or reported by users. Then they are corrected by the vendor, at which stage new errors may be introduced. This causes a long tail of patch management in the supply chain. Most applications consist of hundreds or even thousands of functional modules. All of these modules have different modes of responsibility and accountability for the patch process. The more features a software product has, the larger the potential attack surface.
The legal setting may be very different for each of the modules; some are open-source, others are proprietary. That means that it can be hard to figure out responsibilities. Patching is not as straight forward as forcing a subcontractor into action with legal arguments.
Second, bugs, inadvertent or deliberate, can affect all hierarchical layers of a software product. A bug may exist at the lowest level of on-chip software. Then its effect may go to the operating system level. From there, it can affect the data transport level or the application framework level. Then, at the top of the software pyramid, the actual application may be affected.
All these levels have dependencies: an attack on a low system level may be able to corrupt or circumvent protective measures on higher levels. For example, an attack on the operating system level may outsmart all data encryption on the application level.
Third, protocols — technical standards that describe how data needs to be stored, transported, or processed — are not one-and-done safeguards. They can have theoretical limitations, for instance, by flaws in the design — e.g. requiring low-security standards. Or they may be subject to something akin to the decay half-life of a radioactive element, as is the case with encryption standards for which it is a matter of time before computers become powerful enough to break them.
Fourth, software patches for components in different hierarchical levels have different periodicity. On-chip firmware may be updated every five years or so. A typical framework or platform, such as Oracle or SAP, updates quarterly. Industrial applications may have release plans of a few years. Microsoft introduced the monthly “Patch Tuesday” in 2003. Google now also updates its web browser Chrome on a monthly basis. Ad-hoc patches are published in the case of especially critical vulnerabilities.
All of these updates have functional dependencies. Updating components of an on-chip module or an underlying data communication process in an operating system may break things elsewhere. Users need to address the problem by installing newer versions of the dependent module. This, in turn, may break other dependencies and push the issue to another set of packages.
There is a high level of uncertainty in managing the process. On the one hand, the consequences of choosing not to perform an update are reasonably well known: they lead to exposure of at least one known vulnerability. Yet, what is the likelihood that this vulnerability will be exploited?
On the other hand, the consequences of choosing to implement the patch are almost impossible to predict. Dependencies may be known for a specific version of the software on a system but may vary hugely depending on individual configurations or even on the process state of a particular system. Functionalities may break everywhere — in a particular application, in the operating system, between software modules within one system or between interconnected systems. The result is dependency hell.
Dependency hell makes software chains particularly vulnerable to attacks because the software update process requires careful planning and testing, and this can cause extra delays. One of the most significant data breaches in history, Equifax’s 2017 leakage of hundreds of millions of customer records, was possible because the organization failed to update a vulnerability in a low-level web server module on time — meaning within a few months.⁵ It wasn’t a particularly bad or unusual bug, and it would have been routine work to patch. Yet, for whatever reason — perhaps figuring out how to do it without affecting the availability of their systems — it took Equifax three months to do it⁵ᴮ. That gave malicious actors enough time.
This leads to a fifth reason why the software supply chain is so vulnerable to attacks: the sheer amount of known software vulnerabilities. As of March 1, there were 203.241 vulnerabilities documented in the U.S. government repository of standards vulnerability management database.⁶ For more than a decade, the number of reported vulnerabilities was fairly stable at around 400 per month. But in 2017, the number of reported vulnerabilities doubled, and growth has been exponential since then. This means that secure operations teams get swamped by vulnerabilities every day.
Attackers have automated scanning and intrusion. Which is another reason for extra vulnerability of the software supply chain: attackers have a strategic advantage, because what they need to do to be successful is much easier automated than what defenders need to do. And of course, they only need to be right once, while defenders need to be right all the time.
A final factor for the extraordinary vulnerability of the software domain to supply chain attacks is that including external software components is applied on a massive scale. A 2017 audit and analysis of over a thousand commercial, cross-sector codebases found that 96% of all software products included third-party software components and commercial off-the-shelf components, as well as modules and libraries from both open-source and commercial third-parties.⁷
The whole assumption behind this is that we can build established channels of system verification to gain privileged access to systems and ICT infrastructures. But this is no longer true: the channels are substantially affected by trends within the software domain, such as globalization, decentralization, outsourcing of the software manufacturing process and now also by nation state actors, who are determined to undermine all chains of trust. Add to that outsourcing ICT management and services to cloud computing and managed service providers, and the already massive supply chain problems in the software domain are multiplied by several orders of magnitude.
All in all, software is essentially a sitting duck for supply chain attacks. What can we do?
Bill of Materials
In other industries, like the automotive industry or other manufacturing industries, one tool has proven to be an efficient mitigation strategy for supply chain attacks: the bill of materials, or “BOM” for short. In its simplest form, the BOM is similar to a long list of ingredients, in which all materials and quantities needed to manufacture an end product are listed. In practice, BOMs are mostly nested, hierarchical lists, with the top-level representing the finished product and subsequent levels representing sub-modules of the product, functionally grouped together. Manufacturing organizations started using nested BOMs in the 1960s, inspired by the work of physicist W. Edwards Deming.⁸
Tampering with hardware is not an easy path for attackers, but because of the significant risks that arise out of a successful compromise, it has been an important risk to track for the manufacturing industry. From half a century of using BOMs in manufacturing supply chains, the main learning is that about half of the attacks occur at the prime contractor ⁹. Also, a bill of materials protects best against supply chain vulnerabilities when it is set up as a holistic cross-domain effort, ensuring a seamless connection between engineering and manufacturing.
For this to work correctly, a bill of materials must represent all involved projects, disciplines, parts, components, and 3rd parties. Done with great precision and rigor in an organization as a whole, it is possible to provide deep insight into the product and all its parts and their corresponding supply chain vulnerabilities.
Similarly, a software bill of materials, or SBOM, identifies and lists all software components, information about these components, and the relationships between them. Including, crucially, any known vulnerabilities and published proof-of-concept exploits. SBOMs are a tool that allows stakeholders to better manage cost and ICT and security risks by revealing the presence of known vulnerable software components to various supply-chain stakeholders. If users don’t know what components are in their software, they don’t know when they need to patch, right?
Medical technology is at the forefront of exploring the potential of SBOMs for improving the security of the software development lifecycle — partly because the medical industry has been targeted for years. Hospitals are frequently successfully attacked with ransomware, and also the Pharma and medical industry are regularly attacked, such as in the large 2017 attack on Merck’s R&D manufacturing and sales supply chain.¹⁰
The first proofs of concept have been completed. In 2018, the National Telecommunications and Information Administration (NTIA) initiated a multi-stakeholder process on software component transparency in the medical sector.¹¹ The results from SBOMs in the medical domain are highly interesting.
First, mature data standards that can capture SBOM data already exist.¹² One is the Software Package Data eXchange (SPDX) format, an open-source machine-readable format stewarded as a de facto industry standard by the Linux Foundation.¹³ Last year, SPDX was submitted to ISO for consideration as a global specification. The other standard is Software Identification (SWID), a formal industry standard used by various commercial software publishers.
Second, the proof of concept in the medical sector showed that there is nothing domain-specific in SBOMs. What goes for software in the field of medical technology applies to any software chain in any field, be it finance, energy, or manufacturing. Maturity models, methods for assessing the quality of an organization’s continuous improvement processes based on the use of SBOMs, have been set. That is important because the higher the maturity, the better an organization can turn incidents and errors into improvements. Also, new technological standards, such as a way to give software components a global, unique identifier or blockchain techniques, are being tested in the field.¹³ᴮ
Third, and perhaps most surprisingly: the cost of creating software does not go up when using SBOMs. On the contrary, by taking the whole software value chain into account, costs are reduced because organizations can save hundreds of hours in risk analysis, vulnerability management, and remediation processes.
Finally, multinational companies such as Philips Medical and Siemens Healthcare have created automated systems for SBOM-management and pioneered delivery of SBOMs to their customers in the medical sector, thus putting theory into practice.¹⁴
So, it’s all there; nothing new needs to be invented. But so far, the software bill of materials is rarely used. It’s use is still rare in the medical domain, organizations within critical infrastructures or the financial sector are only discovering its merits. And even in these heavy regulated domains, SBOMS are still rare.
Here are a few ideas on how to change that.
What needs to be done?
First, organizations should view the use of a software bill of materials as a litmus test for their own maturity level. Also, organizations can start selecting their third party vendors based on the existence and quality of their implementation of an SBOM. That may cause a ripple effect, in which the quality software supply chain management may rise.
Second, existing maturity models should be updated and re-calibrated so that the level of implementation of an SBOM is a mandatory requirement for reaching one of the five maturity levels. If there is anything that can be a good indication of the ability to perform a software project and create reliable, trustworthy software, it is the SBOM. Regulators should work with an appropriate industry consortium to develop a software supply chain security maturity model based on this and make it publicly available.
Third, The Linux Foundation should manage a priority list of software tools and open-source projects for any interested party to invest in and support. The Linux Foundation should make tools freely available with expert integration assistance and other appropriate resources.¹⁵
Fourth, standardization organizations should integrate data standards that can capture SBOM data, such as SPDX. Also, the draft Software Bill of Materials (SBOM) standard developed by the NTIA multi-stakeholder process should be made into an international standard.
And finally, regulators should collaborate internationally to drive the SBOM effort further. What has worked for the medical sector in a bottom-up approach should now be supported with the most urgent top-down advocacy effort. Specifically, the draft of the European Union’s “Digital operational resilience act for the financial sector and amending Regulations” (DORA) should immediately correct its omission of not including the SBOM in its first draft, and make current general ideas on third party ICT risk and security management much more concrete, by making SBOMs mandatory for all of the financial industry.¹⁶
Outlook
The transparency associated with SBOMs will help bring to light vulnerabilities and weaknesses throughout the software supply chain. This will support broader efforts in mitigating the threats of supply chain attacks, averting the ultimate risk of our society not being able to trust information technology anymore.
Problem solved? Of course not. The SBOM is a necessary, not a sufficient remedy for the large, global, systemic challenges of software development. Other things are needed — a global regulatory framework that forces the market into action, better technologies — e.g. a blockchain system for the software supply chain — and better training programs that will raise awareness.¹⁶ᴮ
It is of utmost urgency that a great number of stakeholders around the globe cooperate. It’s necessary to address the issue with the highest possible effort and maximum level of international cooperation. If that fails, the cost of supply chain attacks may soon dwarf the benefits of ICT.
The Presidential Executive Order, published 05/17/2021, addresses the issue. In it, NIST is ordered to come with a guideline within the next 180 days, aimed at enhancing software supply chain security.
At least, that’s a start.¹⁷ In march 2023, the White House published the National Cybersecurity Strategy, and SBOMs are also briefly mentioned:
To further incentivize the adoption of secure software development practices, the Administration will encourage coordinated vulnerability disclosure across all technology types and sectors; promote the further development of SBOMs.¹⁸
In the two years that passed since the Presidential Executive Order 14028, (section 10j), that is a humbling result.¹⁹
References
(1) Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURSTBackdoor; Update 19–12–2022: In november 2022, SolarWinds agrees to pay $26 m to investors, to settle a securities class action lawsuit. The company still faces regulatory enforcement action
(2) Deep dive into the Solorigate second-stage activation: From SUNBURST to TEARDROP and Raindrop
(3) CBSnews, 14.2.2021
(4) Alex Birsan, ‘Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies’
(4C) https://www.bankinfosecurity.com/hackers-hit-unpatched-pulse-secure-fortinet-ssl-vpns-a-12958
(5) https://nvd.nist.gov/vuln/detail/CVE-2017-5638
(5B) The process of “figuring out how” was in fact as ugly as it can get. It involved much more than failing to patch software vulnerabilities on time. It was, as in most disastrous hacks, a cascading chain of events, including plain clumsiness, fundamental architectural mistakes, operative errors, bad management and a failing security culture. To give just a few examples: Equifax did not have a proper ticket system for patch management, vulnerability scanners were out of life (leading to false negatives), security certificates required for intrusion detection were expired (by 1.5 years…), the principle of least privilege was not being employed and infrastructural countermeasures were completely missing, such as network segmentation, file integrity monitoring and protection for database credentials (yes, passwords were really stored in unencrypted data files…). See: https://www.consumer.ftc.gov/blog/2019/07/equifax-data-breach-settlement-what-you-should-know. Also: “The Equifax Breach” in Neil Daswani, Moudy Elbayadi, Big Breaches.Cybersecurity Lessons for Everyone, Apress, 2021, pp. 75–94.
(6) https://nvd.nist.gov
(7) Software Assurance Forum for Excellence in Code (SAFECode). Managing SecurityRisks Inherent in the Use of Third-Party Components, 2017
(8) Leitner, P. M. Japan’s post-war economic success: deming, quality, and contextual realities. J. Manag. Hist. 5, 489–505 (1999).
(9) Miller, John, Supply Chain Attack Framework and Attack Patterns, MITRE, 2013, p. 14.
(10) Merck Announces Second-Quarter 2017 Financial Results.
(11) NTIA software component transparency, 2020
(12) Survey of Existing SBOM Formats and Standards — Version 20191025
(13) http://spdx.org
(13B) See this interesting blockchain SBOM patent: https://patentimages.storage.googleapis.com/b8/fc/39/4ae0961f384950/EP3687107A1.pdf
(14) Koninklijke Philips N. V. Position Paper: Committed to Proactively Addressing Our Customers’ Security and Privacy Concerns, 2018 also: Siemens Medical Solutions USA, Inc. Cybersecurity: Protecting healthcare institu- tions against cyberthreats, 2020. Please find a list of open source and proprietary solutions here: Steven J. Vaughan-Nichols, “8 top SBOM tools to consider”, CSO, 25 JULY 2022.
(15) Atlantic Councel, Breaking trust: Shades of crisis across an insecure software supply chain, 2020, p. 29.
(16) Digital operational resilience for the financial sector and amending Regulations (EC) No 1060/2009, (EU) No 648/2012, (EU) No 600/2014 and (EU) No 909/2014
(16B) NIST 800–161, Supply Chain Risk Management Practices for Federal Information Systems and Organizations, 2015; NISTIR 8276: Key Practices in Cyber Supply Chain Risk Management: Observations from Industry, February 2021, https://csrc.nist.gov/publications/detail/nistir/8276/final; Kaiyue Zheng and Laura A. Albert, A Robust Approach for Mitigating Risks in Cyber Supply Chains, Risk Analysis, January 18, 2019, https://onlinelibrary.wiley.com/doi/
(17) https://www.federalregister.gov/d/2021-10460/p-56, section 4 covers Supply Chain Management, and it is by far the longest section of the executive order. SBOMs are explicitly required, and defined as (note that last sentence!)
the term ‘‘Software Bill of Materials’’ or ‘‘SBOM’’ means a formal record containing the details and supply chain relationships of various components used in building software. Software developers and vendors often create products by assembling existing open source and commercial software components. The SBOM enumerates these components in a product. It is analogous to a list of ingredients on food packaging. An SBOM is useful to those who develop or manufacture software, those who select or purchase software, and those who operate software. Developers often use available open source and third-party software components to create a product; an SBOM allows the builder to make sure those components are up to date and to respond quickly to new vulnerabilities. Buyers can use an SBOM to perform vulnerability or license analysis, both of which can be used to evaluate risk in a product. Those who operate software can use SBOMs to quickly and easily determine whether they are at potential risk of a newly discovered vulnerability. A widely used, machine-readable SBOM format allows for greater benefits through automation and tool integration. The SBOMs gain greater value when collectively stored in a repository that can be easily queried by other applications and systems. Understanding the supply chain of software, obtaining an SBOM, and using it to analyze known vulnerabilities are crucial in managing risk.
(18) White House, National Cyber Security Strategy, March 2023. https://www.whitehouse.gov/wp-content/uploads/2023/03/National-Cybersecurity-Strategy-2023.pdf
Updates
27–03–2021: included reference 13B, with thanks to Ben McCarty for the interesting link.
30–03–2021: added reference 4B, as an illustration how easily supply chains are attacked — hacking the PHP’s Git server to add backdoors to PHP source code is a big deal.
01–05–2021: added reference 4C, as hackers hit Pulse Secure and Fortinet SSL VPNs. Added reference 5B providing details on the Equifax Breach.
20–05–2021: added reference to the Presedential Executive Order “Improving the Nation’s Cybersecurity”.
22–12–2021: added note 16B for reference to long existing and relatively new NIST publications.
19–12–2022: updated footnote1: SolarWinds Settles Investor Claims Brought Over Cyberattack
26–06–2023: added a list to 8 open source and proprietary SBOM tools (See footnote 14).
