If it is up to Burr Sutter of Red Hat, there are two significant software development challenges: establishing a robust software supply chain and enhancing the onboarding experience.
Burr Sutter has been with Red Hat as a developer advocate and technology evangelist for many years. At the open-source company, he keeps a close eye on the industry to identify the latest trends in software development. During his recent visit to The Netherlands, we met him to explore what’s currently trending in development. According to Sutter, enterprises face the most issues related to a lack of visibility and onboarding processes.
Where are software components in the infrastructure?
Sutter refers to one of the most significant security threats in recent years to explain the weak software supply chain. In December 2021, the world learned about the Log4Shell vulnerability in the open-source framework Log4j. Even after almost 2.5 years, many enterprises still suffer from Log4Shell. It is still in production. “The problem is, they can’t find it. They know they have a problem, but don’t know how to address it. They need to locate it. Where is Log4J in the infrastructure? Companies probably don’t know,” observes Sutter.
Indeed, when we last wrote a big piece about Log4Shell, we found that, according to security company data, the vulnerability is at the top of many hit lists. Newer data also shows that Log4Shell continues to haunt numerous enterprises.
Sutter acknowledges that it is challenging to fully prevent the presence of software components and dependencies like Log4j in your environment. “It is essentially developers downloading free, open-source material from the internet. Log4Shell was interesting because it was implemented in Log4j eight years prior. The discovery was made in 2021 by an Alibaba engineer, who found that the feature opens a backdoor. If exploited in a certain way, you gain access to the machine. But no one knew it for eight years. That’s why it’s so pervasive. People have deployed it for years without knowing where it was.”
Disallowing open-source in an infrastructure is the only way to prevent an event like Log4Shell. However, Sutter wouldn’t recommend such a drastic step for innovation reasons. It might be advisable in specific situations, such as military technology, but open-source is present wherever innovation is necessary. According to Sutter, about 86 per cent of the code is open-source.
The software and the car
According to Red Hat’s vision, the way forward is to use open-source and enhance visibility. Therefore, Red Hat has built the Red Hat Trusted Software Supply Chain, a product family with several solutions to improve the software development life cycle (SDLC). The solutions include the Trusted Application Pipeline, Trusted Profile Analyzer, Trusted Artifact Signer, Advanced Cluster Security, and Quay. Companies can purchase one, several, or all of the solutions depending on their needs. They are tightly integrated to work together.
Sutter explains Red Hat Trusted Software Supply Chain by comparing it to how car dealers operate when there is an error with some components. When someone buys a new car from a dealer, the dealer records the name, address, and phone number. If there is a recall to fix a component after the buyer has received the car, the dealer knows which model was built on a specific date with the problem. “How did they know you had that car and that part that needs to be fixed? That’s the real-world supply chain,” explains Sutter. “They know that in that model that was built on a specific date, there is the problem. It wasn’t the car itself; it was the brake pad from another manufacturer. There is a flaw. They know the entire supply chain, the entire lineage of all the pieces that went into the car you bought. If that brake pad fails and the customer dies, the customer will sue the car manufacturer. Bring the car back, and we will fix your brakes.”
Tip: What is the new AI project Red Hat InstructLab?
Copy-paste
Red Hat Trusted Software Supply Chain aims to achieve the same level of control for software. In the event of another Log4Shell incident, there is the visibility to identify a flaw, providing traceability for all the components that contribute to the final product. Burr Sutter explains, “If you look at the end-to-end supply chain within that software development life cycle, there is a moment when developers write their original code. They also go to a registry and download dependencies, copying and pasting. Developers love to copy and paste, but the problem is they may paste something with a flaw. We have a plugin that tries to catch that moment of copy-paste. It alerts developers if they have introduced a potential vulnerability through the captured dependency. It warns them, and a developer can choose to ignore it.”
The developer can still commit and send the code to the source code repository. In that case, Red Hat’s pipeline solution runs a check. The solution notifies that there could still be a problem. Sutter adds, “When you commit that code, you sign for the code. We know who made the commit and who made the code change. The bad dependency – there is an acknowledgement of who introduced it. There is traceability, allowing a company to ask the developer to stop introducing vulnerabilities into the codebase.”
After the developer verifies the commit from the repository, Red Hat supports the Trusted Software Supply Chain by focusing on the container image. Sutter explains, “When you build that container image with the application code, including possible bad dependencies and your business logic, we also build a secure environment. It signs, tests, and checks the software bill of materials (SBOM) of that. The signature shows it was built in a specific environment, on a specific laptop. That destination is also the mark that displays all the attributes in which it was built. It provides metadata on how the built environment looked on that day. This helps you understand if the development environment was secure at that time.”
The ingredient list
Regarding the software bill of materials, Sutter emphasizes its usefulness. It is now enforced in many places to use SBOMs. The documentation of the Software Supply Chain provides more knowledge of responsibilities. With the SBOM, a company knows if the product was built properly, at the right location, in the right way, and what went into it. Even if it has a bad dependency, the SBOM indicates its presence.
To support the SBOM, Red Hat offers two additional solutions within the Trusted Software Supply Chain. First is the Quay repository, a container registry for software artifacts. These artifacts are visible, allowing the company to track their location and transfer files place-to-place. This way, it is possible to ship the SBOM and the signature. Additionally, Red Hat provides the Trusted Profile Analyzer tool, which takes the SBOM and stores it. The storage of the SBOM enables companies to know when ingredients are added to the software. Trusted Profile Analyzer allows users to search the ingredient list, proactively monitoring SBOMs to identify potential issues like Log4Shell. It alerts users when such incidents occur and provides insights into how often it is in the infrastructure.
Sutter concludes that working this way represents a significant improvement compared to the traditional method. “Normally, you would use scanners and reverse engineer from the binary for what they think is in there. That is not as effective because some dependencies can be hidden. SBOM makes it more visible and human-readable, allowing people to reason about the file.”
The developer’s developer
Another significant challenge that Sutter highlights is the absence of an effective onboarding process for developers. He shares insights from a customer conversation with a large bank based in the UK, which faced onboarding challenges. “They discovered that when they hired someone new, the new person received a laptop, VPN access, email address, and a door key—everything needed to be an employee. However, the issue was that the new employee wasn’t productive in writing code for at least 60 days,” explains Sutter. Despite possessing the skills to write code, the developer faced challenges because the software development life cycle at the new workplace significantly differed from their previous experience. “Each SDLC is unique per company, and in some cases, there are several SDLCs within a company, with teams having their own individualized SDLCs. Therefore, a new employee must learn how the company builds software.”
To address this issue, Red Hat has developed Red Hat Developer Hub. Sutter describes it as the “developer’s developer,” a tool for setting up the onboarding environment. The portal provides everything a developer needs, including all documentation and context, such as the location of git repositories. Users can experience the SDLC through that template, click through, and see the company’s unique pipeline,” notes Sutter.
According to Sutter, utilizing Red Hat Developer Hub can significantly reduce the time it takes for a new developer to start writing code after joining the company from 60 days to 6 minutes. This minimises time lost and ensures that innovation can proceed because developers can start working immediately, preventing unnecessary salary loss during the initial 60 days. Sutter also points out that in companies where a 60-day onboarding is the norm, developers might even leave within that timeframe for a competitor offering a better onboarding experience.
Reducing onboarding time and improving the software supply chain significantly enhance a developer’s job. These two topics unquestionably deserve attention in an era when software is transforming industries.