Why you should keep your dependencies up to date

Most of us are usually happy to have the latest updates to our components because it means we also have the latest features, but there’s more to gain from regular dependency updating than new features. Keeping packages up-to-date means improved development velocity in both the near and long term. Over time, outdated dependencies accrue technical debt that only gets harder to reduce as time goes on. There are security concerns to consider as well. If a critical vulnerability is found in a package that you’re many versions behind on, and only fixed with the latest version which has conflicts with the rest of your codebase, you may have a lot of work (and time) ahead of you before that vulnerability can be patched.

Why dependency management gets ignored

If updating dependencies is such a great thing, why does it sometimes live permanently on the backburner? Sometimes the pace of development is so great that devs simply don’t find the time for it and in the chaos it can be difficult to explain to a boss why you’re “fixing” something that’s functioning just fine (fine for now, anyway). The longer it goes, the harder an update can be to make, which then in turn makes putting it off even longer the easiest choice, if it’s still a choice that’s possible to make.

OK I’m convinced. How do I get started?

Here are some tips for motivated developers who want to make regular dependency updates as painless as possible:

1. Start small. Getting started is the hardest part–it only gets easier, promise–so start small enough that you can handle the initial load entirely on your own.
2. Spread the good news to everyone else. Your teammates will likely notice and appreciate your initial effort and embrace the new way of doing things. Other teams will see the value as well. Have a talk with management so they know what you’re doing and why it’s important.
3. Automate and build into your pipeline. There are free tools available through both GitHub and BitBucket that automatically locate outdated dependencies and create pull requests for updates. Some tools, like Mend Renovate Community, have features that group pull requests by type and rate the likelihood a given update will merge without any problems, so you can spend less time poring over release notes.
4. Start new projects with dependency updating already in mind. If you’re starting something fresh, put the effort in to get the tools and structure in place to keep dependencies up to date from the get-go. Who knows, maybe in seven years your app will be the “old app” that’s still so up to date no one is afraid to touch it.

Hopefully you find these tips helpful. We have found that the minor pain of introducing regular dependency updates is more than worth the gains of reduced technical debt, new features, and increased security, and we think you will, too.

Until next time, stay up to date, friends!

1. Malicious packages: a different kind of threat

Malicious packages are a different kind of threat because they target you directly. Vulnerabilities may or may not be exploited. They may be accidental. Malicious packages are intentional. They’re specifically designed to disrupt or disable targets, and you need to proactively detect and block them to avoid them contaminating your code base.  

You should be extra cautious about the threats they pose and what you can fall prey to, such as malicious hacks such as event-stream or Circle CI. This holds particularly true in npm or the JavaScript ecosystems — the most popular programming language for open source projects and therefore most prone to vulnerabilities and malicious packages.

Realistically, developers can’t spend tons of time reviewing every single open source component. The smart way to spend AppSec dollars is to focus on implementing a secure SDLC, keep abreast of alerts, and respond quickly to threats that arise, in line with the application security procedures that you already have in place. 

2. Watch your adversaries 

Attackers rely on their targets not paying full attention or keeping track of what they’re using. When you inadvertently open or download a malicious package, they gain access to your code base and data.

So watch your adversaries and understand their methods. Then you can better anticipate them and take measures to thwart them.

For example, Google’s Project Zero team releases information about the chatter they hear on the dark web concerning which vulnerabilities and packages to look for, and they release it as an intelligence feed. They aim to give users the intelligence to help take preventative measures against such threats. It’s valuable information because if you know what attackers are doing, you know what you need to fix or prevent before you fall victim to a cyberattack. 

That information also helps you comprehensively check and document your software supply chain, starting with a software bill of materials (SBOM). It’s important to check frequently and regularly. You can have an exploitable vulnerability in your system for years and never be affected, but the moment that you download that malicious dependency, you’re impacted. Don’t let this happen. Be consistently vigilant throughout the software development lifecycle (SDLC).

3. Pay attention to package provenance

The community relies a lot on ecosystem providers to provide protection when issues arise. Our advice?  Don’t wait for their intervention. Be proactive about identifying the provenance of packages.

For example, we recently issued a wave of malicious package notifications to our customers. In one case, a customer notified us of a false positive because there appeared to be nothing malicious in this package in the particular release 4.12 on GitHub. However, the attacker had intentionally put one lot of code on GitHub and different code in the actual package that was published.

It’s not necessary to check everything, provided you can at least verify that the code that you are consuming comes from the source it claims to come from. Then you obviate the need to analyze binaries or artifacts.

4. The importance of industry collaboration

The issue of package provenance raises the question of whether you can trust certain open source packages and their maintainers. That requires collaboration between the security industry, vendors, and researchers, to better understand what vulnerabilities are out there and what malicious packages should be removed from the repository.

Repository managers are good at removing these, but working with them to react quickly is only effective when everyone pays full attention. In the past, when researchers found malicious packages, the repositories were notified, but packages could sit there for days and get downloaded thousands of times before action was taken. We must work faster, together.

5. Keep up with best practices for the repository

Public forums like the Open Software Security Foundation (Open SSF) or the Linux Foundation have central repository groups that introduce best practices for the repository because that’s the distribution point to the global development community and an important part of keeping the whole supply chain secure.

Repositories handle a huge volume of downloads annually — billions, even trillions in some cases. Maintainers of those repositories are working towards improving their responsiveness. Maven Central is probably the model that everyone would like to emulate. And those are specifically Java open source components. They have the most rigor behind them. Once npm was acquired by Microsoft and GitHub, things improved in that repository. There is still much work to do, and that’s what the Open SSF working group is focused on.

Companies need to play their part too. For example, in Canada, Shopify has a bug bounty. Its AppSec team writes in Ruby and uses Ruby on Rails. They constantly fix bugs within RubyGems and push the fixes into the gems for anyone to use. They consider it their responsibility. However, many companies use open source but don’t give anything back.

6. The value of dependency management

Some developers might argue that there’s little point in updating dependencies to a new version of a package if their code is working acceptably. However, updating your dependencies is like going to the dentist. It’s less drastic if you do it regularly. If you do it every four years, you have to rectify four years of neglect and it’s going to hurt. In security terms, that’s technical debt. Mature software teams understand they need to stay updated.

And that’s when you get a choice about updates. When dependencies are transitive, there isn’t a choice about which are pulled in. Solving problems arising from them relies on the package maintainer to bump them. You can’t do it yourself.

In these cases, some customers feel like they lack control because a malicious dependency could hit them despite following best practices. Or, a vulnerability is disclosed and a new version is released, but they can’t get an upgrade to that version because the direct dependency hasn’t been upgraded yet.

7. Automate to handle volume

Although open source can be exposed to vulnerabilities and malicious packages, the benefits still outweigh the risks with the right security strategy and tools. Ultimately, keeping it up to date is the best practice, and that’s where automation comes into play. 

Take, for example, Mend Renovate. It’s a dependency update automation tool that scans your software, discovers dependencies, automatically checks to see if an updated version exists, and helps you by submitting automated pull requests.

Importantly, a tool like Renovate can handle volume. Millions of developers downloading trillions of packages per year require tooling and automation. Automation is key, especially as attacks will get more sophisticated as we get better at detecting them.

8. Handling CI/CD pipelines to remain secure

The good news is that we now have tools that can be injected into your CI/CD pipeline to thwart malicious packages. And they’ll start to become a standard part of security tooling.

One issue to address is centralization. Companies assume  centralizing open source downloading makes them more secure. But really, they’ve just centralized the path through which malicious packages can be downloaded. You still need to actively stop the malicious packages. However, developers don’t download from the same code repository. They use a range of CI/CD pipelines and tech stacks. That’s challenging to police. Ideally, everyone would use the same CI/CD but that’s unlikely, so the compromise is to use a tool that works with different CI/CD pipelines.

Evolving solutions as threats evolve

There’s no magic recipe yet to completely cure security vulnerabilities and eradicate malicious packages. You need to implement a strategy that combines an approach and tools that work best for you. 

What’s undeniable is that we all face the challenges of a growing attack surface and fast-changing attack methods. This means we must all remain nimble in our responses to evolving threats. Start by using the tools and technology we currently have — but keep in mind that further improvement will come as technology evolves.

This means that regularly maintaining and updating dependencies is crucial  to ensuring application security. Developers and DevOps professionals must be aware of the risk involved in using outdated dependencies, as well as the benefits and best practices involved in updating them. Let’s look at the five key security best practices and benefits for maintaining up-to-date dependencies.

1. Patching vulnerabilities

Software vulnerabilities are generated and discovered on a regular basis, and outdated dependencies can pose a significant risk to the security of an application. Developers can address these vulnerabilities promptly by applying patches or updates released by the dependency’s maintainers. These patches often contain critical security fixes that mitigate vulnerabilities, reducing the attack surface and enhancing the overall security posture of the application.

Failure to promptly update dependencies can expose an application to vulnerabilities that malicious actors can exploit. Attackers actively scan for applications running outdated dependencies and leverage known vulnerabilities to compromise the application’s security. By ensuring that you have the most recent dependency updates, you can minimize the risk of these vulnerabilities being exploited and reduce the chances of a successful attack.

2. Compatibility and interoperability

By their very nature, dependencies are interconnected pieces of software that must work together seamlessly to ensure that an application functions properly. As developers introduce new features, update the underlying framework, or make architectural changes, dependencies may become incompatible or experience compatibility issues with other components of the application.

Therefore, keeping dependencies up to date helps maintain compatibility and interoperability between different software components. By using the latest versions, you’ll get relevant bug fixes, improved performance, and enhanced features that enable integration with other parts of the application. Outdated dependencies, on the other hand, may lack support for newer frameworks, languages, or APIs, hindering the application’s performance and introducing potential security weaknesses.

3. Dependency vulnerability scanning

Maintaining up-to-date dependencies is not just about applying updates. It also involves regularly assessing the security posture of the dependencies themselves, by scanning for vulnerabilities. Dependency vulnerability scanning tools and services can analyze which versions of dependencies are used in an application and provide insights into any known vulnerabilities associated with those versions.

By scanning dependencies for vulnerabilities, developers can proactively identify and remediate security issues before they are exploited. These scanning tools leverage public vulnerability databases, security advisories, and other sources of vulnerability information to identify potential risks. They can provide notifications or reports that highlight specific dependencies that require updates to address security vulnerabilities.

4. Improved stability and reliability

Updating dependencies helps developers fix bugs and address security vulnerabilities, thereby contributing to the stability and reliability of an application.

By staying up to date with dependencies, developers can provide a more secure and trustworthy application experience for end-users. This, in turn, helps prevent potential crashes, performance degradation, or unexpected behavior that could be exploited by attackers.

5. Community support and maintenance

Popular dependencies typically have active communities of developers and maintainers who contribute to their development, maintenance, and security. These communities monitor and address vulnerabilities and provide timely updates and patches when needed. By keeping dependencies up to date, you leverage the ongoing efforts of these communities, benefiting from the collective knowledge and expertise of a wider network of developers.

Be aware that outdated dependencies may no longer receive security updates or maintenance, leaving the application exposed to known vulnerabilities. By using the latest versions of dependencies, you can ensure that you’re not missing out on critical security enhancements provided by the community.

Updating dependencies is a must

Neglecting to update dependencies can lead to security breaches, unauthorized access to sensitive data, and compromised user trust, making dependency management a vital application security practice. Regularly reviewing and updating dependencies should be an integral part of the software development lifecycle, accompanied by thorough testing to ensure that updates do not introduce new issues. By prioritizing application security and staying vigilant in keeping dependencies up to date, you can stay one step ahead of potential threats and build robust and secure software systems.

What’s taking security so long?

DevOps has been a recognized methodology for more than a decade, but organizations have been slower to embrace the security aspect. This may be a matter of incentivization, prioritization, and measurement. If your KPIs and the way you measure success isn’t based on security, then it won’t get prioritized.

There’s a large swath of tooling and responsibilities that DevSecOps encompasses and it isn’t always clear where these responsibilities lie. That’s because different companies place these responsibilities on different teams, all of whom are expected to roll into the DevSecOps principles and deliverables.

Certain companies, especially in regulated environments, have no choice but to ensure that security is baked into their processes. But if security isn’t already part of your developer workflows, it may take time to be adopted. What’s needed is clarity about responsibilities and more collaboration, especially if your company traditionally separates security from DevOps.

1. Bridge the divide between DevOps and security teams

Collaboration is important because it builds a successful DevSecOps culture across teams. Figure out how to put teams together to solve common problems. An example is introducing changes with pull requests, requiring reviewers from developers and security teams. If you’ve got the bandwidth, assign a security expert to the DevOps team and vice versa, so they understand each other’s challenges and how they go about solving them. Ensure that your DevOps team has KPIs linked to security. It amplifies the importance of security.

2. Control and automation

Collaboration presents its own issue among security teams: the issue of control. Much of the security integration into DevSecOps requires security teams to relinquish control and let developers and DevOps teams do the integration. That’s not easy for them, but a more collaborative approach requires sharing control, and figuring out the separation of duties, and how to ensure things are properly implemented.

Control is naturally averse to automation. DevOps engineers are by default people who embrace automation. It’s how DevOps has been able to scale and really accelerate. However, security teams have been slow getting into that game and still tend to do a lot of things manually, and reactively. It can be a big lift getting security teams to think of automation as a way to scale and get ahead of security problems, but it’s essential for scaling security reliably and delivering a robust and future-proof security strategy.

3. Balance speed and agility with risk management with automation

Businesses understand the value of agility, of being quick to fix bugs and introduce new features. We have to think about how security can be agile. This is where automation is key.

Automation is what our most mature customers focus on in DevSecOps because it improves the efficiency of your security tools. It means you can trust your DevSecOps. This is important when we consider that typically, security engineers are heavily outnumbered by DevOps engineers, so it makes sense to automate security testing and policies so that you can continue to develop software at pace.

When there’s usually just one security engineer for every fifteen DevOps engineers, automation removes the burden of manual scanning, checking, and remediation, and frees them to focus more on emerging threats and more complex problems. Automation helps security teams ensure that their business can keep moving at the speed it wants to.

4. Observability

Observability is critical to DevSecOps because you can’t secure what you can’t see. Successfully achieving it involves identifying issues and their potential problems earlier. Shift left closer to the source and before components are deployed or go into runtime. Then, you can find and address any issues before they have any impact. This optimizes the efficiency of your DevSecOps.

Nevertheless, after you deploy you may still find new vulnerabilities and problems, which feed back into the pipeline, and you need to establish who in the development circle needs to solve a particular problem. A lot of companies struggle with figuring this out. The answer is what we’ve called shifting smart, which is simply applying the practice of iterative scanning, testing, and remediation throughout the software development lifecycle (SDLC). This elevates the importance of observability.

In Kubernetes for example, observability plays a huge role. If you are dependent on your policy agents that you might have running in your cluster or scanning agents, you want to make sure that those services are available and successfully serving requests. Observability becomes part of the prerequisites of components that should be deployed to your system so that you have the necessary visibility to secure what you can see.

5. Avoiding alert fatigue

If you see more issues, you risk suffering from alert fatigue. You can avoid this by generating fewer alerts and reducing false positives. Software composition analysis (SCA) tells you whether you’re using a dependency that has a known vulnerability and if you might need to upgrade it. At Mend.io, we make the solution more mature with reachability analysis. This ascertains if a dependency is reachable or exploitable. If it isn’t, then don’t waste valuable time and resources addressing it. It reduces the number of alerts.

Automation also reduces the impact of alerts. Ultimately the reason people get alert fatigue is because alerts trigger them to take some type of action. When remediation is automated, fixing alerts is no longer a burden.  A better understanding of context helps you establish what’s critical. You may find a vulnerability with a high CVSS score, but is it in development, production, or in a runtime environment? Is it connected to anything else? Is it public facing? Is the component actually executed? These considerations affect how impactful a vulnerability could be and whether it’s worthwhile addressing. Contextual knowledge helps you identify when a particular risk should be prioritized, so you only focus on what’s necessary.

6. Legislation and regulation

There’s been a flurry of legislation lately around the world: the Cyber Resilience Act in the EU and the national cybersecurity strategy in the U.S. among others.

Regulation isn’t new in highly regulated environments, especially for government-related projects. But regulations will increase the importance of security strategy and accountability in all organizations. Software bills of materials (SBOMs) will be required to ensure that zero trust and software supply chain security are at the forefront of any projects. This new vigilance will drive more widespread adoption and maturation of workflows, especially those that demonstrate secure delivery of applications with increased velocity via automation.

Furthermore, regulatory compliance is a great way for security leaders to get their teams to adopt best practice. Ultimately, it’s a positive way for them to affect change that will help everyone get ahead of security problems.

7. Impact of AI and generative AI on DevSecOps

Perhaps the biggest risk of AI at the moment, but it’s hopefully something short-term, is what’s known as hallucinations in AI. That’s when AI is sometimes very confidently wrong and you can’t always trust it. With large language models (LLMs), in particular with generative AI, think of AI as an advisor and not a solution. View it as a way to make suggestions and generate ideas not necessarily seen before. You can use LLMs like GPT4 to give security teams an option on alerts to generate an idea of how they might solve them. But take note. Presently, proceeding with an AI-generated solution without due diligence could be risky. Some people will implement AI incorrectly and cause spectacular failures. Nevertheless, AI offers many benefits, especially when it comes to speeding up processes like scripting infrastructure as code, for example. My advice is: embrace this technology but be circumspect when using it.

1. Analyze your needs.

For many companies, regulatory compliance drives the need for an SBOM. Software supply chain security has become an issue to which numerous international governments and legislative bodies have recently turned their attention. In the U.S.A., for example, the White House Executive Order (EO) 14028 requires federal agencies to get SBOMs from all vendors and supplies, who are mandated to document components and processes, to demonstrate that their code is secure and legally compliant.

Throughout the world, industries handling sensitive data, like healthcare and finance, and any companies conducting mergers and acquisitions, or those trading as public companies, are obliged to be transparent, and this transparency requires the use of SBOMs. Companies from all industries that use or develop, produce, purchase, or operate software and applications will benefit from using SBOMs. SBOMs give them far better visibility and understanding of their software supply chain. They enable companies to check the components of their code base and ensure that they are updated and protected from attacks.

2. Don’t use a wrench when you need a hammer.

SBOMs have a bad rep for being difficult and time-consuming — but the real problem is that many companies are using the wrong tool.  Using email and spreadsheets is cumbersome, slow, and prone to human error.  And frankly, they can’t keep up with the rapidly escalating volume of components and dependencies that need cataloging. Most tools are not suited for managing SBOMs. Even when they disclose vulnerabilities, they don’t necessarily show how to address them. They may only cover a specific set of programming languages or package managers. Many tools also can’t generate SBOMs automatically in a machine-readable format, and they can’t scale up easily. 

If you want to generate SBOMs quickly and reliably, use a tool that’s purpose-built to do so. Look for a tool that reveals all open source libraries, tracks and documents both direct and transitive dependencies, and automatically updates whenever a change is detected in any component. Good SBOM tools also recommend ways to remediate any vulnerabilities that don’t risk breaking your build.

3. Plan for automation and continuous remediation. 

The White House executive order requires software inventories to be automatically generated in a machine-readable format, so look for tools that support full automation, particularly if doing business with federal and governmental bodies.  Management of your software supply chain is far more efficient and effective when your tool continuously discovers and addresses any risks. It should help avoid harmful components and send alerts when it detects anomalies during development or production. It should also be capable of identifying risks in downstream components.

4. Look for comprehensive coverage. 

Your tool should provide an up-to-date and accurate inventory of all your supply chain components that covers all the technology stacks and programming languages that your organization uses. It should also support standard SBOM formats. If your tool isn’t comprehensive, then you could generate SBOMs that do not provide a complete or accurate view of all of your software, components, and dependencies.

5. Keep it simple. 

Your tool should be simple to adopt, ideally working within your regular workflow to operate as seamlessly as possible. It needs to be scalable and versatile, so it can handle a growing volume and variety of components and produce low false-positive rates.

Wherever possible, we create integrations that overcome this problem. As we mark the start of Atlassian’s Team ’23 event, we are excited to announce a new enhancement to Mend’s Jira integration capabilities that supports “Jira Security.” Now Jira users will have a single place to view and triage security alerts from mixed security vendors.

Why integrate with Jira?

Support for Jira is a very important consideration for many of our customers. Developers are often asked to use Jira, and many security professionals already prefer it as their central record of security issues. As such, it has become a vital tool for application security purposes, alongside tools such as ours.

With this in mind, Atlassian, the producer of Jira, has sought to make security in Jira a primary focus, instead of a bolt-on. As Mend works to deliver the leading AppSec solution on the market, it stands to reason, indeed it’s critical, that we give customers new capabilities to manage and triage security findings in Jira, and this dovetails with our mission to respond to customer needs by providing cutting-edge integrations for them, that make their work faster, easier and more secure.

Benefits

According to research by Atlassian, the average Jira customer has around three security vendors who push data to Jira or would like to. This situation poses the problem of how users can easily view the data that the vendors provide, without the need to switch between different pieces of software. Plus, there’s a need to make it as simple as possible to collate and compare information side-by-side. By viewing all vendors in one place, using the integration with Jira, users will save valuable time and resources when they’re security scanning. And now, developers will enjoy more flexibility and choice to secure their software and applications when using Jira.

Additionally, the integration enables users to find and fix issues and vulnerabilities quickly and early in the SDLC. Integrating Mend enables users to send security findings directly to Jira Security, and Mend users will now be able to adopt and implement cutting-edge capabilities from Jira so that they can better manage their security more easily.  The new ability For Jira Security users, the integration enables them to triage Mend findings in Jira Security before they become serious issues. For both Mend and Jira users, the integration accelerates the early detection and remediation of vulnerabilities that expedite security processes by anticipating and addressing issues before they can compromise your code base.

Keep yourself at the cutting edge of application security

This new enhancement of Mend’s Jira integration capabilities reinforces Mend’s position at the forefront of the application security field. Not every application security provider can offer such capabilities and integrations, nor constantly seeks to introduce new ones. When you’re considering which application security vendor to choose, be sure to ask the vendors on your shortlist how they will provide you with an agile and innovative security solution that can adapt as your needs grow, and change, and that can easily integrate with vital tools like Jira.

Known variants of the existing exploit rely on a few specific conditions. However, the conditions are fairly common and as time goes on, experts believe new forms of attack will emerge.

Spring4Shell: What to expect

Due to the widespread use of the Spring Framework and the severity of the vulnerability, CVE-2022-22965 has been given a critical (CVSS score of 9.8) rating. The Spring team emphasized in an announcement that “the nature of the vulnerability is general, and there may be other ways to exploit it”. 

While the new versions of Spring Boot resolve all known vulnerabilities, additional vulnerabilities may still be found. The December 2021 Log4j incident provides a recent example of the emergent nature of vulnerabilities. Starting with CVE-2021-44228 published on December 10, 2021, and followed by a string of vulnerabilities that were published within days of each other. 

As with the Log4j vulnerabilities there are likely to be numerous targeted attacks, necessitating numerous patches. More updates are sure to follow as new ways to exploit the vulnerability come to light. It is therefore extremely important to monitor what’s happening and be aware of relevant dependency updates. Dependency automation tools such as Mend Renovate will monitor updates and notify you automatically.

Learn more about Spring4Shell Zero-Day Vulnerability

Spring4Shell: How to prepare

Experience with the Log4j vulnerabilities has taught us that the best-prepared companies were able to completely mitigate and remediate their risk within hours of the announcement. Their success in dealing with the challenge can be attributed to their dependency management automation. Implementing dependency management best practices pays off in fire drills such as Spring4Shell, as companies can upgrade dependencies with the click of a button. Otherwise, understanding where spring is installed and then analyzing the risk can take hours upon hours. 

Using tools such as Mend Renovate to automate dependency updates, combined with Merge Confidence data crowdsourced from hundreds of thousands of repositories with Renovate installed, can cut the time needed to confidently update all vulnerable occurrences. from days to minutes.  

Mend Renovate can help

Mend Renovate is a dependency update automation tool. It scans your software, discovers dependencies, automatically checks to see if an updated version exists, and helps you by submitting automated pull requests. With more than one hundred million downloads, Mend Renovate has already identified and mitigated the Spring4Shell vulnerability on thousands of repos around the world.

As a response to Spring4Shell, Mend Renovate was updated to open a PR containing a fix to SpringShell for users with repos containing the vulnerable spring libraries. Early data at Mend shows that only 33 percent of affected organizations have already fixed some or all occurrences of the vulnerability. More than 90 percent of the affected libraries are brought into applications as transitive dependencies, meaning they are a dependency of one of the explicitly declared dependencies of the application. As a result, many vulnerable organizations may well be unaware they are affected by Spring4Shell, as they don’t see any Spring libraries in their dependency list.

Incorporating dependency automation into your workflow ensures agility and reduces risk in future situations where you need to act fast. 

To remediate existing Spring-related vulnerabilities and mitigate the risk of future zero-day vulnerabilities, install Mend Renovate. It’s free.

The announcement of Log4j vulnerability cve-2021-44228 sent security and development teams into a tailspin and highlights the one of biggest challenges of open source security: dependency management. The open source libraries that make up approximately 80% of our applications are often a tangled web of dependencies. If tracking all of the open source components within a codebase is a challenge, tracking the direct and indirect dependencies is almost impossible to do without smart automation. 

While AppSec and development professionals agree that keeping dependencies up to date is crucial for secure coding, far too many prefer to wait until a “critical” update comes along before they take action. This “If it ain’t broke” attitude is far too common, and as a result, keeping libraries updated is often given a lower priority in favor of putting out the fires of the day. 

The recent Log4j vulnerabilities don’t leave any room for delay. Organizations can’t afford to add Log4j remediation to the bottom of developers’ to-do lists. 

Introducing Mend Renovate and Remediate for Log4j 

In order to help development and security teams address this challenge, today we are releasing a remediation preset for Mend Renovate and Remediate — included in both our free and commercial products — which enables users to identify and fix the Log4j vulnerability from hundreds of downstream dependent packages of Log4j.

This remediation preset helps address the challenge teams have been facing when it comes to open source security fixes in general, and that the Log4j brought to forefront updating indirect dependencies. Many packages in the Maven and Gradle ecosystems use Log4j, so remediating it requires more than just upgrading Log4j in direct dependencies — it may also require upgrading multiple indirect dependencies.

Mend has found hundreds of packages in use by our customers which have both vulnerable and fixed versions available, and we have generated a set of rules to identify and fix these.

The list is by no means exhaustive. For example, it does not include certain older versions of packages for which we’ve seen no recent use. We will continue to update the list both:

• As we improve algorithms to detect more packages and fixes, and
• As more open source packages themselves remediate Log4j and publish a new version, making them eligible for this remediation list

Additionally the list itself is open source so contributions and corrections are welcome.

Usage instructions for the preset can be found within the repository. The preset is available for all editions of Renovate (Open Source, the Mend Renovate App, and Mend Renovate On-Premises) as well as Mend Remediate, which is part of Mend’s commercial offering.

As news of new Log4j exploits continues to dominate headlines, it’s crucial for developers using Log4j to have the ability to quickly and confidently update Log4j to a secure version. Mend Renovate combined with Merge Confidence helps developers support that strategy.

Learn More: Get free tools to detect and fix Log4j vulnerabilities at our Log4j Vulnerability Resource Center.

At Mend we believe that open source risk management is a pillar of software supply chain security, and Diffend helps us extend our capabilities in this area. While 99.999% of open source releases may be safe, our customers trust us to help identify the ones that could do harm and should be avoided. Enter Diffend.io: an industry-leading tool with several innovative features. 

The last year has shown that malicious actors will go to great lengths to compromise software supply chains, and open source risks need to be kept in check. We know that burying our heads in the sand and never updating a dependency out of fear is not a viable strategy, especially considering that pretty much every software development outfit regularly starts new projects with latest versions of open source dependencies anyway. Companies with critical projects depending on open source need a solution they can trust to deeply evaluate open source package risk and steer them clear of it.

What is software supply chain security?

Software supply chain security is a trending term in software development circles lately, for unfortunate reasons. If you think of a company’s traditional physical supply chain, for many it might take an Oceans 11-style scheme to penetrate and exploit. Software, on the other hand, which almost every company in the world today needs to build in some form, has many avenues of penetration which are difficult to secure.

Although the SolarWinds hack did not directly involve open source, it’s the most widely known instance of software supply chain security. Attackers gained access to a build system, used that to compromise published software, and then were able to use this to compromise over 100 other large companies who were users of the software, plus multiple government agencies. 

Supply chain incidents haven’t stopped there. Just last week, the popular code coverage company Codecov disclosed that attackers had compromised one of their install scripts, which in turn could have been used to exploit many of Codecov’s 10’s of thousands of users.

These incidents show a trend where attackers target components of a company’s software supply chain to gain access and then leverage such access for greater gain — causing greater damage to the unfortunate victims.

Software supply chain security and open source

There’s no debate that securing open source is an essential part of securing the software supply chain. Research has shown that most software projects are comprised of 60%-90% open source in the final product, which is a large potential attack surface. But more importantly, open source software has much less inherent protection to begin with. It’s great that there’s no gatekeeping around who can publish open source, but our risk management needs to be calibrated accordingly.

Why Diffend.io?

At Mend we look at open source risk holistically rather than purely from a security perspective. In 2020 we launched Merge Confidence, which reduces the risk of unintentionally “breaking” updates in open source, which is more common by occurrence, but thankfully much less impactful than a malicious act. When evaluating our progress towards automatically detecting and blocking malicious dependency updates, we quickly realized that Diffend.io was far ahead of the industry, thanks to the tremendous foresight of its founder Maciej Mensfeld and his successful collaboration with Tomasz Pajor. 

In fact, since its public launch in early 2020, Diffend’s combination of automatic scanning with a system that flags packages for review hasn’t missed a single known malicious package on the Rubygems registry, and has been directly responsible for the majority of such packages being “yanked”, which in turn protects and entire community. Just last week, Diffend helped detect and remove 60 packages from Rubygems which were posing as innocent “security research” like Dependency Confusion, but actually opening up malicious backdoor capabilities. 

A few more examples of Diffend’s innovative capabilities and ability to plan for future threats:

• Diffend detected and blocked the Dependency Confusion research gems months before this research was made public.
• Diffend was already built with capabilities to address Dependency Confusion directly by specifying private dependency patterns which should never be downloaded from the public registry.
• Diffend even has capabilities to detect and alert when dependencies in production get “yanked” from the registry. Diffend users were warned within 15 minutes of the mimemagic incident earlier this month.

Diffend and Mend

We’re especially happy to not only have acquired Diffend but also to have its Founder Maciej Mensfeld join our team as Senior Product Manager for Supply Chain Security. Maciej’s vision in creating Diffend and foreseeing these types of security risks has been unparalleled. We’re excited to have him join to expand Diffend’s capabilities cross-language as well as integrate Diffend data and capabilities into Mend’s enterprise open source security capabilities.

Existing Diffend capabilities will remain free, so you are encouraged to sign up and create a free account so that you can be the first to know about the new languages and capabilities we will be rapidly adding throughout 2021. All capabilities will also be integrated into Mend’s Enterprise product range, including our advanced reporting and policy controls.

Welcome Diffend, and welcome Maciej!

Installation and enablement

We host the Renovate source code on github.com, and so use the Mend Renovate hosted app for maximum convenience.

We use a renovate.json config file in the root so it’s easy for everyone to spot, and it extends our default organization preset in github>renovatebot/.github. We’ll describe how our organization preset works later.

Now that the repository is installed via the app, and onboarded via a config file, it means the hosted app will usually run:

• On schedule, usually every 1-2 hours
• On-demand, if anyone ticks an embedded checkbox in any Renovate Issue or PR
• As-needed if webhook activity indicates that a Renovate PR has been merged, Renovate’s config has been updated, or a notable package file has been edited any other way

Dependency update automation

Automating dependency updates is the main reason people use Renovate, and we aim to keep the project as up-to-date as possible. After all, a professional painter shouldn’t live in a house that needs a new coat, should they?

The key points about how we use Renovate to update dependencies are:

• We extend the config:base preset that all new accounts are recommended. Importantly this means that the most common “monorepos” already come pre-grouped without us needing to configure those.
• We set stabilityDays to 3, meaning that we don’t want any PRs created automatically for new releases which are less than 3 days old. There aren’t many circumstances where any project needs a release that soon, unless it’s an internal release or something you’re specifically waiting for.
• We don’t raise major updates by default. Instead, they are buffered for approval in the “Dependency Dashboard”, which is like a mini dashboard in the repository. This lets us raise them on-demand when we are ready for them.
• We defer updating Microsoft’s dev container for VS Code to once per month. It usually updates more often than that, but we don’t have any urgent need for updates and prefer to keep noise down.
• We don’t custom group package updates — unless they are from known monorepos which must be updated in lockstep. This way one bad update to a dependency doesn’t block all the other updates if they all combined into a single PR.
• We keep the default config:base settings of maximum 2 PRs created per hour or 20 concurrently, but we usually don’t get close to that at all, and try to merge PRs right away once they reach stability and pass our tests.

Organization default preset

The renovatebot organization on GitHub has a .github repository used to share common files, including Renovate configuration files. There are two files which are relevant:

• A renovate-config.json file which is used by Renovate to determine the default onboarding config whenever we add a new repository. By having this repository file in place, it means the hosted app proposes our custom config instead of the default, meaning one less step for us to perform every time a new repository is onboarded.
• A default.json file which is the org’s default preset

Renovate’s org preset also contains some pretty clever “regex manager” rules which we use to apply custom dependency updates to our Dockerfiles and documentation. Both of these are worthy of their own post!

Repository assimilation

We’re a believer that great tools should fit into a developer’s workflows, rather than request developers adjust to a tool’s workflows. For Renovate, this means adapting to a project’s conventions for branch naming, commit syntax, labeling, etc.

In Renovate’s case:

• We use semantic commits, defaulting to “chore” but using “build” for non-dev dependencies so that each update triggers our semantic-release config.
• Speaking of semantic-release, we use “build” semantic commit type — even though it’s a dev dependency, because we always want to test that any new update works and releases.
• We don’t have a need for labels, and don’t apply them. Renovate doesn’t pollute project’s labels by default.
• Renovate automatically assigns two of the project maintainers to each PR.

Merge Confidence

A great new feature introduced by Mend last year is Merge Confidence, and we rely on it heavily.

We built Merge Confidence on the premise that:

• There is always the potential that new releases contain an unintentionally breaking change or bug.
• Relying on your own tests alone is rarely enough to be certain.
• By aggregating test results and version adoption amongst Renovate’s user base, we can determine with a high degree of accuracy whether a new release is safe or not.

In practice, that means Pull Requests for new versions look like this:

Based on the above, it’s clear why the Confidence is “High” and I have no problem merging it, especially as it passed my tests too.

Downstream automation

Another aspect of how we use Renovate to build Renovate is downstream automation. It was mentioned earlier that in most cases you don’t need new releases immediately after they’re released, but internal packages are an exception that an increasing number of Renovate users utilize to improve productivity between projects and teams.

Similar to how many projects are built, Renovate is developed and published across multiple repositories. Once a new release of Renovate is made, we have repositories which build and publish a Docker Image as well as a GitHub Action. App users are often eager to try out new Renovate features right away, so we make sure there is no stability days restriction on our own packages and that PRs are created right away.

Further, there is no need for us to review or approve our own packages in this case, so we also enable branch automerge. What this means is that usually within about an hour of releasing Renovate, the downstream Docker and GitHub Action repositories have already received an automated upgrade of Renovate and are on their way to finishing the process. Because we release up to dozens of times per release, this is not something that can be handled manually.

Additionally, the Mend Renovate App also needs updating, and a similar process applies. In this case although we create the PR immediately, we do require a manual approval/merge before updating the hosted app, because sometimes there are new features which should ideally be observed post-upgrade just to make sure all goes smoothly. We wouldn’t want an accidental bug taking down the app “automatically” during an hour when nobody’s watching closely, so we use a manual approval step for upgrading Renovate in the app. This is again not something which would scale well with manual dependency updates and commits.

Wrapping up

We hope sharing our experience with Renovate  helps you find ways to tweak Renovate to your needs, including custom package updates, scheduling, stability, and workflow features. If you have any questions, always feel free to post to Renovate’s Discussion Forum.