Photo by Felix Mittermeier on Unsplash
In the recent past at Wix Engineering we migrated our backend systems to the Bazel build system. You can learn more about it here and even hear a dedicated podcast interview about the journey here. This blog post is aimed for developers who use Bazel as their build system, if you want to learn more about Bazel, start here.
Dependency Management Challenges
Managing external dependencies when you have a large codebase can be quite challenging. Using external dependencies allows us to add the functionality that someone has already implemented before by simply adding libraries to our projects. Yet, on the other hand, you also get an array of things to worry about, like licensing, security, version upgrades, or conflicting transitive dependencies.
Juggling all of the concerns can quickly become a time consuming mundane task, and in some cases even significantly slow down development of business-oriented features.
Here’s one common scenario - a web framework that depends on an older version of a utility library, while the database driver requires a newer version of it. The catch? Newer version has non-backwards-compatible breaking changes. All you can really do here is rollback your changes and find a working combination between conflicting versions of the library, right?
But unfortunately, resolving conflicts might take several development iterations, and by then some of the code might already be serving customers in production. The key principle here is to have a quick feedback loop within dependency management, so that we know about dependency issues as early as possible.
Reproducible Builds with Bazel
When the Wix backend codebase (which is more than 5 million lines of code, mostly Scala) was migrated from Maven to Bazel, the task became much easier. The Bazel build system is centered around the concept of reproducibility and its design dictates it - you can think about the build process as pure function, and about external dependencies as impure side effects which are pushed out to the build boundaries.
Bazel builds must produce the same results when the same input is given, and workspace rules are used to turn external downloads into reproducible build inputs. Reproducibility is guaranteed by keeping track of checksum changes for external artifact archives. When an artifact is downloaded, its checksum is calculated, and the next time the artifact is downloaded again, if those checksums match, we then know that the build input has not changed. Reproducible builds are cacheable and can be used to improve build times significantly. To learn more about how Bazel works, check its website.
Dependency Management Requirements
At Wix we decided to tackle the external dependency management requirements first:
Builds must be reproducible - building any revision from Git must yield the same result.
Easy to use - developers tend to seldomly add external dependencies - it should be easy to learn how to add them, and using them in code should be trivial. Introduction of a new dependency should be as simple as adding its Maven coordinates to a high level artifact list.
Scalability within a large codebase - external dependencies must be defined in a way that makes it easy to track within a large codebase, build performance should not be impacted, and the overall logic must support the growing number of dependencies.
Not impacting developer productivity - external dependency updates, deletion or additions must be efficient without significant impact on developer build times. For example, resolution of dependency closures shouldn’t take too long or trigger the redownload of artifacts if there were no changes done by the developer.
Artifact Pinning
To achieve reproducible builds we had to undergo a paradigm shift from the Maven style high-level artifacts, where artifacts are resolved upon request and end-state depends on the exact time a resolution happened. We moved to so-called artifact pinning, where artifact checksums are committed to a version control repository. Pinning/locking is a common practice in some of the build systems, and is done by resolving the current dependency closure and then committing its dependencies’ versions to a file. By using checksums, the build system ensures we produce the same result each time a build is triggered. Our pinning process generates .bzl files inside the `third-party` folder, which then is committed by the developer.
Ease of use
As with every custom system, usability is a huge concern. A typical developer changes external dependencies very rarely, and some never do. If they have to learn how to add new dependencies, the procedure should be very simple. Ideally it should be a simple list of coordinates of high-level Maven dependencies which is updated by a developer, and transforming that list into Bazel definitions is automated within the tool. We like how artifacts are defined in an open source Bazel dependency management tool rules_jvm_external, and decided to use it’s artifact definition DSL. With rules_jvm_external, dependencies are defined as a simple list:
At the time of writing we had close to 600 entries in this list. After any developer modifies artifacts, they run bazel run @managed_third_party//:resolve. This command takes care of resolving the dependency closure and turning it into .bzl files under `third_party`. Each resolved dependency gets pinned with a custom macro `import_external`, for example:
Due to historic migration from Maven, our dependency resolution system is based on Aether, which is the core library used by Maven itself to resolve dependencies. That’s another difference from rules_jvm_external, which uses Coursier to resolve dependencies.
Scaling to Support Hundreds of Developers
We have about 400 backend developers who use our Bazel repository. And so it is important that any changes to artifacts have minimal impact on the critical build path. When we evaluated rules_jvm_external it turned out that with every change it added additional 40 seconds to the initial build time. It’s a significant impact as dependencies change quite frequently (even if individual developers do it rarely, in a large organization like Wix, with over 1,000 developers, all these changes accumulate to quite a few each week).
Performance
rules_jvm_external loads dependencies via a central repository rule, which does some heavy lifting for all the dependencies at once. So each time there is a change in dependencies, it needs to be reevaluated. We came to the conclusion that we need to have more optimized loading, and to achieve that, we came up with an architecture in which each dependency has a repository rule which only gets called if there are changes in a particular dependency. For developers who do not work on the code that is dependent on the change in question, there will be no impact at all. Even if core libraries are changed, they are loaded in parallel allowing the build graph to finish building significantly faster.
Future Improvements
Despite the fact that we are happy with our current way of external dependency management, there are still many things to be improved. Among other things, we want to:
Be able to deprecate artifacts if their classes were migrated to other artifact(s)
Have a stricter control around what gets on the classpath
Be able to consume non-linkable artifacts as linkable in some places
Have a more standard way of Maven dependency support (that’s why we evaluate community efforts like rules_jvm_external)
Have a better visibility around how the dependency graph is impacted by changes in high level artifacts
Have more automation around version upgrades
And last but not least, strict version conflict resolution
This post was written by Vaidas Pilkauskas
You can follow him on Twitter
For more engineering updates and insights:
Join our Telegram channel
Visit us on GitHub
Subscribe to our YouTube channel