So you've decided to make the switch to Bazel as your build tool - good choice! Now you're planning the next step. Migrating to a new build tool, especially for a project with a large codebase, can be quite challenging. There are many existing configurations and assumptions that will need to change. This is especially the case with Bazel.
It has stricter rules on how your code should behave (e.g. network sandboxing means you can’t download something from the internet in the middle of the test), and the build configurations are much more verbose and explicit. But in return, migrating to Bazel will result in much faster and more reliable builds.
The problem
As Wix Backend had thousands of Maven module definitions for more than 10 million Scala/Java LoC, we knew the migration process would have to be automated. SInce our primary goal was to dramatically improve build times, we decided to go all the way and narrow our basic build unit from module level to a Java package level. This strategy is recommended for all cases like the one described here.
This added complexity, since we knew in advance that our pom.xml files will not contain all the information needed for encoding package-level dependencies in Bazel. To solve this, we needed to do 3 things.
The solution
First, we searched for a source code analysis tool to find all the dependencies at the file level. We found an exact fit in the Zinc Maven plugin.
Next, we created Exodus — an open source tool that completely automates the build configuration migration by:
parsing the zinc analysis files and creating a package-level dependency graph,
resolving the complete 3rd party dependency graph (with help from Aether),
outputting the required Bazel build configurations, optimized for speed.
Finally, we created migration scripts to be run in Jenkins on a dedicated server, in order to have more continuous efforts and to give us an easy way to analyse results and correct/override any issues. Of course, in your case you can start running Exodus locally, maybe only on a subset of your maven repos, as a proof-of-concept.
With the help of Exodus, we managed to migrate Wix backend code in a much shorter period of time and gain 10X faster build times.
How Exodus Works
Maven Analysis - The migration process starts by analyzing your project’s current Maven setup.
During this phase, Exodus analyzes:
The code modules found in your project, ignoring pom modules.
The dependencies between these code modules.
The resource folders these code modules contain. Out of the default hardcoded:
main\resources
test\resources
it\resources
E2e\resources
Internal Code Graph - Based on the analysis described above, Exodus creates an internal, fine-grained, code graph using the following process:
Query a code index (i.e. Zinc analysis) for a list of files and their dependencies for each code module to form a file graph.
Transform the graphs created above to a package-level graph since we don’t want to maintain targets for specific files. Rather we’re aiming for package level granularity as 1:1:1 meaning one target per one directory that represents one single package.
Remove cycles from the package graph, so it can actually be built, by finding strongly connected components and aggregating cycles to targets. The aggregator targets are declared in the first shared ancestor.
Translate the above DAG to a Bazel package graph where each package contains the relevant targets.
Write the package/target graph to Bazel using the relevant rule names and conventions.
External Code graph - Bazel does not currently support transitive dependencies of external dependencies (Maven jars as well as other Bazel projects). The following steps expand the transitive graph of your project’s dependencies as well as the dependencies defined in the organization’s “source of truth” so that you can easily depend on new Maven projects without knowing the entire graph.
Exodus builds a compile and runtime dependency graph of dependencies defined in:
Your project.
The organization’s “source of truth”.
Then it writes the above graph out in Bazel-specific form.
External dependencies are declared in the WORKSPACE file in the root of the project and in bzl files in the third_party directory.
Override Mechanisms - Exodus does a good job at automating the migration process but false positives and negatives exist, which is why we have override mechanisms in place.
Cool Exodus Features:
Creates Bazel targets for each package instead of each module for much faster builds
Supports Java code
Supports both junit4 and junit5
Supports Scala code and specs2
Flexible override mechanism
Exodus is completely open source. You can find the code and documentation here:
github.com/wix/exodus. You can also Download the latest release and try it out today on your repository!
For more information on Wix’s migration journey from Maven to Bazel check out this blog series Natan wrote: Migrating to Bazel from Maven or Gradle? 5 crucial questions you should ask yourself
This post was written by Natan Silnitsky
You can follow him on Twitter
For more engineering updates and insights:
Visit us on GitHub
Subscribe to our YouTube channel