From 60 Repos to One: How Wix Tackled Monorepo Migration - Part2

Wix Engineering
Jul 3
5 min read

Please notice: This is Part 2 of a series of articles by Zachi Nachshon. Read the Part 1 to learn how Wix began migrating 60 codebases into a 22GB backend monorepo, and what we learned along the way.

Reducing Build Durations

As a first step towards reducing the build durations, we wanted to integrate the existing target selection that we have in the virtual monorepo builds into the real monorepo build process.

Target Selection

This methodology enables developers to build only the modified parts of the monorepo rather than the entire codebase. This approach offers dramatic efficiency gains over full builds, especially in large codebases.

When selecting targets based on what changed, Bazel performs incremental builds that significantly reduce build times by focusing exclusively on affected components.

This methodology enables building only the modified parts of the monorepo, along with any affected sources.

For example, when making a change to a target like //my/library, we need to build both the dependencies it uses (like //framework/utils) and the targets that depend on it i.e //other/service/using/my/lib. This ensures that we haven't broken anything along the dependency chain.

Benefits:

Build: Only the Bazel targets linked to the source change are selected.
Velocity: This results in a shorter feedback loop.
Build Duration: The overall build time is reduced due to the selection of fewer targets.

Target Selection v1 - Naive Approach

The open-source GitHub project Tinder/bazeldiff is being used in our virtual monorepo builds. While it works well for smaller repositories, we've found that it has limitations when used on a large-scale monorepo. For example, a small PR with minor code changes that typically builds in a couple of minutes might require a wait of approximately 15 minutes for target selection using Tinder/bazeldiff.

Using the naive approach, the CI build pipeline requires developers to wait a fixed amount of time while it calculates differences by scanning the entire monorepo. This means that as the monorepo grows in size, this scanning process will increasingly extend build times, creating longer delays regardless of how small the actual code changes might be.

How is bazeldiff utilized on our build agents?

We determine the difference between the master branch and the PR branch by:

Calculating hash on all Bazel targets from master.
Calculating hash on all Bazel targets from the PR branch.
Comparing the hashes of both results.

Result:

The results are promising. We have significantly decreased build duration, but this optimization has resulted in a new, longer optimization time. It seems that improving one aspect has negatively impacted another.

Target Selection v2 - Graph Based Approach

We explored alternative target selection methods, something that will allow us to avoid the penalty of waiting for a fixed amount of time even on small changes. We wanted to have the ability to query an existing graph for the Bazel targets that were affected by the PR change.

Why choose a graph DB as a solution?

The suggested solution was to use a Graph DB, which was inspired by the Bazel dependency graph itself.

The plan was to create the graph only once, separately from the build pipelines and then apply delta updates as a pre-build step during build invocation. By doing that, we avoided the computational expense of regenerating the entire graph each time.

This strategy is proving to be far more efficient than the traditional approaches that either rebuild everything or rely on file-level hash diff operations to determine changes. As repositories grow with additional targets, this intelligent dependency tracking becomes even more valuable. While full-graph regeneration may become increasingly expensive, the delta-based approach scales gracefully, maintaining fast build times even as the codebase expands significantly.

How did Wix utilize a graph-based solution on build agents?

Wix selected AWS Neptune, a high-performance and serverless graph analytics database, to store the Bazel dependency graph of the entire monorepo. The graph was incrementally updated as users committed changes.

The build agent then followed this process to select targets to build in the following way:

Identifying the modified source code from a user PR.
Transforming the modified source code into Bazel target labels.
Querying the dependencies graph to identify all targets dependent on the modified Bazel targets, while excluding from the query the Bazel targets added in this PR.

Only the Bazel targets affected by the code changes were partially built (those that depend on my change and those that my change depends on).

Result:

Success! We’ve lowered target selection duration from 15 minutes to a maximum of 20 seconds.

Additional improvements

To further improve build duration, we updated our build command with the following optimizations:

--host_jvm_args=-Xmx79g Limits the maximum heap size for better memory management during large builds
--host_jvm_args=-XX:+UseG1GC Enables the G1 garbage collector to optimize JVM performance in large-scale builds
--host_jvm_args=-XX:MaxGCPauseMillis=200 Limits garbage collection pause times to 200ms, reducing build latency
--experimental_repository_cache_hardlinks Uses hard links for repository cache files to reduce disk space usage and improve build speed
--discard_analysis_cache Discards analysis data after the build to free up memory
--experimental_remote_discard_merkle_trees Reduces memory usage by discarding Merkle trees after remote caching
--local_cpu_resources=15 Limits Bazel's used cores to optimize parallel build performance without overloading the system
--local_ram_resources=79000 Limits RAM for local build processes, enhancing memory availability for resource-intensive builds

On the road to production

The monorepo build was complete, and the next step was to incorporate it into daily work processes. This would allow for feedback and improvement of the solution before the actual migration. Our first goal was to replace our cross-repo builds with a single monorepo build a.k.a monorepo-checks.

Cross Repo Builds

When a change requires testing across the entire distributed virtual monorepo, it can trigger up to 60 separate repository builds. Cache is not shared between those as they are separate repositories that differ from each other.

Remedy

We’ve introduced a GitHub status check that allows users to perform an ad-hoc monorepo build on their PR changes. This serves as a cross-repo alternative to building/testing changes on the virtual monorepo repositories. Users can initiate this build by opening a PR and including a special GitHub commit tag #mrc, which triggers the build on the PR changes.

This approach consolidated between 1 to 60 VMR cross repository builds into a single monorepo build, which enhanced user experience and visibility. This streamlined process provided a centralized platform for building and testing, as opposed to managing 60 separate repositories.

Improvement

Previously, a cross-repo framework change that triggered large invalidations across 60 VMR repos could have taken ~4 hours to complete, including build retries. Now, we’ve reduced that to less than 40 minutes on the monorepo which resulted in a faster feedback loop.

During the process, we also enhanced CI builds providing build information, allowing developers to have a self service experience.

Stay tuned for Part 3, where we'll dive deep into the local development challenges that emerge when working with large-scale Git index monorepos and unveil Tapas - Wix's innovative in-house tool that bridges the gap between Bazel's powerful build system and Git's sparse checkout capabilities, making monorepo development not just possible, but pleasant.

Go deeper - watch Zachi Nachshon's Wix Engineeting Conference 2024 talk - From 60 Repos to One: How Wix Tackled Monorepo Migration like a Tech Giant: