Photo by Jasper Boer on Unsplash
TL;DR
How we at Wix made the transition of our backend engineers to the Bazel build system by creating and maintaining a proprietary IntelliJ plugin, thus allowing a sane local development environment without freaking out.
LONG
Over the past 3 years, Wix CI build system went through a big migration from Maven to Bazel, which cut down CI build times drastically. You can read all about that here and here.
The goal of this blog post is to shed some light on the challenges we encountered along the way with a focus on our backend engineers' local environments, with us trying to create seamless development experience, similar to what they were used to before, but on top of a new, challenging, and some might say complicated, Bazel wiring.
Which IJ Bazel plugins are used in Wix?
Wix-specific plugin a.k.a wix-intellij-plugin. An in-house developed plugin with a set of features that allows a wiring-free and smooth Bazel experience, based on input from our own engineers which we have received during our Bazel migration. Integrates with internal tooling developed within Wix. We are considering open sourcing it in the future as well as some of our internal tooling to support it.
Fork of the upstream Bazel-IJ plugin. This way we can control the pace at which upstream changes reach our engineers’ environments. We prefer to keep the fork as close to the upstream as possible, which is the reason we've only added a handful of features to it such as Scala support, which is the dominant language being used by Wix backend engineers.
Why create wix-intellij-plugin?
Not all backend engineers share the same experience, which means not all of them have had the chance to work on diverse build systems. The common ones are Maven/Gradle for JVM-based languages, but Bazel is a whole other story.
Every new backend engineer joining Wix goes through an onboarding process, which helps them get familiar with company’s standards, frameworks, best practices, etc. The cognitive load can be quite overwhelming though. And then on top of that they have to learn a whole new glossary of Bazel terms - such as labels, target file, macros, rules, and also put to memory a set of instructions on how to do certain things. Which in itself can be quite frustrating, since build system internals is out of their getting started scope.
The Bazel build system introduced quite a few challenges to the local-dev team, most of those revolving around the sudden change to our engineers' day-to-day development experience. From the plugin point of view, the complaints were around Bazel wiring as it is something that seemed like an uncharted territory and led to IntelliJ plugin support requests to try and automate/hide certain complexities in favor of velocity.
What we wanted to accomplish
After gathering enough data we came to realize that our solution should meet our engineers at their homecourt which is within their IDE. Most of our backend engineers work in IntelliJ, and since we had the option to control Bazel by using the API from Bazel-IJ plugin, we’ve set sail and started on a journey of creating our own IntelliJ plugin.
Heads up (NOTE)
For this topic, I won’t talk about the (virtual) monorepo we have here at Wix and the local/remote cache mechanism challenges involved with it. Rather, I’ll talk about the engineer's perspective and how they felt about the Bazel build system when writing code, expecting to do so with the comfort and assurance they previously had with other build systems.
Let’s get started
For starters, let’s review the process of adding an import statement to a file when working with Maven/Gradle compared to Bazel:
Maven/Gradle
Built-in functionally with IntelliJ which indexes all symbols in a Maven-oriented project to make it available to be used
Add the import manually or write down the symbol so IntelliJ suggests an auto-import for you based on the fact that it knows how to resolve, since it was indexed already
No wiring required, source code gets picked up and compiled just by using a strict directory hierarchy i.e. /src/main/java/...
Bazel
Rely on Google Bazel-IJ plugin, making sure the file you’re using is properly synced as a target in .bazelproject view file
Understand that every file in the workspace is backed by a BUILD.bazel since Bazel is all about granular targets
Find the Bazel label which corresponds to the symbol we’re about to add to the file
Figure out the Bazel label structure according to its build unit e.g. should it have a prefix or not?
Getting familiar with Bazel macros/rule and the deps attribute so we could add our label
What if the file we’re editing is represented as an aggregated target?
You got the point… there are many steps an engineer should be familiar with to even just add simple dependency
Challenges
Think about a company like Wix that has a few hundred backend engineers, a company that keeps recruiting on a daily basis (check out our careers page) - the amount of support questions we used to get on a daily basis was overwhelming!
How did we mitigate the problem?
We’ve introduced a set of development velocity features in our dedicated IntelliJ plugin to help overcome the learning curve by granting a facade on top of Bazel wiring. We created an async learning curve by allowing an engineer to start coding while improving their understanding of the Bazel terminology and wiring at the same time.
We’re not trying to hide Bazel from our engineers, rather trying to automate the workflows around it as we believe that every engineer should get familiar with the ecosystem he/she is working on.
Our labels/symbols indexer service a.k.a LabelDex
Before we introduce our velocity features, we should talk about the core service they rely on - LabelDex.
LabelDex is a service developed within Wix that, as its name implies, indexes all classes, packages and Bazel labels within our distributed virtual monorepo. It exposes APIs for consumers - such as IntelliJ plugin - to find symbols/labels by name, get next-token suggestion, fetch dependency graph metadata and much more.
It is the engine and the critical part behind major Bazel dependency management features introduced by Wix IntelliJ plugin.
Introducing Auto dependency a.k.a AutoDep for source dependencies
For every new Bazel symbol added to a Java/Scala file that can be resolved by IntelliJ, AutoDep manages all the Bazel wiring behind the scenes by adding a Bazel dependency on-the-fly to the target BUILD.bazel file. There is no wait time since it runs in the background asynchronously.
Example use cases:
Manually adding an import statement
When using Intellij "Import class…" suggestion on newly added symbol
There are two scanning options to resolve dependencies:
Scan on every opened file - scan all symbols on every opened file which is active in the editor
Scan only on newly added imports - scan a file only if a dependency was added, thus preventing scanning irrelevant opened files
Why do we have the “scan on every opened file” option?
We want to have strict dependencies in our BUILD.bazel files rather than +1’s, it’s best practice we’ve learned the hard way, maintaining lean BUILD.bazel files with strict dependencies leads to a lean dependency graph which eventually affects build times.
Having AutoDep auto-scan helped us align the BUILD.bazel files faster into strict dependencies, usually by engineers that maintain these files' domains.
What is +1 dependency?
A dependency that is being received transitively via another dependency declaration.
What is strict dependency?
A dependency that has a direct correlation to the symbols it represents.
What if we need to align larger scopes at once rather than single files?
For that case we’ve integrated on top of IntelliJ code inspection mechanism. Just select the scope you wish to scan and choose AutoDep: Java or AutoDep: Scala.
Introducing symbols browser a.k.a VMR Symbols for 2nd / 3rd party dependencies
What is VMR?
VMR means Virtual Mono Repo, a distributed monorepo that holds together all the Wix monorepos. I won’t dive into that topic as it is better explained in an excellent Wix blog post.
A possible scenario is that a Wix engineer wishes to consume a symbol (utility class for example) from a different repository. In this case Bazel does not have knowledge of that external symbol and IntelliJ cannot resolve and suggest an autofix.
For that, we’ve created a velocity feature called VMR Symbols, integrated as a new tab in IntelliJ search everywhere panel. It lets you search for all symbols across the distributed mono repos and manage the required Bazel wiring on the relevant target file so the end goal would be just selecting which symbol to use and that’s it.
Helping our engineers with “which Bazel sync mode should I trigger?” question with AutoSync
Another velocity feature is helping our engineers with Bazel syncs decisions. Bazel allows different sync types: partial, incremental, non-incremental (full).
Each sync type affects the amount of time an engineer has to wait for it to complete, especially in a large workspace when using a broad target sync scope.
Selecting a non-incremental sync on a large workspace, for example, causes a cache invalidation which might cost an engineer a hefty amount of idle time.
For that scenario we’ve added a new ability that triggers the appropriate Bazel sync. For example - adding a new file to the workspace triggers an automatic partial sync to the relevant target it is a part of, thus preventing the usage of other potential time consuming syncs.
How are we introducing new Wix internal Bazel macros?
Let’s take a real life scenario:
A new macro gets introduced by the framework team. It is for creating a new service executable and/or a replacement for a previous deprecated one.
The framework team wants to have a high adoption rate on that macro. Creating detailed documentation around it and sending Slack notifications about it would require a context switch from every engineer and would force them to start a small research on how to use it. Not to mention that he/she would need to have minimal proficiency with Bazel terminology since they would have to deal with some manual Bazel wiring.
Our solution to this challenge is to expose the macro creation within the IDE as part of the common flows our engineers are accustomed to and simplify the manual Bazel wiring involved. The development experience should be the same as adding a new file or directory from the IntelliJ context menu.
This way the engineers’ adoption rate would be higher due to the following:
They stay within their IDE comfort zone
They interact with a guided creation menu for each of the macros, simplifying Bazel terminology
Macro/rules templates are controlled via a separate CLI utility which IJ plugin triggers using bazel run; it means that updates and changes to the macro don’t require a plugin update, meaning loose coupling
Conclusion
Bazel is a great build system, it did wonders for Wix CI, cutting down build times significantly. When switching to a new technology that solves a big pain lots of big enterprises face these days, one has to think about other aspects of the development process and if they lose focus in favor of solving the big problem at hand, starting with the local development all through CI process and production deployment.
The local environment, especially the IDE, which is the place engineers dwell most of their time over, is a big part of the software lifecycle. Engineers must have the ability to develop freely regardless of the build system and other technological changes their environment is going through.
Any setback in the development process eventually limits development velocity which in turn holds back the business from conquering its goals in the timeframe they were scheduled for.
This post was written by Zachi Nachshon
You can follow him on Twitter
For more engineering updates and insights:
Join our Telegram channel
Visit us on GitHub
Subscribe to our YouTube channel