We love telling the story of our daily challenges and how we solved them.
Throughout 2020, we shared our experience and insights on engineering culture, architecture, scaling, mobile, frontend and backend and open source (yes, Covid-19 as well). All written by our very own Wix engineers. Here’s a quick recap of our 2020’s blog posts, podcasts and meetups -
This year we celebrated our 13,000 members on Meetup! - join us here
Continually Growing Your Engineering Skills While Scaling the Organization: Wix Engineering Culture
Wix is an engineering-focused company: our founders are engineers and about 50% of Wix’s staff is R&D. So to keep our engineering in best shape we spend 20% of work time enhancing our software engineering skills and building a strong, innovative and collaborative engineering community both within and outside of the organization.
How we do it, what we spend all that time on and the results we get (all during a global pandemic) - all told from the inside by the Server Guild Manager, Aviva Starovolsky-Peisach.
Helping Seniors During the Covid-19 Crisis
The Israeli Ministry of Finance and Welfare asked us to build a system to organize their outreach efforts to senior citizens and those in other at risk groups. Their volunteers would be calling to check on one million Israeli individuals: do they need food delivered; home assistance; any other form of help? The very next day we assembled a team.
In a week the system was ready for testing. And in another week, the system itself and both the user-facing website and app for volunteers was available to users. This is the story of how we built this web application so fast using Corvid. Oh, and don't miss the special podcast interview about it with Yoav Abrahami.
Contributing Code in the Age of COVID-19: How We Came Together For an Open Source Hackathon
3 hours and 35 participants equals 1,158 lines of code, 19 completed tasks and 21 pull requests. We transformed a traditionally offline Goodness Squad event - a meetup focused on contributing to open source projects - completely online utilizing some pretty cool tools and solutions. And yes, we even had an online afterparty to celebrate this event.
Want to get a taste and learn how we did it? Check out this article by Daria Tkhorevskaya.
International Women in Engineering Day 2020
On June 23 we marked the International Women in Engineering day, and this year we wanted to celebrate it by bringing you the stories of some of the women working as engineers across Wix Engineering R&D sites around the globe.
From ways to engage more young women to pursue careers in engineering to sharing best career advice and inspiration - Bringing the voices of our women-engineers with short and insightful interviews with Adi Marantz, Anastasiia Chepka, Brigita Pliška, Gal Sharir and Mariia Borovyk.
Meet Asaf Yonay, Head of FED Infrastructure at Wix Engineering
By his own accord, the three biggest challenges Asaf Yonay, Head of FED Infrastructure at Wix Engineering, faces are: Understanding and supporting the company vision, balancing support for existing products with creating new innovative tools, and Scaling up Wix, the FED guild, and the FED Infra group. Read the full interview with Asaf to learn how he's tackling all that, while also doing what he loves most.
Leading Server Software Infrastructure Group at Wix Engineering: Meet Yuval Perry
Here, at Wix Engineering, we’re always on the hunt for technical challenges and personal growth. So, it’s no wonder that our Server Guild Manager, Yuval Perry, joined us as a result of his own journey of microservices challenges.
We sat down with Yuval to hear more about his take on software infrastructure management, development velocity, propagating knowledge, and his strategy for dealing with technical debt. The result was an insightful interview. You can also hear Yuval's story here:
- Backend Engineering -
How We Implemented Idempotency in a Billing System at Scale
So you have a system processing hundreds of millions of dollars annually, and each transaction needs to go through smoothly and perfectly. Not to mention it has to happen exactly once.
At Wix Engineering we solved this with Idempotency and the exactly-once approach.
And created an infrastructure which allows developers to take any method, or function, and add to it idempotency or exactly-once semantics with near zero effort. Read Lior Asher's blog post to learn more.
How to Build 1,000 Microservices with Kafka and Thrive
In today's environment of globally distributed large scale web applications, reactive, event-driven design is mandatory. In this talk, Natan Silnitsky showcases how we implemented a reliable, scalable, event-driven, distributed Mega Microservices architecture on top of Kafka but without Kafka streams.
We, at Wix Engineering, have built powerful (soon-to-be-os) libraries for data streaming and data storage using Kafka features like compacted logs and exactly once semantics. These libraries offer low latency, high throughput event processing and also high volume data-storage with very simple and easy to understand APIs that work seamlessly together with other technologies such as gRPC.
Natan presents the challenges faced while creating and fine-tuning such a system. An architecture that is comprised of a 1,000 micro-services that need to handle incoming traffic from multiple data centers while preserving eventual consistency and high throughput:
Lessons-learned from Migrating our Build System to Bazel
The Bazel User Conference is an annual gathering of people passionate about the Bazel build system. It is the perfect place to get your dose of great tech talks, demos, and lightning talks presented by members of the Bazel community.
This time we had a lot to share: How we’ve made (and are making) local development with Bazel great, how we used Bazel to build a parallel and asynchronous artifact publishing pipeline, and 6 lessons we learned from migrating Wix build system to Bazel (with no downtime).
Watch our very own Ittai Zeidman, Or Shachar and Liron Tal share their insights on the stage of BazelCon 2019. You can also hear our Bazel-journey story here:
Principles on How to Build Fast at Scale
Three years ago, we were given a big challenge - Transform Wix Build System, then based on Maven and Teamcity, to a new system that will support our exponentially growing scale. We ended up choosing a build tool that originated at Google and completely revolutionized our builds.
Nowadays - the system builds in a fracture of the time - even with our largest codebases. In this talk, Or Shachar breaks down the key-principles that made the new build system so fast, how each one enables a faster, more efficient, and more stable build system, and touches upon how this was implemented using Bazel. For more engineering updates and insights:
Making Order in CI/CD Mess
CI/CD is a complex process involving code, services, and developers - stitching all these entities together is essential, especially when managing a huge scale CI/CD, like the one we have at Wix.
In this article Igal Harel describes the reasoning behind, and the challenges that lead us into building a CI/CD portal. He also explains how we implemented it and what the impact was of this unique portal on our engineering processes.
NPM Flattening for Zero Cold Start Serverless
Have you ever waited five or more minutes for 1Gb of NPM packages to download just to write three lines of code? We have. It doesn’t scale.
We built a freemium serverless platform and we wanted to let our users use NPM. We wanted to allow them to use any NPM package, with zero cold start while keeping sane download and deployment times while minding our storage. Now here’s the engineering challenge: The naive ‘npm install’ approach has several disadvantages. It’s slow, which prolongs deployment time.
It keeps the same package over and over again for different users which increases storage usage. It would force us to either have a long cold start if we choose to copy the node_modules when the container starts, or keep dedicated user containers online, which costs a lot.
In this talk, Schachar Levin unveils the solution we implemented to face this challenge and support complex npm, Inc. dependencies trees allowing multiple versions support, deduplicating packages, minimizing storage and download time, while making node able to import packages in a new and efficient way:
How (Not) To Add Deadlock To Your Critical Flow - 3 Simple Steps
The story of a Deadlock - How a library ended up causing a deadlock six months after it was released, and what a new feature that was recently developed by Wix, combined with java's ConcurrentHashMap, has to do with it. More on this and other lessons learned, alongside some practical advice, in this blog post by Guy Nahum.
How We’re Able to Host 1 Million Sites per MongoDB Cluster
MongoDB is a very capable database tool. But as with most things, there are strengths and weaknesses. In this article we explain how we got to discover one of its limitations and what we changed in our data model to compensate for it. This enabled us to host 1 million sites in a single non-shared MongoDB cluster. Read all about it in Giedrius Graževičius' article.
MySQL JSON Data Type and Migration
At Wix Engineering, we are using JSON in blobs Data Type as far back as MySQL 5.5. In this talk, Baruch Assif Osoveskiy shares both our wins and losses, our successes and our challenges, when it came to migrating our blob JSON data to JSON data types, and how we use the new data type to improve search time and reduce IO cost:
Working with NoSQL in a SQL Database
The X Dev API is a new protocol for non-blocking, asynchronous calls to MySQL. In this talk, Lior Altarescu explores the benefits of working with this protocol and connectors, and the challenges we encountered during the process of adopting X Dev API in Wix Engineering. We will share how we are incorporating the protocol in our massive multi dc architecture, and how it helps us, at Wix Engineering, rollout to production faster:
- Front-End Engineering -
Is Stencil a Better React?
A potential rival to React, a framework, capable of getting certain things done even faster, one that can make your coding life easier? Intrigued? So was Yoav Abrahami when he discovered Stencil - a React-inspired web component library. It is using the same JSX and some of the same concepts as React.
Yet, Stencil compiles to an optimal bundle, creating Virtual DOM that is consolidated directly into the DOM itself (no VDOM to VDOM comparison). The result is standard web components with optimal performance. In his article Yoav walks us through building the TodoMVC app from scratch with Stencil and dives deep into what the framework is capable of, comparing and contrasting it to React and Svelte. Read here
A Promise of a Bright Future With Async Iterators, Generators, and Pipes, Part 1
In a followup post to Dan Shappir's previous blog post, "Breaking Chains with Pipelines in Modern JavaScript", Dan takes iteration and piping to the next level. He shows how they can be extended to handle sequences of events, in addition to collections of items. In other words, to iterate both over containers of elements in (memory) space, and over elements generated dynamically in time.
You may know this approach as Functional Reactive Programming (FRP), implemented by libraries such as RxJS. Unlike such libraries, the library Dan implements is exceedingly simple and small, yet provides all the expected functionality. This is thanks to the use of new JavaScript features, especially asynchronous iterators and generators, and the for await statement. Read here.
Tipping the Scale
Times Square is (at least before Covid-19...) the most visited place globally with about 380,000 pedestrian visitors a day - roughly the same amount of people who go to Wix’s dashboard. Building a page which receives such high traffic requires having great performance and intelligent architecture.
In this talk, Eyal Eizenberg goes over the challenges of rebuilding Wix’s dashboard, the architecture and how we got it to load under 1 second:
My Website is Slow, Now What?
You built a great-looking, feature-rich website. But it also has a big problem: it's just too slow. As a result, your visitors are frustrated, and your bounce-rate is high. So now, what do you do?
In this session, Dan Shappir explains what web performance actually means, and how to measure the performance of your website. Dan shows tools and techniques you can use to locate performance bottlenecks, and various methods and technologies you can use to overcome them. Mostly he provides a list of action-items for improving the performance of your websites:
Interlude: Rethinking the JavaScript Pipeline Operator
In his two latest blog posts (Breaking Chains with Pipelines in Modern JavaScript, Promise of a Bright Future With Async Iterators, Generators, and Pipes, Part 1) Dan Shappir made significant use of the pipeline operator that isn’t yet part of the JavaScript / ECMAScript standard. This has resulted in pushback from some readers, especially given how much this operator seems to impact JavaScript semantics.
In this blog post Dan explains his decision to use the pipeline operator, and the whole motivation to add this operator to the language. He then goes on to show how additional research indicates that this operator isn’t actually necessary, and that the same benefits can be achieved without it. Read here.
React Hooks: SOLID View
About a year ago, we were celebrating the release of the new React API. ‘Rewrite everything with hooks’ was pretty much the mainstream idea. The community was in awe - certain use cases seemed to have become much simpler, there were new patterns, and the approach to designing components was changing.
Yet, when the dust settled, we all had to face the truth: there are no silver bullets out there. Turns out, we’re paying a certain price for those improvements. Watch Roman Yakobchuk clears things out and compare the pros and cons:
Problem Solving Using the TypeScript Compiler
TypeScript’s type system is very powerful and can express very exciting things. This technical-yet-humorous talk is done almost entirely hands-on, in an editor, inside a .d.ts file. By using a certain amount of generics, recursive types, and conditional types, we code together.
Watch Gal Schlezinger reveal how to solve FizzBuzz in the type system itself, by getting the solution with no runtime code whatsoever.
- Mobile Engineering -
How we Implemented Shared Element Transition in react-native-navigation
The limited screen size of mobile phones requires applications to split their content into many screens. This creates a sense of fragmentation for the user navigating in the app, as he constantly needs to reorient when in a new screen.
To improve continuity and reduce the sense of fragmentation, we can transition elements the user has already established a connection with, as the user navigates through the app.
This technique is called Shared Element Transition.
In this post, by Guy Carmeli, you will learn how to add Shared Element Transitions to your application and how we implemented it in react-native-navigation, our open source navigation library for react-native.
React Native at Wix
Wix App is one of the first applications built with React Native from scratch. After a 5-years journey, we are excited to look back in retrospective, and recount some of our daily challenges and how we solved them, alongside a couple of major decisions we’ve come to make over the years, and more potentially useful insights that we strongly believe the community could benefit from. Read all about it in Omri Bruchim's blog post.
Best Practices for Push Notifications in React Native
Push notifications are ever present in apps these days, a great way to communicate with users and increase engagement. Yet, how can you create an experience that’s both intuitive and expected by users? What are the key guidelines we should stick to? And how can we test the push notifications flows?
In his new blog post, Yogev Ben David discusses best practices to take your push notifications game up a notch, demonstrates how Wix handles notifications and consequent navigation, and goes into using Detox e2e to test push notification flows. Read here. You can hear more about React Native at Wix here:
- QA -
Lean QA: Monitoring, Experiments, Automation
Have you ever wondered if you want/need/are able to test in production? Is it even possible without bringing harm to your users? Does it bring any value? The approach isn’t new in the world.
In this talk, Kateryna Chernikova goes over how we have set up lean QA for our more than 30 independent teams. This includes both – manual and automation testing, all teams ownership on QA, experimenting, monitoring and so on. Watch:
Automation Strategy Step by Step
Do you trust your developers to release on their own? Do you sleep well if they do? Do you know what your next step is? High-velocity development requires us to adopt continuous integration workflows, which means our reliance on manual QA has to drop significantly. For Quality Assurance Engineers it means we need to learn how to build an effective automation strategy based on our resources.
In her talk, Yevhenia Hlovatska reviewed exact steps you can use to define your product’s strategy in order to sleep well while your automation rocks:
- Open Source -
Building a High-level SDK for Kafka: Greyhound Unleashed
It’s nearly impossible to find a Wix backend service that doesn’t use Apache Kafka. Our Kafka clusters process over 1B events per day and we’re still growing massively - in fact soon we will increase these numbers by an order of magnitude.
When we just adopted Kafka, we wanted to help developers provide callbacks to handle events instead of dealing with `while(true)` loops and offset commits. So we wrote Greyhound - a declarative and lightweight library. Since then it grew to something much bigger - a complete Scala/Java High-level SDK for Kafka that we use to provide abstractions for patterns used in Wix microservices, view topics and consumer lags, setup replication, configure topics, and much more. And of course, we have open-sourced it.
Learn more about the story behind creating Greyhound and glance at what it's capable of in this article by Noam Berman.
Introducing Repluggable - Solving The Dependency Model Challenge with Micro Frontends and Contracts
Most modern web apps are built using a very straightforward dependency model - in most cases you simply import the module you need and use it. Great for small apps, but as you scale, your dependency graph starts to look like a bowl of spaghetti...
Enter Repluggable - our new library that implements inversion of control for front end applications and makes development of medium or high-complexity projects much easier. It helps power the Wix Editor, and is now also open source. Read more in Itay Shtekel's blog post. And you can also watch his talk:
Building the Best Web Gallery, and Why the Hell It Was so Hard
"A web gallery is simple, they said. There must be a 3rd party that does it, they said.
Well, there wasn’t. And now there is. Building a high-quality web gallery sounded like an easy task, but it proved to be one of the biggest challenges I ever took. A gallery that is simple to use, creates beautiful layouts and loads fast, that was the goal.”
This is the short story of how Guy Sopher and his team built Pro Gallery, the challenges they faced and why they decided to open-source it (hint: make the internet beautiful again).
Detox in 2020
This year we released the 17th version of Detox, our mobile E2E testing framework, which marks the first of a three-stage rewrite of Detox’s iOS implementation. It is a process we’ve been working on for quite some time.
For the past half a year, we’ve been doing some pretty radical changes in Detox, some for better user experience (which we understood as first party users, must be improved), some for much better performance, and others for stability.
This release is a great opportunity to share, and also review some other key features we made in other areas of Detox. Read all about it in Rotem Mizrachi-Meidan’s post.
Quix: Now also Supporting Amazon Athena, Google BigQuery and Generic JDBC
Quix is our open source data exploration tool for Presto. Get acquainted with Quix, explore the main features, and discover what the future holds - all that and more in this blog post by Valery Frolov.
- New Podcast -
As said, major part of our engineering culture is to take an active part in the global developers’ community by sharing the best of our technical expertise. Be it via our open source projects, our blog and our Medium publication, our YouTube channel or our newsletter.
In 2020 we started our new podcast - Wix Engineering Podcast, by Wix Engineering, for developers and engineers.
In our podcast you can hear the stories and insights from our very own, alongside some of the most prominent voices in the tech community, as we will cover topics from architecture, scaling, mobile applications, and web development, to management, culture and much more.
You can find it on Apple Podcast, Spotify, Google Podcast, Deezer and our own Wix Engineering site. Here our 2020's engineering podcast stories:
Stand Up and Fight. Special guest: Håkon Wium Lie
In our first episode, “Stand Up and Fight”, Håkon Wium Lie, the mind behind the Cascading Style Sheets (CSS), tells his own story of one developer’s willingness to fight for the principles he believes in, even against the biggest technology company in the world at the time, Microsoft - listen here to part 1.
In part 2 you’ll also hear it from Gilad Segal on how Wix Engineering is taking CSS to the next level.
Growing Pains
Ittai Zeidman, our Backend Engineering Lead, was in the hospital with his wife and newborn, when he got an urgent call from the company’s VP of R&D. A crisis was unfolding: the build system was broken, leaving hundreds of developers unable to do their work. This crisis wasn’t an isolated incident: it was the result of a series of problems resulting from the company’s success and fast growth.
Ittai and his peers faced a serious challenge - but they knew they weren't the only ones: Google, Facebook, and Twitter were also serving millions of users. Utilizing and learning from the experience of these companies, Ittai helped transform Wix’s build system from the ground up with Bazel, an open-source Continuous Integration system.
Him and his team tackled these significant scaling issues, and managed to implement company-wide infrastructure successfully. Hear all about it directly from Ittai Zeidman and Or Shachar on our podcast episode. Plus, Natan Silnitsky will introduce Exodus, our open-source tool that can easily migrate JVM code from Maven to Bazel. Listen
Out of Your Comfort Zone
- I am running around 60 microservices and managing them is getting difficult, any advice?
- Oh, it’s quite simple. Delete your integration and test environments first.
Sounds crazy? Or maybe brilliant? That's exactly the dilemma our Head of Backend Engineering, Yuval Perry, faced back in 2012. He was then working for a company which was running more than 60 microservices - and managing things was getting difficult. He approached Aviran Mordo - our VP Engineering - for advice. That's how he got that "simple, start by deleting things" advice, which definitely wasn’t the answer Yuval was expecting…
In a new episode of our Wix Engineering podcast, Yuval explains how he had to step out of his comfort zone in a major way to ultimately come up with a solution he himself described as ‘either crazy or brilliant’. Listen.
How Svelte Beats the Odds. Special guest: Rich Harris
There's a blazingly fast web framework out there, one that's easy to learn and battle-tested in real world applications. And no, we’re not talking about React.
Since Svelte was invented by Rich Harris, at every conference and in every Twitter thread, he has had to fight an uphill battle against other frameworks, just to be noticed. In our newest podcast episode Harris tells the tale of Svelte and shares his insights into having developers give a relatively unknown framework a go. Also, how Wix Engineering is testing Rich’s framework, shared by Eliran Hezkia. Listen here.
Crisis Mode - Building a Nationwide App During a Global Pandemic
The global pandemic tore through every country on Earth and the world was devolving into panic.
That’s when Yoav Abrahami received one of the most important assignments of his life: build an app that would service the entire nation of Israel through its pandemic response. And he had just one week to finish the job. Despite little time and a harsh deadline, a prototype was done in time. Listen to Yoav tell the story of how we pulled it off in our podcast episode.
Breaking Down React Native
For years mobile development teams split their talent between Android and iOS in a way that was costly, slow, and inefficient. Then, in 2015, Facebook developers came up with a cross-platform solution called React Native. In this episode we follow our Wix mobile engineering team as they adopt React Native and work with it for several years while building the official Wix app.
Listen to Lev Vidrak and Omri Bruchim as they take a deep dive into Reach Native.
Manage Through the Pandemic: Top Voices Panel
This year we sat down with leaders of some of the most prominent tech companies to discuss the biggest challenges facing our sector today:
What’s it like to readjust an entire multinational organization during a pandemic? How can a growing company keep innovating, and avoid collapsing under the weight of its own success? How much freedom should managers give their developers to ensure maximum productivity and creativity? And many more.
We had the pleasure of hosting Yuval Kesten, Director of Engineering Facebook, Karin Moscovici, VP R&D Riskified, Ohad Jassin, GM Azure Media AI Microsoft, Gai Berkovich, COO Waze, and Sivan Brezniak, Viewer Company Manager Wix.
The unique discussion we had became the foundation for a special podcast episode, summarizing all of the great insights from the talk. Listen here, and also watch it here:
- Newsletter -
Every month we're sending out our monthly newsletter, where we cover our engineering insights and feature in-depth articles and news from around the global tech industry. Newsletters are a great way to keep your engineering knowledge sharp, while getting these goodies directly to your inbox.
For more engineering updates and insights:
Join our Telegram channel
Visit us on GitHub
Subscribe to our YouTube channel