Coding with the CEO, E14: Full Transcript
- Wix Engineering
- Jul 27, 2021
- 13 min read
Updated: Aug 31, 2021
Is good enough good enough? For years Wix engineers were building fine services, but as systems became more complex, it was taking too long to do it. The code was clean, but it took thousands of lines for each service.
It was inefficient, so Wix’s CEO & Co-Founder, Avishai Abrahami, decided to step in. He set up weekly, hands-on coding sessions with some of his best developers, to chart a new path.
The goal he set for Wix Engineering was that if a project took months before - it would now take days, and every program with 10,000 lines of code - would now be 100.
Listen to this unique story, its challenges and impact.
You can also listen to this episode on Apple podcast, Spotify, Google or on Wix Engineering site. And you can also read the full episode here:
Hi, and welcome to The Wix Engineering Podcast. I’m Ran Levi.
HOW DEVELOPERS MATURE
Just about every developer, over the course of their career, goes through the same evolutionary process.
Avishai: So usually after three years that you’ve been developing, right? You kind of get to a point where you know how to do loops and “ifs” very well, and you think that now you understand programming.
That’s Avishai Abrahami, Wix’s shy, timid, always politically correct CEO.
Avishai: And then there’s like six years of talking so much bullshit, that you just don’t want to listen to them. It’s like they’re so full of themselves.
Hmm. Never mind.
Avishai: Then they get to this place where they start getting to talk about architecture and architecture will be - how do you connect things to things. And at that point they’re starting to reexamine how they work and get much more productive.
To get to this point--where you begin to see the bigger picture, not just the trees but the whole forest--takes a lot of experience. The longer you’re at it, the clearer the picture becomes.
Avishai: And then something happens, usually between 15 to 20 years, which is you start to understand how to do most with less. OK? And this is kind of like how to develop software that is really, really good at being minimal.
THE PROBLEM
Avishai’s been a developer for decades, so he can see the forest. He knows how to design efficient software.
But among the thousands of developers who work for him, few have reached that 15- to 20-year mark. And so, often, forests are missed for trees. Code that could be very minimal ends up maximal.
Yuval: When a big project starts to emerge, we review the roadmap.
As a manager for Wix’s Server Guild and Server Infrastructure Group, Yuval Perry was encountering lots of Wix projects that just seemed to be way too...maximal.
Yuval: In a few projects we took a look at them and when they started off in the first phase, they didn’t deliver almost anything. They delivered like 12 microservices that do CRUD work only and it took several weeks for each one to be published to production. And all of them, 12 of them, took us almost, I think, 36 weeks in order to publish them.
Ittai: That specific project that Yuval mentioned, is a big rewrite.
Ittai Zeidman, CTO of the Server Guild.
Ittai: So that is kind of what – you know, lit up the issue, right? Of saying, OK, we’re not adding any features, we’re just doing a rewrite. But still we’re going to pay a very big cost on something that is and will be still relatively simple. So why are we paying that big of a cost?
This kind of story just kept repeating itself.
It wasn’t that everything was going horribly wrong all the time, or that all the developers didn’t know what they were doing.
Avishai: The issue here is not talent. The issue is what are the patterns that people tend to believe in
Complicated patterns, instead of simpler ones.
FIRST MEETING
Ittai: What happened was that, you know, during the project that Yuval mentioned, he said, “OK, I want to understand why it’s complicated. I don’t think it should be that complicated, let’s understand.”
Avishai called a meeting. Together with Itai, Yuval and some of his other top guns, they went step-by-step through the typical development cycle to try and figure out what was slowing things down.
Ittai: Then he finished and he said, “OK. Now I understand why it takes you that much time. You’re recruiting amazing people. I worked with that developer. Your problem is not recruitment and the problem is not the estimates inside the companies. The problem is you guys,”
Avishai pointed at the infra team. At Ittai and Yuval.
Ittai: And said, “There’s no way people need to do all that work again and again and again. It has to be inside the infra.”
They tried defending themselves.
Ittai: “No, but we think that everyone should know all of these lines”, and so on and so on. Then that was the stage where he said, “OK. We need to be aligned on how to build software”.
How to build software. It’s funny, in a way: we’re talking about one of the biggest, most successful software companies in the world trying to figure out how to build software.
But of course, we’re not talking about how to write HTML. We’re talking about philosophy. Not how to write code, but how should you write code.
Yuval: It’s a shift in mindset.
Yuval Perry.
Yuval: I mean, we had our mindset in one place and Avishai, he had his view on how an R&D organization should be built.
Of course, shifting the mindset of a whole company isn’t the kind of thing you achieve overnight. You can’t just write an email, like:
Dear everybody,
On Tuesday can you all drastically change the way you’ve been doing your jobs for your entire lives?
Thanks,
Avishai.
P.S. Who ate my yogurt in the break room fridge? I clearly wrote “Avishai” on the lid.
Ittai: I think that Avishai realized that unless he brings us to the table, right, and gets us to do the mind shift, then there’s no way to bring those 1000, 2000 developers into the game, right? Because people can say, “OK, this is the directive and we will do it.” But they will do it in the least effective way. But if they’re on board, if they’re, “Yes, this is what we need to do”, then you will get 10 times more out of them.
ABSTRACT DISCUSSIONS
Yuval: That’s why we decided to do a weekly meeting.
A weekly meeting. A convening of the president and his top generals. They called these sessions: “Design Patterns”.
Ittai: Because he thought that we don’t understand patterns of design. So he said “You need – we need to be aligned”, which is a nice way of saying, “You need to understand my view of patterns of software design”.
The concept of Design Patterns is ancient, in software engineering terms - it was first introduced in the 1970’s. In a nutshell, it is the idea of using general, resulable solutions to common problems that developers are likely to encounter again and again. In other words, how to not reinvent the wheel each time you need to build a new carriage.
Interviewer: What are these conversations like? What are you talking about?
Ittai: A lot of the early meetings were completely abstract and did not talk about Wix at all. Or only used Wix as metaphors and analogies of commonalities.
OK. Why – how can we find common patterns between different services? What are patterns of messaging? What are patterns of persistence, OK? How do we know that something happened in the universe? How do systems recover from faults?
CASE STUDY: DATABASE
Interviewer: So can you paint a picture for me of when you’re looking at the source code, what exactly you find wrong with it?
Avishai: Well, yeah, of course. So I’ll give you the most basic thing, right? How do you approach a database, right? How do you access the database? I think that – and this is – normally you would – the developer, they will do some kind of way to connect to the database. Then a way to connect and select the data. And my thought is that well, if you’re always doing the same things, you actually should have zero source code, new lines of code. It should always be exactly the same.
You should only find ways to override what is not the default.
I want to give you an example. The most used software on the planet,the most used are the operating systems. And operating systems in many ways are developer tools. Why? Because you write software in them. But they allow you to have a very clear API, a very clear way of overriding things and that’s why it’s so easy to consume them as a developer.
So in my mind, if you want to build something that allows access to the database, you should be able to have an operating system for accessing databases. It should be a very closed software that is very good at doing 90-something percent of the thing that you want to do and then 5 percent not be able to do it at all.
And if you don’t need anything different, you actually should only change the configuration and that’s it, it should work. And that is a very different approach than “how do you easily write a database program?”.
CUTTING 1500 LINES TO 1
After weeks of very abstract, high-level discussions, they finally got around to the code.
Yuval: This discussion came to the point that we would show a service in production, and the code, and then we scanned the code and would throw remarks at it. And everybody shows his ideas on how it should be structured in a better way.
Avishai: And we started to analyze. How much of this source code we actually need.
Ittai: So, you know, it’s like a bit of trash talking also. Avishai says, like, “See, what do you think can be removed? Oh, like you’re not seeing it,” and stuff like that.
Avishai pushed hard. And he didn’t budge, even a little. For example, in one case, they looked over a project from Wix Booking. How much of its code could be removed, they asked?
Avishai: It was about 1500 lines of code and I think – I don’t know how many lines of testing - and the answer was one line.
1,500 lines. Remove 1,499 of them.
Avishai: That’s insanity.
RESISTANCE
As Avishai tried convincing them of his way of thinking, the infra team wasn’t exactly on board.
Ittai: And we’re fighting with him. “No, this has to stay!”, and this and so on and so on.
Avishai: They all explained to me that this will never work, this is not how you develop software.
You can’t really blame them. Imagine your boss telling you that you have to be 1,500 times more efficient.
Avishai: It’s always scary, right? And there is a way that we know how to do things. Why do we have to change it, right? I’m sure that if you ask somebody in a monarchy 500 years ago, would they consider democracy? They would be like, “Oh my god. That sounds awful. No.”
In meeting after meeting, Avishai gradually tried convincing his team to consider democracy.
Avishai: We just went and analyzed case after case after case and, you know, it’s just really smart people with me in this room.
It’s interesting, because they don’t come with stupid arguments against it. They come with brilliant arguments against it. OK? That’s the thing. These people are brilliant.
But then they started to play with that. And after about a month, or a month and a half, their perspectives started to change on what they do.
Gradually, the infra people came around to Avishai’s philosophy.
Avishai: After they grasp it--assuming they are prepared and they have the seniority and knowledge and intelligence to understand the benefits, they just become addicted. Now they all became, “Oh my god. This is how we have to work. We have to work like that.”
After weeks and weeks of debating, Avishai had broken through. Except that wasn’t even half the battle.
CONVINCING DEVS
Because it was one thing for Avishai to convince Ittai and Yuval about his theories. It was another thing for Ittai and Yuval to then go back to their guilds and convince everybody else about something it took even them so long to come around to.
Avishai: Then they went back to their teams and said, “Listen guys, we need to work like that,” and the next thing that happened is that pretty much everybody from those teams has threatened to quit Wix if we move to work like that.
They just weren’t used to these kinds of ideas.
Ittai: If I touch on what Avishai said - he said that it takes people, you know, after they code for 10 years and then 12, 15, 20 - it takes them time and maturity and experience to understand that abstraction and commonality far outweigh the cons.
So this is something that took us time. And we understand that it will take time for others.
Educating the engineers wasn’t easy. Like Avishai, they first tried to demonstrate the abstract principles, before applying them to Wix use cases. That didn’t work.
Yuval: We created code in a dark room without Wix concerns. So we took a service that does almost nothing, saved it to the database, presented it, showed all of the right abstraction levels - and then we gave it to the engineers and it wasn’t really connected to day to day work.
Ittai: It was not connected to what people actually needed. But it was – according to our understanding which was too limited.
Yuval: So what we did immediately after we understood that, we took services from production and rewrote them with the team until they did exactly the same in our way. And this is where we were more connected to the ground.
LESS IS MORE E.G.: GDPR
Yuval: So I will give you an example. So for instance we had a challenge called GDPR.
GDPR is a set of data privacy regulations that apply to citizens of the EU. When it was first written into law in 2018, basically every company you can think of had to diligently rewrite their software to comply with it.
Avishai: You have to go to every place. You are accessing a database - check if that’s personal information, if it is, you have to encrypt it, you have to find a way to erase it, you have to do a lot of different things.
Wix distributed documentation to all their developers. Everybody had to read over the new rules, and apply them in their code. It was terribly labor-intensive. The laws were complicated, and software is already complicated, plus there was no room for error. Even previously simple tasks now had to be rigorously reevaluated.
Avishai: If you have customers’ database and you have 20 ways of accessing that database from different places, you have to do it 20 times. Multiply every access that you did. It’s a massive effort.
Think of how that adds up across a large company. We’re talking about tons and tons of new code. They needed a way to contain that bloating.
So, using the principles of Design Patterns, the infra team took a step back. They looked at the bigger picture.
Ittai: Every developer that implemented or complied with GDPR, let’s say they had 300 lines of code, OK? As far as they were concerned, they had the minimum lines of code that they needed to comply with GDPR. The fact was that this was just a local optimum of lines of code and if you try to go for the global optimum, you say “There’s no sense in everyone doing those 300 lines of code”.
Everyone was writing their own GDPR code, which was excessive.
Yuval: In the iteration after this thinking group emerged, we inserted this concern into the framework, we took this principle into every component that we added, every component, like SPI, data extension, plugability, whatever.
Instead of 300 lines of GDPR in every individual project…
Ittai: We write, let’s say, a thousand lines of generic GDPR. But then that saves, let’s say, maybe 30,000 lines of code across the services.
Avishai: The team that uses those systems doesn’t have to think about those things anymore. They just have to say “Oh, this is a personal identity field”, and that’s it. By adding that notation to that field the infrastructure team is solving that for them and the infrastructure team is doing it once - and that’s a massive improvement.
COMPANY IMPROVEMENTS
Gradually, over time, more and more projects started to align with the Design Patterns philosophy.
Ittai: So at the beginning, like I said, there wasn’t really change, because it took a lot of time, you know, of the mindset.
Avishai: At that point, it was the first time that things started to change, right? People were like, “Oh my god. It’s so much easier. It’s so much better. I want to use that”, and that team just suddenly started to ask for more and more projects to do that in that way.
Then it kind of started to drip around, within the company. To the point today that I think all of the new projects opening are now built in this kind of a concept and not with our old techniques.
Interviewer: So how has the company overall as a business improved as a result of doing these developments?
Avishai: So the places that are using that - the development performance is dramatically faster.
We assume that it saves about 75% of the work, which means that you have 25%. Out of that 25% I believe we can save another 50%.
Ittai: They are seeing amazing results with respect to developer time, with respect to the number of lines of code. We’re talking about reductions of 70, 80 percent of lines of code, which are cut.
Some services, complex services, from 25 days to 3 days, working days.
TAKEAWAYS
Interviewer: So do you think that these design principles, the work that’s now starting to be done at Wix, applies to any company out there, or is it just sort of companies like Wix, large companies, companies that do the kind of thing that you’re doing?
Avishai: Well, it’s a really good question. I think if you are building your first product and you have a very simple first product, which is what you should be building when you’re doing a startup and you’re just starting, then the benefits from that are not going to be huge.
Yuval: In Wix, deploying services to production is by the dozens every day. So we need to automate it.
If we were a small company and services would be deployed once in three months, I wouldn’t even address this issue.
Ittai: I super agree with Yuval, right? You know, find a product market fit. Don’t waste time on preparing for the future that will never happen and you start to fail and you’re out of money - I was there, it sucks to run out of money. So that’s on the one hand.
On the other hand, there is the other problem of you actually succeeding and now you – like you’re in deep mud, right? And that’s a question. How do you know where to invest at what point? And to that, I can probably just say hire good people who have failed and succeeded and hopefully will have the intuition on what to automate at what point.
But I still think that taking that concept of less, doing less with more, it’s still valid even at a small stage, because you can think about the open source and the ecosystem as you infrastructure group.
Interviewer: Is there anything else that you guys haven’t yet gotten to that listeners might want to take away from this story?
Ittai: One thing that listeners might be interested in hearing is “what does this project actually contain”, right? And what does it have? What does this mean, these abstractions and so on and so on?
To truly, fully understand Design Patterns, we’re going to have to talk about “Nile.” Probably the biggest, most important application of their philosophy. When Wix Infrastructure put everything they learned to the test.
Ittai: I think this is something that we can dive into further and maybe we’ll do it in the next episode. So stay tuned.
That’s it for this episode, thank you for listening. For a full list of our previous episodes - visit wix.engineering/podacst. The Wix Engineering Podcast is produced by PI Media - written by Nate Nelson, produced by Yotam Halachmi and narrated and edited by me, Ran Levi. Special thanks to Moard Stern from Wix. See you again next episode, bye bye.
For more engineering updates and insights:
Join our Telegram channel
Visit us on GitHub
Subscribe to our YouTube channel
Looking forward to seeing what the change was actually