The global pandemic tore through every country on Earth and the world was devolving into panic. That’s when Yoav Abrahami received one of the most important assignments of his life: build an app that would service the entire nation of Israel through its pandemic response.
And he had just one week to finish the job. Despite little time and a harsh deadline, a prototype was done in time. Listen to Yoav tell the story of how we pulled it off in this podcast episode:
You can also listen to this episode on Apple podcast, Spotify, Google or on Wix Engineering site. And you can also read the full interview here:
Hi, I’m Ran Levi, welcome to the Wix Engineering Podcast.
Recently, Yoav Abrahami--Chief Architect at Wix--spoke to the writer of our podcast, Nate Nelson, about one of the most important assignments of his life.
Take me back to the day that you guys get a call from the Israeli government. What happens?
[. . .]
Yoav: We got a call from Avishai and he says, “Hey, there’s a project by the government,”
For context, Avishai Abrahami is the CEO of Wix. He also happens to be Yoav’s brother.
Yoav: “There is some project by the government. Talk to this guy to figure it out.” So I’m calling that guy and he’s like, “Yeah, there is something, we’re going to check it out”. And the next day, I get a call from another person, who says, “Hey, there is a project. Let’s try and see if we can do something here.”
A big project was coming together. Yoav’s team was being approached by the government, with the urgent task to create a software program that could accommodate the entire nation of Israel. It was exciting. High-pressure. A little scary, maybe. And there was one other little problem.
How quickly did it need to deploy?
Yoav: So they wanted to deploy it by next week.
Next week.
Yoav: The call was on Thursday, they wanted to have something up and running by next Monday - Tuesday.
In this episode of our podcast: how to build an app for an entire country in seven days or less. If you really have to.
Many of you out there listening to this show are, yourselves, developers. I don’t need to tell you how long it takes to create a good, working app from scratch all the way to full deployment.
But for anyone who isn’t familiar with the process, just imagine all the work that must go into designing the app, developing the backend, developing the UI, then testing it, debugging it, re-testing it, and getting it approved by app stores.
And that’s just what’s necessary to build some generic, throwaway app. Yoav and his team were building something highly sensitive, with very specific requirements. And they only had one week. For reference, it sometimes takes me one week just to finish a particularly big sandwich.
Usually, it’s inadvisable to rush out an application so quickly. In this case, however, the situation was highly time-sensitive.
Netanel: So my name is Netanel Oschry. I’m the Social Welfare and Social Security referent here in Israel from the Ministry of Finance. So generally it means that I take care of the Social Welfare Ministry budget, throughout the year and also in coronavirus time.
Coronavirus isn’t the kind of virus that waits until you’re ready for it. Back in the late winter and early spring, when it began to spread worldwide by the millions, schools, healthcare systems, businesses and people were scrambling to adjust.
The duty of those in government was to coordinate response efforts--to get as many vulnerable people as possible the kind of help they needed, as quickly as they could get it. For the Israeli government, in this case, that meant calling up people who could help.
Netanel: I was looking for the technological solution to map the elderly population and one of my colleagues here in the Ministry of Finance who’s in contact with also Wix and other high-tech companies told me that he has perfect people for me who are looking to volunteer and contribute at the time of coronavirus.
Yoav: So we’re getting on a call with a few people from the Ministry of Finance, the division of – that is responsible for the budget - and someone from the Ministry of Welfare. And they have this idea that, “Hey, we have a lot of people in Israel that are in potential need, especially elderly people but not only them, and we want to call them and check on what’s going on with them with the corona status. Do they need some kind of help, maybe food delivery, maybe medical help, maybe something different? And we’re wondering if we can create a system to do that?.” And we’re like, “Hey, you know, that sounds like a super great idea. Yeah, we can do that.”
Netanel: So we came up with this Golden Guard
So what is the first step to take you on an assignment like this? Tell me about, like, the first day that you guys all got together to work on this project.
Yoav: So the first thing is to get the right team in place. So what we’ve done, the first realization is that there is an idea, it’s a rough idea, but no one really knows exactly what it means. So the first challenge is to get someone from our team who can go and figure out what is the thing that we need to build. And more importantly than that, what are the key elements that have to be working on the first release in order to start moving this forward.
So with the right team and with the right focus of what we need to deliver in order to start this process running forward, you can start to create the project.
The goal of the project was to create an app that could help elderly citizens in two key ways.
Netanel: One is to keep them safe mostly by reducing their need to be outside and get out of their houses and exposed to the virus, and two, making sure they have the food, security or nutritional security they need and also all other social services they need within this time that they are closed in their houses.
In other words, the app needed to connect volunteers with people who may be in need of assistance.
But underneath this simple premise laid a whole host of technical challenges. Take, for instance, the simple problem of where to store data. For a company like Wix, this kind of thing is almost an afterthought. Modern software companies store so much data that worrying about it would be like worrying about brushing your teeth.
But software companies don’t usually have the considerations governments do. Generally speaking, private companies are better-suited to doing quicker, looser jobs, whereas government bodies have to deal with immense legal accountability and political bureaucracy. So with Golden Guard, the Israeli government wasn’t just asking Wix to build an app that was good. They were asking Wix to build an app that was good and also adhered to strict regulatory and legal guidelines. Yoav and his colleagues had to get used to this way of thinking--to make certain, for example, that the app could screen volunteers according to not just high standards, but government standards, and that a whole set of privacy and safety laws were being strictly, uniformly adhered to.
Yoav: You know the first thing that we wanted to make sure was that we don’t hear tomorrow that, hey, Wix has built some applications and then someone has downloaded the list of 100 – so 1 million citizens with all of their details.
When you’re building an app for hundreds of thousands of people, and the government is putting their stamp on it, there is no room for error. Not the best thing to hear when you’ve only got a week to build your perfect, spotless program. Still, a solution to the data storage had to be found.
Yoav: What we’ve done there is we’ve taken one of the industry leaders, Google Cloud, and we’ve used their loud SQL solution, which is already binding with different regulations. We made sure that it is configured with the encryption on disk which is what you need for the different regulations. We did another layer of client encryption, data values encryption on top of it, and made sure that only two people in Wix have access to the database. So that kind of creates the first level of a data separation and makes sure that no one has access to it.
The second level is to make sure that in our application we have at each level the right checks that the right person can access the right API. Because, you know, one of the easy ways of actually getting into a database is to not directly getting into a database, but just finding an API that exposes the data. So for that, we added in our project two security people from day one as part of the project.
Some of them were initiated by Wix, some of them were initiated by the government - just to make sure that we don’t have any issue like that. The good news - we didn’t have an issue.
So the database problem had a solution. Check. But even for an app as simple as this, there were still many more technical problems to solve, and not much time to solve them.
So you have your assignment from the Ministry. It’s to build this app that’s going to connect people so that they can essentially call each other. Now, from the outside, this seems like a kind of a simple thing to make. Tell me, why it wasn’t so simple?
Yoav: There’s a few reasons why it’s not so simple, and some of them are more technology-oriented and security-oriented, and some are just - simply to figure out what is the right application. And just to give you a sense on each side of the problems or challenges - one of the challenges was that we have volunteers and they need to call elderly people.
Now, the volunteer is someone that you don’t really know. It’s someone that has applied online and although you’ve checked them and you’ve checked them in all kinds of different ways (which is a challenge by itself), you still trust them only so far. So you don’t want a volunteer to see the actual phone number of the people they are calling. You need to be able to let them click a button in the application and initiate the call between the volunteer and the person they’re calling in such a way that neither can see the number.
In order to do that, we need to use a third-party service. In this case, the service was Twilio which is a telephone provider, a company that has expertise in telephone providers and API, and they would actually call the volunteer and then they would call the elderly person and connect the two calls into one. That, by itself, also didn’t prove that simple because what happens when the elderly person doesn’t answer, or when the phone number that you got from the Ministry is wrong, which we have quite a few of those, or when it is busy?
So, do you need to take that person and get him back into a queue to call him later? What happens when the volunteer doesn’t answer? If the volunteer is busy, or the volunteer’s phone number is wrong? So that turns into quite a lot of different complexities going just in that area. And we’ve already mentioned about checking the volunteer. In order to check the volunteer, the way that the Ministry wanted us to do that was to use a “Mimshal-Zamin” - which is the portal of the Israeli Government.
And in order to do that we need to also create an integration which is a very secure integration with their team using some specific protocols... that requires to have certificates and domains and exchange keys with them and make sure it was in the right APIs and then there were different issues and bugs.
That was just another challenge. And there’s more, regulation and penetration testing...
Between cybersecurity, privacy, data storage, vetting volunteers, and actually, you know, building an app that worked in the first place, Yoav and his team didn’t have much time to dwell on any one technical issue. Instead, they needed a platform where every given issue could be identified, addressed and fixed in a very brief window of time. So they took a big bet on a relatively new development paradigm.
What is Velo?
Yoav: Velo is a product that I started about six years ago. And the idea is to take the best from Wix, which is the ability to design and “draw” user interface, and then take the best from the Cloud world, which is basically things like serverless compute and serverless databases, and connect those two together. And to do it in such a way that when you want to create an application, a production application, you can draw user interface, code the backend, the integration, use the database, click a button - and have it deployed on four different regions across the world in three different continents and just have it production ready from day one.
And why was Velo relevant to this project in particular?
Yoav: I think the main value of Velo is that you can develop 10 times faster compared to anything else. And when you need to deploy a project in such a tight schedule with such high volume and such complexity, Velo is the ideal solution for it.
It was over half a decade ago that Yoav had the idea to build a system to help developers get their products to deployment faster. He couldn’t have known, at the time, that it would become so necessary to himself down the line.
So you know in the world of computing… we developers, we love a lot of freedom. We love to make a lot of decisions. Making decisions takes time. Checking technologies take time. Trying to connect different things together takes a lot of time. Configuring, building servers, configuring the network, configuring how different services interact, all of that takes a lot of time. What we’ve done with Velo - we’ve eliminated most of it.
You get a ready to use environment from day zero, which includes everything that you need. It builds both the development environment, which is online, so there’s no set up, you just start working. It includes the key elements - the ability to create an application, the database, a server that you can write code against and it’s already pre-integrated. You get it so that its deployment and build are already done for you. So the time it takes you to go from a new project to production applications, it can be as low five minutes.
Thanks to Velo, a few sleepless nights and a little elbow grease, Yoav and his team were able to release a working app within the window required by the government. But the job was hardly done. Because of course they couldn’t build a flawless app in one week’s time! While the developers were burying their heads in this database or that, this piece of code here, here and here, they hardly had time to consider all the possible issues actual users might come up with.
Yoav: For instance. So we had a question -do you need any help with food? And if you do, what kind of food are you eating? Is it a regular or is it a – are you a vegetarian or you have any allergies or anything like that? And at some point, they added the question”do you want Kosher food or not?” And then there was a question, if those two questions would be mixed together or not. And then later on it was decided that we’re not going to ask about Kosher food or not, we just assume it’s going to be Kosher for everyone.
We added another function later on which is an emergency function, we call it Red Button. You’re calling someone and saying, “Hey, I didn’t eat anything for the last three days.” Well, maybe you don’t want to wait for reports and data export the next day. We want to send a volunteer there now to help with that. And things like that, things that we learned along the way and changes that keep on happening simply because you’re working on an application to link people, and things change, and you want to be better.
Ultimately, though the app made it to deployment in only a week or so, it wouldn’t be finished for a while longer than that.
Yoav: The way that Velo works is that you don’t really deploy in a regular sense, you don’t have a specific machine where the application is deployed on or a device to copy the application to. The application is all over the web. What we did see is that as we started releasing the application, after two weeks of working, eight days of development, another week of proofing and starting to round up the application, we had only the minimal set of things that we needed to make it function. But then there was another set of things we needed to build to make it more productive - things like reports, dashboards, ability to export the data and send that exported data back to the Ministry so they can actually act on it.
I think we have invested another month and a half in improving the application as we go and as we learned and as we got new requirements.
Allow me to apologize, listeners. I promised that I’d tell you how to build an app in seven days or less. And here we are, with an app that took nearly two months to be finished.
But did I lie to you? I don’t think so. The goal of this episode was to tell the story of how a team of developers managed to take an assignment, and turn it into a live app for thousands of people in only a week’s time. Did Yoav’s team build the app in time? Yes. Did it work? Certainly. Was it perfect? No. Did people use it, even if it wasn’t perfect? They did - many of them.
Yoav: At the top usage, we had 800 people doing calls concurrently, doing 15,000 calls a day. And the system just worked.
Netanel: At the peak, we provided 130,000 elderly people with hot meals for each day of the week using both volunteers and the army. Six thousand of them told us that they needed food, two thousand needed medication. And we transferred this data to the HMOs and they helped them with the medication they needed.
Another 6000 told us they suffer severely from loneliness and asked for our assistance. So we either got a volunteer to call them once every two days or we made up other solutions with the municipalities.
I’m thinking of calling this episode How to Build an App in Seven Days or Less. Yoav, from your experience, how can development teams deploy apps in timeframes of weeks or even days when the circumstances call for it? What’s necessary to get the job done?
Yoav: If you want to get an application done fast and right, first, you need to have some person that can give the right focus. You need to make sure that you’re building the right thing from day zero, you don’t have time to build the wrong stuff. It might be a product person, it might be a manager, it might be just any other developer, but someone needs to say ”hey, this is what we’re doing and those are things that we’re not doing”.
Second thing, you need to have a team that knows how to work together. And once they see that this is the target, they know to go in a very aggressive way and build up for that target. And the third thing, you need to have the right technology to support that.
Yoav didn’t mention it, but I’d like to add a fourth thing here.
If you want to build an app in days, rather than months or years, you need to have a common purpose--a goal that every single member of your team is not only striving for, but cares about. Golden Guard was not a lucrative project. It didn’t launch anyone’s careers. It is only because these developers actually cared about the impact they were making that they were willing to stay longer in the office, drink Red Bulls late at night, and create something that people really needed.
As of this writing, Israel has become one of the first countries to not only end quarantine nationwide, but end social distancing and the wearing of masks and gloves as well. The fight is not over. Luckily for everyone, though, if a second wave does come around, we’ll have had months, not days, to prepare.
That’s it for this episode, thanks for listening.
For more engineering updates and insights:
Visit us on GitHub
Subscribe to our YouTube channel