From large-scale multinational companies to mom-and-pop shops, millions of individuals, businesses and organizations across the world have their own websites. Many of these are hosted on the website builder platform Wix.
Based in Israel, Wix has more than 225 million registered users worldwide who use its powerful platform for building and hosting websites. “We empower people and businesses across the world to create, manage and grow their online presence, regardless of their coding and design knowledge. Our ongoing success is built on the reliability of our products,” explains Jonathan Ginzburg, Head of Production Engineering at Wix.
Wix processes billions of requests every day to serve its customers and its customers’ customers. To service these growing demands, Wix needs a reliable and secure infrastructure that can load sites fast and ensure that they are available 24/7, 365 days of the year. Wix aims to eliminate downtime from its customers’ vocabulary.
In addition to its resilient infrastructure, Wix’s subscribers are attracted by its innovative tools and its trusted range of security features. With cybercrime forecast to cost the world $10.5 trillion annually by 2025, it’s clear why Wix puts so much time, effort, and investment into ensuring that all of its customers’ websites remain safe, secure and available.
What Wix’s customers don’t necessarily see is the speed at which the platform is growing globally. “Wix grew very quickly in a very short period of time,” explains Stanislav Panich, Head of Networking at Wix. “In 2017, we were running three regions worldwide that were serving actual user traffic. Then all of a sudden we had workloads hosted across the United States, Europe, India, and China.”
Wix’s new found global success posed a lot of new challenges for the company in terms of performance and reliability. It realized that it needed to develop a much more scalable infrastructure that could grow to keep up with the continuing expansion of its customer base, and it needed to do this without impacting on the service it provides to its customers. Key to creating this secure, dependable and scalable service, was being able to efficiently serve Transport Layer Security (TLS) certificates en masse.
“When https became the standard working protocol, I had to resolve a huge challenge at scale to make sure that all of our customers’ websites, wherever they are in the world, were aligned with the new encryption protocol standards.” says Ginzburg. “This meant inventing a unique solution for swiftly terminating an enormous amount of TLS certificates, while continuing to assure reliability for our customers.”
Plus, as Wix’s customer numbers multiplied, it became crucial to minimize network latency for users wherever they were in the world. Wix wanted to bring users as close as possible to the data centers serving them. It realized that it had to deploy more and more local data centers to reduce latency and improve the reliability of its service. Should a problem occur with one local data center, it wanted to fix it without impacting other areas of Wix’s infrastructure, particularly security and reliability.
“The Wix internal solution, which is still in use, wasn’t initially at our multi-cloud production reliability standard. Having backups to ensure the reliability of our service is of top priority, so we began looking for a partner who would help us to increase reliability and that’s how we started out on this journey with the Google engineering team,” reveals Ginzburg. To help guarantee resilience and security and no breaks in service, as well as save time, complexity, and money, Wix chose to work with Google Cloud.
“Our engineers worked closely with Google engineers to develop the Cloud Certificate Manager (CCM), which is capable of terminating millions of certificates. This joint engineering effort created the ability for us to ensure a higher SLA to our customers, as well as performance improvements.”
Panich adds: “With its truly global presence, we saw that Google Cloud could ensure better connectivity and TLS termination times for Wix than we could achieve if we built the infrastructure ourselves,”
Wix charged Google Cloud with doing something that no other cloud provider had accomplished before: handling the millions of TLS certificates that Wix manages and packaging up edge networking and cloud infrastructure in one.
In this white paper, we explore how Wix worked collaboratively with Google Cloud to achieve this in a way that benefits both organizations.
With its truly global presence, we saw that Google Cloud could ensure better connectivity and TLS termination times than we could achieve if we built the infrastructure ourselves.
Stanislav Panich, Head of Networking at WixHow websites are usually securely served
Transport Layer Security (TLS), formerly called Secure Sockets Layer (SSL), is the main protocol that ensures the security of your internet connection and any sensitive data that you serve on your website. Web servers use TLS certificates to prove their ownership of the domains they serve.
When people sign up for the Wix hosting and website building service, they either bring their own domain name with them, which means that they have to upload their existing certificate to Wix, or they create a new domain through Wix, in which case Wix issues them a certificate. With so many customers, Wix has to handle millions of TLS certificates.
Prior to moving its TLS infrastructure to Google Cloud, Wix spent a lot of time and resources building systems that could handle all these certificates. “To ensure that we would be able to serve so many customers with TLS certificates we had to create a dedicated service for TLS termination. We then also had to integrate with two external providers to renew these certificates,” says Panich. “Working across three or four subsets of regions, we had to manage availability during high traffic spikes, which sometimes affected service stability. And for us performance is everything, our customers expect their websites to load fast and we can’t afford any problems with our service ever.
Wix takes any threat to the reliability and stability of its service very seriously, and so it began looking at how it could better manage its rapidly growing TLS certificate service, either by making improvements to its in-house operation or by partnering with an external company. “Having an in-house solution made sense for us when we started out, but it couldn’t keep up with the growing demand for our services,” says Panich. “So we began looking at alternatives that would improve stability and make sense in terms of performance and cost-effectiveness compared to our own setup.”
The unified power of Google Cloud and Wix
Wix wanted an integrated cloud vendor that could process millions of TLS certificates and offer edge networking and a comprehensive cloud infrastructure, without compromising on service or security. At the time, this simply didn’t exist.
In 2017 the company entered discussions with all of its cloud partners and other global cloud operators, who were all keen to collaborate with Wix on creating such a service. Wix chose to work with Google Cloud largely because it had recently started using a number of Google Cloud regional data centers and was impressed with how flexible and innovative Google Cloud was proving to be, while aligning with Wix’s focus on reliability.
This joint engineering effort created the ability for us to ensure a higher Service Level Agreement (SLA) to our customers, as well as performance improvements.
Jonathan Ginzburg, Head of Production Engineering, Wix“We had a number of parallel projects running with Google Cloud, we were running our Cloud DNS service on Google Cloud, for example,” says Panich. “And we could see Google Cloud was really committed to working collaboratively with us and investing in meeting our TLS challenge.”
Wix outlined the requirements for Google Cloud to address and specified how it wanted the Cloud Certificate Manager to be designed and run. The Cloud Certificate Manager needed to be able to acquire and manage TLS certificates, handle Wix’s scale and focus on three key areas: security, latency, and edge networking. “Google Cloud invested a lot of time and money in this project, and the Google Cloud team became almost an extension of our in-house team. They were amazingly professional, patient, and collaborative, and dedicated to ensuring that we worked together to build the next generation of TLS termination,” says Panich.
Marcin Walas, Software Engineer at Google, adds: “Our Google Cloud engineers worked alongside Wix’s engineers as a virtual global team, we had regular meetings to demonstrate our progress and to figure out what testing was required at each stage of the journey.”
Ginzburg adds: “This joint engineering effort created the ability for us to ensure a higher Service Level Agreement (SLA) to our customers, as well as performance improvements.”
Aligning the Cloud Certificate Manager with the Cloud Load Balancer to suit Wix’s requirements
To allow Wix to serve its millions of domains using Google Cloud’s networking capabilities, Google Cloud extended its existing Cloud Load Balancer solution to work with the Cloud Certificate Manager. “The Cloud Load Balancer runs as a front end for applications right at the edge of the internet, which helps Wix to run its applications faster,” says David Gingold, Software Engineer at Google.
The Cloud Load Balancer is now core to the Google Cloud solution for certificate management that Wix uses for its Cloud Certificate Manager. But, to ensure that at no point Wix’s customers experience any instability or disruption to their service, there were multiple stages to run through before Wix was comfortable enough to adopt and adapt the Cloud Load Balancer for its Cloud Certificate Manager.
First, the certificate infrastructure was moved to and integrated with Google Cloud. “For this, we had to ensure that the new system would work according to our requirements, and, as it was an external system, we had to create a secure communication channel between Google and Wix with a clear set of expectations and an alignment of phases for the project,” says Panich.
“This involved working on the systems’ SLA, support channels, escalation paths, and internal Wix communications. We monitored the internal and external systems, and implemented mechanisms that verify that all certificates populated in the Cloud Certificate Manager are correctly installed and distributed, and that, in the case of KPI violations, such as performance or operational problems, there are automatic failovers.”
While still in the launch phase, the Cloud Certificate Manager was heavily monitored to measure everything from the performance of termination to the TCP connection. “We did find some bugs in the code, but we were able to fix them fast without impacting our users,” says Panich.
The Wix global Cloud Load Balancer uses a single IP address that is served from hundreds of locations worldwide. Although this simplifies certificate management in one way, it adds challenges when it comes to monitoring specific regions. “As we are now monitoring multiple locations with the same IP address, we had to invest in developing an automated external monitoring system that identifies where we have problems in different parts of the world and then enables us to fix these issues without affecting our customers,” Panich says. “So now we are confident that should communications drop in a specific region, we can manage it and we can track and quickly escalate unique edge cases.”
Rolling out the Cloud Certificate Manager
Once Wix was satisfied that it had ironed out all the glitches in the system, it moved on to the rollout stage. “We first agreed that the Cloud Certificate Manager system was ready, and Google Cloud and Wix had to align their own dependencies to ensure we were able to start using the product without affecting our users. Then we began synthetic testing, where we created a complex external testing network to ensure we could continuously measure and identify our new system’s availability, stability, and performance. This meant that we were able to avoid potential problems with actual users’ traffic,” says Panich.
Wix identified several customers who were willing to participate in the testing stage, keen to experience enhanced stability, resilience, and performance. Happy with this test rollout, Wix then migrated a further 100 customers across to its GCP and continued doing this in carefully monitored batches.
“We rolled out a small subset of sites with engineers, developers, and networking professionals from the Google Cloud and Wix teams monitoring the process and keeping our customers up-to-date with how everything was proceeding at all times,” says Panich. “Then the rest of the rollouts were split up into groups and were processed on schedule in background mode. And from there, we started moving millions of sites across. We also planned the rollout to have full control and the ability to rollback to Wix termination on the level of each domain by changing the DNS, plus there was another global kill switch to move all Cloud Certificate Manager traffic back to Wix.”
It took two years for Wix and Google to design and build the Cloud Certificate Manager model that Wix had requested. With all the issues resolved the Cloud Certificate Manager was then deployed worldwide and it is now the only cloud solution on the market that can manage millions of certificates.
The results
Six months since its full-scale launch, Wix is happy to report that it can manage millions of TLS certificates with the Cloud Certificate Manager, and, more importantly, it now offers one of the most reliable services worldwide. It has optimized site performance and reliability, saved data transfer costs, and reduced and simplified its on-premises infrastructure.
“Alongside the assurance of reliability, our customers saw an improvement of site loading times, and we were able to reduce TLS termination times by 25% on mobile sites and 50% on desktop sites,” adds Ginzburg. While moving to a managed services architecture on Google Cloud has also freed up Wix’s developers for more innovative projects.
“Next, we want to create a package that integrates the Google Cloud CDN with the Cloud Certificate Manager and improves our cache capabilities, so that they are run as a service through Google Cloud,” adds Panich, who is always looking at the next part of the Wix operation that he can further optimize to improve the overall experience for the end-users.
Panich will now be looking at all of Wix’s different workflows to see which require more resources than is necessary and which could be integrated or adjusted to improve reliability and enhance business decisions.
Once Wix introduces additional features that integrate with the Google Cloud CDN, it hopes to see its TLS management simplify further and a reduction in its infrastructure stack sizes, which will start helping it to further reduce costs too. One of these additional features is Cloud Armor, which Wix is already experimenting with. This will allow Wix to filter traffic and see patterns in that traffic that could indicate potential DDoS attacks.
Google Cloud and its suite of products have hugely benefited from partnering with Wix on this project as well. “Prior to the Cloud Certificate Manager’s creation, a Google Cloud Load Balancer instance could handle up to fifteen certificates. Now integrated with the Cloud Certificate Manager, it can reliably handle 10 million-plus, which is phenomenal,” says Gingold.
Conclusion
Instead of focusing its energies on building its own secure edge network and working with multiple vendors for its CDN and load balancing needs, Wix now has an integrated system that packages both requirements, simplifying its operations and cutting costs. The Cloud Certificate Manager and related Google solutions enable Wix to concentrate on its core mission: hosting websites and developing its website-building platform, while optimizing site reliability and security for its customers.
Google is now working with other partners who are looking to scale their TLS certificate processing. “It was super helpful for Google to have Wix as a lead customer, as it enabled us to align what we were building with what a customer of Wix’s scale needs,” says Gingold. “Google is on a mission to grow and develop with our customers, and our experience with Wix is testimony to how this is best achieved.”