The festival attracts close to 800,000 visitors each season, with hundreds of thousands of tickets being sold and managed via the MICF website. Serversaurus has been collaborating with MICF by providing infrastructure, devops, support and 24×7 management for the festival website since 2016. We’re proud to boast a zero downtime partnership since then, with traffic growing year-on-year. Overall traffic is up 65% since 2016, with views up by nearly quarter since 2018, reaching 4.8 million in 2019.

The website and underlying infrastructure is one of the festivals most important communications channels, interacting with audiences, artists and key stakeholders. The website is critical throughout the entire festival, supporting patrons in navigating and planning the enormous festival program.

Serversaurus achieves high levels of redundancy from a hardware perspective, by running the infrastructure in a parallel configuration across a range of independent servers, including dedicated database and caching infrastructure. The entire puzzle is connected and kept in configuration through distributed service discovery, HAProxy, Ansible and in-house Go applications.

We can’t wait for 2020!

This has been a complex and time-consuming process, requiring us to completely maintain and interconnect two physically disparate platforms, while continuing to provide 100% uptime for our customers.

Storage resilience

Our new v2 platform comes with a range of features and benefits, including storage and virtualisation resilience by design, accomplished through a new interconnected distributed storage network. Our new storage platform maintains two copies of all customer data across a network of SSD powered hypervisors. What this means, is that if a server fails with your virtual server on it, it will be automatically migrated onto standby capacity within the network. 

This design also allows us to hot-migrate customer infrastructure for maintenance. Commonly, underlying infrastructure maintenance on competitor platforms requires you to completely rebuild your infrastructure on new hardware via infrastructure automation tools. Our model allows you to move the entire virtual machine without touching the contents of the machine itself.

Network resilience

Our new v2 cloud has also given us the opportunity to redesign how to best network our infrastructure. Our cloud consists of three network layers: Storage, management & transit.  We’ve coupled storage & transit onto stacked switches which run in parallel, allowing one switch to completely fail while maintaining 100% uptime. Our management network mirrors this topography through bonding. This means we can handle a complete switch failure on both of our core networks without disrupting availability. This level of hardware redundancy runs right the way up through our transit provider and associated 2N datacentre architecture.

Instant VM provisioning

Other than better performance, better reliability and better architecture, in addition, we’ll be offering public access to our VM provisioning dashboard within our Melbourne PoP.

Overview

If you’re a small to medium sized web development agency used to working primarily in single-node environments (often utilising off-the-shelf-CMS platforms in traditional LAMP environments), where do you go to achieve some semblance of scale, when you land a large project without completely changing tack? Often, to scale within a more traditional PaaS / 12 Factor Application environment, developers are forced into a complex deployment and development methodology to scale their applications. This is perfectly fine if you are developing a product/application from the ground up which will be locked to a PaaS provider’s application ecosystem, however, many web agencies are building on top of their existing CMS platforms, or utilising off the shelf CMS solutions such as Expression Engine, Craft, etc.

We recently worked on a fairly large website with a difficult traffic profile (ticketing, high profile media announcements, etc), built in a standard LAMP based CMS stack, which required a simple solution both from a technical and management perspective – a solution we could configure & setup which could still provide enough control for developers to do their work, without having to interact with us for cluster management or requiring the developers to necessarily customise their application for the environment itself.

To achieve simplicity of deployment which was familiar to the developers, along with an architecture the application itself could function in, we developed an application cluster which was completely controllable via a custom management UI. This solution featured traditional web service technologies including load balancing, MySQL, nginx and Varnish caching, which could be completely managed by the web dev agency independently of us.

Deployment Groups

Special requirements included two deployment groups of web nodes (Deployment Groups app-a, app-b), allowing the developers to transparently pull a two-node group offline for management/code deployments, without disrupting the live website. This deployment architecture still left two parallel web nodes live online at any one time in order to cope with high traffic, even if a code re-deployment was required during a busy period.

Caching

Because of the expected high traffic profile of the site, Varnish caching was a mandatory requirement to ‘protect’ the dynamic infrastructure from unnecessary work. As anyone who has worked with any kind of caching will know, there are times when caching needs to be eliminated to debug – we provided a caching ‘switch’ to pass-thru all traffic at the click of a button.

Cache Invalidation

Additional caching management tools included an invalidation dropdown for the live and staging environments, allowing developers to easily flush/ban the Varnish cache on demand.

VCL Includes

As a means to provide a simplified method for developer access to custom VCL rules, we provided an interface for the developers to easily append their custom Varnish rules to the main Varnish configuration.

Technologies

The primary nodes were built using CentOS templates, managed by a suite of customised Serversaurus Ansible recipes which were distributed across a range of physical hypervisors for redundancy. The entire cluster sits behind a HAProxy loadbalancer, utilising the Consul key value store for dynamic service configuration management, managed by a customised Go-based management UI.