Conductor Builds First-To-Market Platform With Mission And Amazon FSx For Lustre

Seizing on the new opportunities available through Amazon FSx for Lustre – the high-performance file system optimized for quickly processing data-intensive workloads – Conductor brought in Mission to expertly leverage the AWS technology and develop a platform prototype in just two months

Highlights

• Mission accepted the challenge of meeting Conductor’s time-sensitive AWS storage architecture goals, and delivered a highly functional proof of concept in time for demonstration at the SIGGRAPH conference – an event critical to Conductor’s business development.
• Mission’s expertise in AWS best practices and optimization, experience working with customers in the media & entertainment industry, and ability to harness Amazon FSx for Lustre gave Conductor a competitive edge in developing a powerful platform on AWS.
• Becoming the first-to-market multi-cloud rendering solution is a key business differentiator in Conductor’s industry.

“Mission’s knowledge of AWS and the media & entertainment industry – along with its talent around FSx for Lustre – took us from an idea to a functioning proof of concept within just two short months. Thanks to Mission, our demonstration at SIGGRAPH was very successful and really couldn’t have gone smoother. I would definitely recommend Mission for anyone needing to build a storage platform on AWS.”

– Francois Lebel, Director of Engineering

The Challenge

Seeking to develop the industry’s first rendering solution with a multi-cloud architecture, Conductor needed to quickly build a prototype of its platform on AWS. Bringing in deep AWS technology expertise was a requisite to achieve that ambition on a tight two-month deadline.

Conductor has five years of experience providing cloud rendering services as an extension of studio clients’ on-prem render farms. The work involves completing data-intensive graphics processing workloads that include the complex special effects seen in major motion pictures. When it launched, Conductor had selected another public cloud platform because it offered per-minute billing, which (compared to per-hour billing) worked better for the company given that many rendering jobs take just minutes to complete. (Per-second billing has since become ubiquitous across the major cloud platforms.)

Even then, Conductor had its eye on a multi-cloud strategy and the potential for achieving new levels of scalability by leveraging multiple clouds and their global compute resources. Many of Conductor’s prospective clients also preferred to store data with a favored cloud provider, so becoming the first multi-cloud solution could increase Conductor’s ability to access and service new business.

The strategy accelerated when AWS announced Amazon FSx for Lustre. Conductor immediately recognized FSx for Lustre as providing the production-ready file system needed to bring Conductor to market on AWS within a significantly expedited timeframe. It engaged AWS, which began working with Conductor as a strategic partner to bring Conductor’s solution to the AWS Cloud. With no previous environment in place on AWS, Conductor also made the decision to start from scratch rather than reproducing its existing platform.

Driven by the potential rewards of being the pioneering multi-cloud rendering solution, Conductor planned to introduce a functional proof of concept of its platform’s new capabilities on AWS at the SIGGRAPH conference (then just a couple months away). However, Conductor realized that meeting that tight deadline would require outside expertise. It asked AWS for recommendations on a managed services provider with expertise in AWS, the media & entertainment industry, and Lustre – as well as a potential partner with the confidence to build a proof of concept for a greenfield environment (utilizing AWS best practices) within a few weeks.

The Solution

On the advice of AWS, Conductor enlisted Mission to build out a new AWS environment utilizing FSx for Lustre.

AWS referred Conductor to Mission as a managed service provider with proven success among customers with various use cases in the media & entertainment industry , as well as expertise with AWS and high-performance distributed file systems. In working with Mission, Conductor found that the provider’s knowledge of the specific requirements and constraints inherent to Conductor’s business – for example, the need to efficiently store highly uneven data sets with either millions of small files or few very large files – meant that Mission was able to communicate and collaborate easily with Conductor, further accelerating proof of concept development.

FSx for Lustre cost-effectively delivers the ability to handle high-volume data inputs down to smaller file systems. Because Conductor’s typical workloads are short lived – running from as short as one minute to up to ten hours – FSx for Lustre’s focus on distributed workloads and high performance (rather than just high availability) stood out as a strong fit for Conductor’s goals in supporting AWS workloads.

Mission’s project team included AWS architects that have been working with Lustre for more than a decade, thus delivering the hands-on experience (and confidence) necessary to ensure the viability of Conductor’s new high-scale architecture. Mission’s technical expertise in this area then enabled the provider to quickly and optimally leverage FSx for Lustre to match Conductor’s objectives. Mission’s close connections with AWS also proved invaluable in getting highly technical queries answered quickly.

The proof of concept was ready for live demos and presentation – featuring full tests on AWS – in time for SIGGRAPH.

The Benefits

With Mission, Conductor achieved a working proof of concept of its platform on AWS, and the confidence to go into production with its pioneering multi-cloud rendering solution.

Throughout development, Conductor gained significant value by trusting Mission’s technical acumen. If not for Mission, Conductor reports that its team would have had to work around the clock to complete the project in time, learning via trial and error. Mission instead enabled Conductor to keep its focus locked in on the rendering services work it does best.

In building Conductor’s high-performance AWS cloud storage solution, Mission completed more than a month’s worth of pure programming work, providing a proof of concept that is close to production-ready. Mission also gave Conductor the confidence to proceed with the AWS data storage architecture – proven to be fully-viable at scale – that it plans to push into production in just a few months. From a business perspective, Conductor expects its new multi-cloud infrastructure to quintuple the number of studios it can support.

Once the solution is in production on AWS, Conductor also expects to work with Mission again to implement an elastic cloud-native file server solution for duplicating data files at the block level, in order to improve the efficiency and cost effectiveness of bringing clients’ data into the cloud.