Mission’s AWS Machine Learning Expertise Helps JibJab Launch Brand New Product Line

Executive Summary

JibJab, the original provider of personalized eCards and other digital entertainment products, needed a managed cloud services partner with deep machine learning expertise to help build out a new head cutting technique for their new product Starring You Books. JibJab required an ML algorithm that could quickly and automatically crop a user’s face and hair from an uploaded image, and then produce print-quality images that customers can place within stories they create. Mission successfully built the ML solution that JibJab required to launch the product – which prepares customized images from customer uploads in an average of five seconds and with 90% accuracy.

“We talked to a few external companies and Mission was our clear preference. They understood our problem, and portrayed very clearly how they could use existing and cutting edge technology to solve it. It gave us the confidence that if we needed something changed or explained, Mission would be able to do it in a way that we’d be able to understand.”

Matt Cielecki
VP of Engineering

The Challenge

To support its new Starring You Books product line, JibJab sought a technology solution for improving both the customer experience and the quality of the images going into these printed book products. Traditionally, JibJab has enabled users to place faces of themselves or others into eCards and other media by uploading photos, and then using a simple interface featuring a “peanut shape” cropping tool. The result was a cropped oval face placed onto characters within selected products (making yourself into a dancing elf, for example).

With Starring You Books, users would be able to create and fully customize multi-page books featuring face images, which JibJab then prints and delivers as a physical product. But shifting to a printed book offering called for higher-quality images and a new design for how faces are displayed – moving beyond the “peanut” to also including a person’s full face and hair (while ignoring everything else in the uploaded image). JibJab also wanted to offer customers a much faster and less manual experience around image preparation.

JibJab recognized the potential to leverage a machine learning computer vision algorithm able to detect faces within uploaded photos, automatically crop both face and hair from the image, and perform image post-processing to arrive at a print-quality result. However, JibJab lacked the in-house expertise to complete this machine learning project. Exploring the marketplace, JibJab also found that it didn’t want to pay the high licensing fees associated with utilizing an existing ML algorithm. This left the company seeking a cloud services provider with deep expertise in designing, building, and training machine learning algorithms, and the project-delivery and support experience to help JibJab achieve its project goals.

Why AWS?

JibJab has been on AWS for seven years following a migration from on-prem infrastructure. During that selection process, JibJab also analyzed and vetted Azure and GCP before choosing AWS due to its services and pricing, as well as the pace of its feature releases and hardware upgrades. “AWS continues to push the boundary of what’s possible with cloud infrastructure, making it easier to use while remaining performant and cost effective,” said Cielecki.

Why Mission?

JibJab had worked with Mission on a previous AWS infrastructure project and trusted Mission as a highly-capable and hands-on partner. “We brought in Mission before, and that partnership went really well,” said Cielecki. “Our VP of Technical Operations, who had led that engagement from our side, recommended Mission when the need came up again to find an AWS Partner.”

As it now scoped out this new initiative, JibJab’s legal department advised that the company would need its own machine learning algorithm created completely from scratch to avoid any potential licensing issues, increasing the pressure to select a reliable partner able to build ML solutions from the ground up.

In talking with Mission about its project needs and the unique capabilities of Mission’s Data, Analytics, and Machine Learning (DAML) practice, Mission impressed JibJab by detailing not just how to achieve the requisite ML solution, but the specific reasons behind what was being done. Mission gave JibJab full confidence that the project would be successful, and that the ultimate handoff to the JibJab team would go smoothly.

“It became evident from day one that Mission wasn’t just going to throw something over the fence for us to use; the team was going to ensure that we understood the rationale behind the processes and technologies put into action.”

Matt Cielecki
VP of Engineering

Strategy and Solution 

Mission began training the machine learning algorithm to automatically recognize faces and hair in user-uploaded photos. First, Mission performed image data labeling by leveraging LabelMe and AWS SageMaker Ground Truth (achieving 70% time-savings with its AI-assisted automatic segmentation tool) to create a training data set and begin data preparation. With that labeled data set of 1000 images, Mission then performed data augmentation – adding blur, sharpening, rotation, noise and other alterations – to expand the data set to 17,000 images. Using that robust data set, Mission leveraged Detectron2 running in SageMaker to detect objects within images and perform instance segmentation to identify faces and hair.

From a workflow perspective, the solution’s process begins when a customer uploads a new image, which lands in an S3 bucket. AWS Rekognition-powered face detection then looks at the image. If it detects a single face, it moves to the next step; if there are multiple faces, the solution asks the user to crop the image down to a single face. The solution then performs an image quality pre-check, assessing whether the image shows the front of the face, if it is blurry, if anything blocks the face, and if the image size supports a quality result. (This step is crucial to product success: enforcing high image quality reduces book returns and increases customer satisfaction.) The image then goes to a Detectron2 model running as an endpoint in SageMaker, which performs instance segmentation to find the face and hair to use in the final image. The next step is image post-processing to remove the image background and blur the edges to improve the final appearance. Finally, the solution extracts facial landmarks that JibJab uses to place the head into the product. The customer can then position the final image within the book they are creating using a GUI and approve it for printing.

Results and Benefits

With the ML-fueled product, JibJab customers can now upload photos and have the algorithm complete its work and provide a final book-ready image in just five seconds. While Mission’s early discussions with JibJab set a goal for the algorithm to deliver high-quality images with 85% accuracy, in practice the algorithm is achieving 90% accuracy.

“It’s a drastic difference just seeing the improvements week-over-week in terms of what this solution can do. Every week the facial landmarking gets better and more accurate, the techniques for background removal get more refined, and post-processing around the edges is smoother. Thanks to Mission’s iterative approach and training with different types of data, we’ve seen significant and continued improvements.”

Matt Cielecki
VP of Engineering

If customers were asked to prepare images themselves instead of the algorithm, that image preparation work would take at least a minute, and without the same guarantee of a high-quality result. Now, customers have a seamless self-service method for creating their entire Starring You Books in just minutes. The ML algorithm enables a much better experience for the customer, and a much better product.

Next Steps

Mission is continuing to improve on the machine learning algorithm powering JibJab’s solution, with the targeted goal of increasing its accuracy to 95%.

AWS Services

• Amazon S3
• Amazon Rekognition
• Amazon SageMaker
• Amazon SageMaker Ground Truth

Third-Party Integrations

• Facebook Detectron2
• LabelMe