Mosaic - Brier Curling Game (2018)

The Tim Horton's Brier was coming to Regina and as the presenting sponsor, Mosaic wanted something engaging and memorable for the fans on site.

Brought on during the concepting phases, I helped come up with and subsequently continued to push for an experience that involved computer vision. The general concept was pretty simple: Allow someone to design their own curling rock, then allow them to throw it down the ice a few times and see how it performs. The final result was something I'd never attempted before, but I was confident enough in how I'd accomplish the constituent parts along the way that I didn't hesitate to give the go ahead to sell the client on the concept.

Working from the great deal of detail provided general concept, and approaching the problem with a vast background in computer vision no idea what I was doing, I broke the problem down into the milestones required for success, prioritizing deliverables to ensure that we hit a minimally viable product as soon as possible, with the client's additional goals considered past there.

Initial research and planning notes.

The rough plan for tackling the project was:

1. Get images captured from the webcam and process captured images to extract a the drawn shape
2. Figure out how to import that shape into some sort of Javascript physics engine
3. Bolt on a graphics engine
4. Build actual product
5. Add on leaderboard component

Part 1: Computer Vision Proof of Concept

Some people tackle a project by doing the easy parts first. I prefer to tackle the hard parts from the get go. This gives me a better idea of whether I need to be trying to make up time during the easy parts, and overages in the early stages of a project have many more options available for correcting—clients are much more receptive to changes in scope or deadlines in the early stages of a project.

I didn't want to get into a bottomless pit of computer vision papers or libraries if I could help it, so first priority was developing a quick and easy algorithm for converting patron's drawings into some sort of useful format.

I quickly came up with a simple algorithm for digitizing the kinds of images I needed to deal with (permanent marker line drawings on white paper):

1. Convert the image to monochrome to simplify what I needed to deal with
2. Begin flood filling from the center of the image, replacing all white pixels with 'Marker 1' - if you hit an edge, abort
3. Begin flood filling past the edge of the 'Marker 1' area, replacing all black pixels with 'Marker 2' - if you hit an edge, abort
4. Begin flood filling past the edge of the 'Marker 2' area, replacing all white pixels with 'Marker 3' - if you hit any 'Marker 1' pixels, abort
5. Locate all 'Marker 2' pixels which border a 'Marker 3' pixel, and mark them as 'Boundary' pixels
6. Trace a path along the 'Boundary' pixels

An intermediate debug image showing some of the results of the fill algorithm.

After this, I should have a fairly long list of pixel coordinates ('vertices') making up a polygon of the patron's created shape. Presumably we can get from there to a body within a Javascript physics engine in pretty short order.

In reality, the solution actually played out about as planned. There was some tweaking along the way with regards to a few fiddly constants, but the overall concept was sound. The algorithms were implemented in a way that made use of some of Chrome's experimental canvas features, allowing me to run the computationally heavy parts in a web worker and keep the UI itself responsive and smooth even while constantly performing math-heavy operations in the background.

Part 2: Physics

A part that I thought would be dead simple turned out to be the first whiff of trouble. The physics engine was unable to operate on concave polygons, requiring that they be decomposed into a set of convex polygons. That presented some unacceptable constraints on the shapes I'd be able to import and simulate.

Several digitized and imported polygons, stacked as they fell in the simulated gravity.

Thankfully, this is a mostly solved problem and I was able to find a Javascript library for decomposing arbitrary polygons into a set of convex polygons that would work with the physics engine. Unfortunately, it turned out that there were some undiscovered bugs in the physics engine that I managed to suss out. After developing a workaround and reporting the issue via GitHub, my workaround was referenced by the library's author for other people experiencing the same issue.

Once I had that worked out, I was able to use the physic engine's renderer to display the imported polygons and run a fully functional proof of concept - I could hold drawings up in front of my laptop and have them appear in a web page with full physics simulation/interactivity.

Part 3: Rendering

Design assets as provided by the designer.

With the application digitizing drawings and appropriately simulating the physics, next was to start researching, integrating and synchronizing the rendering engine with the physics and other systems and integrating the design provided by the designer.

Part 4: Actual Product

Clients don't pay for technical demos. They pay for a product that fulfills a need. In order for this product to fulfill the need, it needed to be appropriately gamified and polished. Adding in scoring, selection screens, queues, and other aesthetic touches, the product began to take shape.

I revisited the physics setup and calculated the appropriate densities and researched and used the appropriate coefficients of friction to ensure the curling rock simulation would be as realistic as anyone would expect out of a tabletop game.

As the technical side took shape, I began setting the software up on the final hardware, sourced and purchased needed equipment (webcam) and developed the physical specifications for how the webcam should be mounted to provide to the contractors building the final enclosure.

The final product was setup on a 46" Touchscreen coffee table and built into an enclosure to provide more space for branding as well as the appropriate enclosure for the camera.

Part 5: Leaderboard

Screenshot of the leaderboard

Much like a Kickstarter stretch goal, a leaderboard for displaying the day's results was to be developed if time allowed. Thankfully a smoothly executed project allowed plenty of time to accommodate this wish from the client and it was provided alongside the curling game product.

Set up on a separate display, this provided an additional draw to get people into the booth on site.

Final Notes

Thanks to some unfortunate timing and being the only developer in the organization, the actual executions involving the Curling experience coincided exactly with the time I would be out of the country and skiing in Montana. Some other developers were contacted to potentially provide on-site support if the need arose, but none felt comfortable taking on this particular project.

All went well, however, as all components of the experience were thoroughly tested and performed flawlessly when it came time for the week straight of executions and interactions with the public that they endured.