A Different Kind of Canvas
Explaining the Techniques Used on What Dreams May Come

What Dreams May Come is a visually rewarding exploration of the Afterlife, guided by director Vincent Ward’s (Map of the Human Heart) touch. This multi-layered film involved approximately 250,000 individual elements designed, rendered and composited by a team of 30 animators and compositors over the course of nine months. Not only does this film introduce a highly interpretive subject matter, but it also introduces new worlds of visual effects. “The style and level of artistry that was the heartbeat in creating Vincent’s vision of Heaven and Hell, at times led us to enhance what was available to us at the visual effects houses by way of hiring independent artists,” explains visual effects producer/supervisor Ellen Somers. “The process of choosing companies and artists was more similar to building an art department: matching talent and style that were right for the project rather than the decision being company- based.”

Digital visual FX supervisor Nick Brooks and visual FX supervisor Joel Hynek started work on this project in the summer of 1996, looking for partners to help develop it. Pierre Jasmin, technical supervisor and effects developer, joined Brooks in October 1996 and together developed the techniques and tests which helped to greenlight the movie. When Brooks and Hynek were preparing to shoot, Jasmin went back to Quebec to work on the software. They joined forces again in San Francisco in May 1997 and hired senior software developer Pete Litwinowicz who had done parallel work in painterly effects in the research division of Apple Computer. Dan Piponi from Cinesite Europe and George Borshukov from Berkeley University were hired to develop 3D integration and reconstruction tools.

Stuart Robertson, visual effects supervisor at POP Film, where most of the visual and digital effects were carried out, sums it up this way, “This is the first film where people are going to be moved by visual effects. So far, it’s been about ‘wowing’ people with effects, but this is not about technology, this project is really about creativity. For the past eight to nine years people have seen all these images and said, ‘What amazing visual effects.’ The end purpose for this film is to get people to say, ‘What a beautiful scene.’” Much as good cinematography is about painting with light so subtlety that it does not interfere with the viewing of a film, Vincent Ward’s idea for the effects in this film ran along much the same lines. Robertson adds, “In this film, we have used all this technology as a vehicle for creativity.”

For The Painted World, Chris’s interpretation of his spiritual reality, Ward turned to the team of Brooks and Hynek to help him extend his vision. Brooks recounts, “It was very clear from our earliest conversations with Vincent that What Dreams May Come would require us to develop images that no one had ever seen before. Chris’s (Robin Williams) voyage through the Afterlife would have to be an emotional and spiritual voyage through realities yet to be visualized.”

Vincent Ward’s vision encapsulated an art world that Chris would draw upon for his Heaven. It was to reflect his passion for 19th century painting and the wonder of a released soul.

Brooks didn’t feel that any existing techniques used in visual effects and computer graphics would allow them to bring this vision to life. “Traditional 3D computer graphics may provide the basis for creating completely new environments, but to design, model, light, texture and render 8 minutes of a fully photoreal and interactive environment seemed bound to lead to huge expense and overblown schedules.” Brooks felt that their best possible avenue would be to rely on the beauty of live action photography as the source of all subsequent techniques.

Their approach was to start with original first unit live action photography shot on location in Montana. Through Machine Vision analysis, a unique, indepedently owned image-based paint animation software, 3D reconstruction techniques, existing computer graphic and digital compositing packages, they were able to combine new and old techniques into a hybrid technology. Says Brooks, “We digitally reconstructed the photography, enabling us to create new spatial, temporal, textural, and chromatic relationships that are based on a 19th century painter’s vision of the world.”

With this project, Brooks and his team—including technical supervisor, effects developer Pierre Jasmin, senior software developer Pete Litwinowicz, visual FX supervisor Joel Hynek, visual FX producer Donna Langston, art director Joshua Rosen, and CG supervisors Mike Schmitt, Karen Ansel, Scott Gordon and Peter Travers—were able to create the first level of a powerful set of new imaging tools capable of transforming live action photography and integrating synthetic objects. “We have experienced some very interesting revelations on how to combine and manipulate live action photography with 2D and 3D synthetic imagery. These revelations have allowed us to create fully 3D objects (such as wet paint strokes) that match motion and spatial changes within the scene.” The effect of these new techniques are so advanced, that Brooks is confident the net effect of What Dreams May Come will be felt on other productions in the future.

They used techniques that were a hybrid of new and existing tools. The two main areas of the process were Scene Preparation and Paint Generation.

Using Scene Preparation, which accounted for approximately 50% of the work flow, the artist prepared a shot for painting. Then they deconstructed the shot in terms of the scene’s dimensionality, the motion of the objects in the scene (plants blowing in the wind, waves on the lake, etc.), the motion of the camera (no motion control was used on location, so every frame of the camera’s motion was reconstructed), characters interaction and areas of the frame to be painted in with different color and brushstrokes. More often than not, they did not have a green screen of the actors and the dog to help extract them from the scene, notes Brooks, so a large amount of matte generation was required to extract the actors from the scene, before painting, so that they could be composited back over the painted frame.

With the use of Tracking and Machine Vision, they used state-of-the-art advances in computer vision technology to track the motion of objects in sequences from frame to frame. They extended standard vision techniques with a suite of tools that allowed the user to fix, modify or otherwise enhance what the computer automatically calculated.

The results of tracking and subsequent hands-on manipulation were sets of motion maps for each sequences. The motion maps were one of the key components in their process. They provided information on how water, flowers, grass and trees were moving or blowing in a scene, regardless of whether the camera operator was moving the camera. “This led to incredible savings in shootings schedules,” says Brooks, “as we had vastly reduced the need for motion control, with all the associated setup and planning times.” Thus, the director was able to work unimpeded by the technology.

For the 3D scene and camera reconstruction, the team was able to reconstruct the camera’s motion and lens characteristics, based on careful survey and laser range finding data collected from the location at the time of the shooting. This allowed them to then reconstruct a 3D CG wireframe of the scene. Orange balls were placed in the scene with the actors to aid with this reconstruction.

For the 3D Laser Capture, the team captured 3D laser scans of the location in order to get a cloud of x,y,z points that locate objects in the scene. A Laser Scanner is essentially a laser camera that collects 3D clouds of points by scanning a laser over a location and analyzing the laser path. The resulting data looks much like a radar image in 3D. Once this cloud of points was in their posession, they developed further techniques for handling the large amounts of data, sorting and surfacing it in terms of the individucal component objects of the scene (i.e., trees, trunks, flowers, ground, etc.)

As for the techniques involving Paint Generation, all the above scene preparation methods were used to lesser and greater extent depending on the complexity of the scene and Chris’s interaction with it. Some shots required a low level of machine vision tracking, while others required a full laser scan. Essentially, they dealt with the technical specifications for the scene on a shot by shot basis. Once they had a degree of construction, they were ready to generate moving paint strokes. They used three custom systems, two of which are image-based animation systems, the third is 3D.

The paint particles from the systems were textured with real oil paint scans. A library of paint strokes crafted to match artists such as Monet and Friedrich was compiled. With this system, they could now do what had been impossible before: paint in time. Paint strokes could animate, grow, die and evolve in a way that artists before had only dreamed of.

With the careful preparation and deconstruction of scenes, they were then able to attach paint strokes to photography of blowing flowers and other organic objects and track the objects over time. This allowed for the introduction of new objects not photographed into the scene. They were able to generate new flowers, lakes and mountains using a 3D procedural animation system and integrate them into the scene. To render the 3D objects, they used custom shaders from high-end rendering packages such as Renderman and Mental Ray.

For the Matte Painting, which began as a traditional glass matte painting (done by Sid Dutton of Illusion Arts) 12 digital matte paintings were created, which enabled them to transform the photographed vistas into more dramatically and compositionally balanced painted scenes. In a number of cases, they were able to project these paintings into 3D scenes, so they could relight the vistas with beams of light and integrate synthetic waterfalls, mist and dynamic light washes. For the final compositing, they used high end compositing software, and combined all the 2D and 3D work, then color-timed it into the original photography.

Digital Domain also had a starring role in creating The Painted World. Within the Painted World, Albert’s character becomes visible when Chris begins to accept the fact that he has died. Before this, Albert can only be seen as a blurred figure. It was up to Digital Domain’s Klaas Henke to create the surreal temporal effect utilizing innovative compositing techniques.

Also in this world, there is a sequence which involves a tree that Annie has painted for Chris. It becomes a part of his landscape in his Afterlife. According to visual effects supervisor Kevin Mack, this was their most ambitious sequence. He notes, “Rather than creating a 3D model of a tree with branches and flowers, we grew this tree digitally using an L system which is a formal computer grammar used to define the development of branching structures and trees. This system was developed in the 1940s by a botanist named Meyer Aristid Lindin. This language was eventually expanded to a computerized 3D system.”

As Ward described his vision for the tree, Digital Domain was able to grow it to his specification from the genetic description. Mack adds, “The way this tree grew exemplified the growth of an actual, living tree. We had twelve pages of computer hash that created rules by which this tree should grow and the computer executed this long set of instructions.”

In the film, each leaf reacts indepedently to blowing wind as a real tree would. To make it appear to be an actual tree, they used a particle system to simlate wind blowing the leaves. Each leaf is a particle in a system; so they would place a leaf in every particle subject and expose it to turbulence and mass inertia which would result in the leaves being blown multi-directionally.

Another challenge for Digital Domain, was to bring the painted bird to life. Explains Mack, “We animated a 3D bird and texture-mapped it with colors and textures to the 3D geometry or surface of the bird. Since the bird changes colors, we used feathers—like he was made out of paint. We created these matte paintings with brush strokes to simulate painting. For example, when the bird flies into the oil-painted branches, the leaves splash bits of color.”

As for the texture-mapping process, “It begins with making a painting or scanning a texture into a digital image which is then wrapped onto an object or model; almost like wallpapering a sculpture. We had to work out where all the creases were and create a map from the top to the bottom of the bird,” adds Mack.

The majority of the effects were done by POP Films, which oversaw the digital creation of several of the major scenes including Marie’s World, Bridge City, Chris and Annie’s Cottage Interior, The Library, Pre-Hell River, Ship Graveyard, Sea of Faces, Descent into Hell, Deep Hell Cathedral and Reunion.

A final company involved was CIS, which primarily contributed to the Reunion pullback, consisting of a photoshop composite and matte painting. “This scene mimics the Friedrich Tree pullback, the difference between the two effects being that it is more of a ‘photo real’ world, but still painterly in nature, if you will,” explains Somers. “At the end of the film, when Chris has come to grips with death and has found his wife, they utilize a reverse view pushing into the cottage. Since this was done earlier, the second time it becomes a more ‘real’ vision of this environment.” The Reunion pullback is credited to matte painters Michael Lloyd and Sid Dutton.

Robertson concludes, “I hope this is one grain of sand towards the idea that the effects become transparent so that the story, character and imagination all just wrap you up in the illusion and storyline like this film does.”