Music Video Animation in Substance Designer (Breakdown)

I had the pleasure of creating a video for track 3 of the Kick Horns' new album Tall Tales, titled 'Jofish'.

I was given a lot of freedom to make whatever I wanted to go along with the (excellent) piece, but I did have some guidance. I was told that the music's story could be thought of as the journey an unborn baby takes in the womb, the way it grows and fills with potential, and that ultrasound images might be a good reference point for the visuals. Personally, I also loved the idea of emphasising the instruments themselves in a big collage, and I set about trying to find a way to combine these two ideas. I was also asked to create a strong change for the chorus, so I kept that in mind.

First, I chose three instruments from the arrangement - the soprano saxophone, trumpet, and trombone - and modelled them in Blender. I then arranged the scene (it took a couple of tries to land on a composition I was happy with), exported it, and baked it down in Substance Designer.

The chorus was the easiest to create, since I just tried to replicate the look of real metal. I could have rendered this traditionally with PBR maps of course, but in the interests of stylisation, optimisation, and cohesion with the more abstract sections of the video, I chose to make it manually. I faked a generalised lighting effect by warping noise over the baked normal map (I also used the Light node for highlights), then layered several grunge maps over the top to break up the surface in a more believable way. To colourise it, I simply used a few gradient maps and good reference. Lastly, I simulated a depth of field effect using a non-uniform blur linked to the height map, and I added bloom by blurring and re-adding the brightest parts of the image over the top.

It took a lot of experimentation to come up with the technique I used for the abstract 'ultrasound' portions of the video, but I'm very proud of what I ended up with. I've never seen it done before, and it's pretty mesmerising. The way it works starts off similarly to the metal effect, since I warped some noise over the baked normal map for a non-specific lighting effect. Unlike the metal, however, I then passed it through a tri-planar mapping node. The trick is that the tri-planar projection angle varies over time (using Designer's built-in $time variable). I had it rotate over two axes to make it less apparent what was happening so it would feel more chaotic. I love how, despite the chaos, it maintains a definite sense of rotation as it progresses. I set up the rotation speed so it would complete one full rotation across the entire duration of any given verse or chorus, and I also eased the timings, so it would slow down as it approached the end of its cycle. This created a wonderful sense of anticipation as the chorus approached. I also set the disorder parameter on the original noise texture to vary with the song's BPM, which just had the effect of pulsing the 'light' throughout the song in an unobtrusive way. I think it does a good job of emulating the shifting, noisy background of an ultrasound, or the way light gets refracted through the surface of the ocean. Finally, for a splash of colour, I created a mask from the brightest parts of the image and used it to let bits of the metal effect 'peek' through. I felt this nicely foreshadowed the chorus, and made the chorus feel like it was bursting out of the preceding verse, not just playing independently.

The other chorus and outro are simply variations on these two effects - the second chorus takes the metallic 'peeking' effect and dials it up to encompass almost the whole image, creating a hybrid effect that I really like, and the outro is simply the abstract effect with no 'peeking' whatsoever, so it's completely greyscale. I felt that was a nice way to wind down the visuals as well as the music, and it also made the credits more legible.

Animation is certainly not the normal use-case for Substance Designer. The only way to even achieve it is to make use of the $time variable to vary some parameter, then open the graph in Substance Player to view and export the result. The workflow is not smooth, and the render times are long. Still, I don't think I would or even could have created what I did here in any other software, and I really enjoyed the challenge. I'm really pleased with how this project turned out.