Interactive audiovisual augmentation of physical paintings


Concurrently with the beginning of 2019, I started an artist-in-residence at the 'Art And Artificial Intelligence Lab' at the Computer Science Department of Rutgers University. When reflecting on the current use (and future possibilities) of deep learning for visual art generation, I thought of the importance to train neural network algorithms based on my own visual inputs derived from the results of my artistic work. The generative output of deep learning algorithms would then reflect some aspects of my creative process and with it the signaling of novel avenues to integrate this technology into the workflow of my artistic practice. When assembling my own image data set to train a Variational AutoEncoder (VAE) algorithm (results to be shared in the near future), I came to the conclusion that images representing hand-made physical paintings (acrylic on canvas for instance) and paper drawings should also be included along with all the algorithmic art (in digital format) I've done until now.

Because I've been given more attention to algorithmic art creation during the last three years than to traditional painting and its associated physical act, I realized that under the current direction, and when integrating deep learning into my art practice, I would end up becoming 'an artist of the mind', neglecting my body to the sole purpose of keyboard interaction on the computer. Not a desirable path for myself! Consequently, I outlined a new approach to the use of algorithmic art within my creative practice as follow:

>> use algorithmic art to augment the creative act of producing traditional paintings

>> use deep learning algorithms to output visual work from the result of learning aspects derived from the process of my own creation of physical paintings

>> use algorithmic art for creation of artworks at the intersection of genomics and tango (because I don't have an interest in doing these from the perspective of traditional painting)

This stance would from now on entail that the use of algorithmic art will only take place in those instances were I cannot create the art by the physical action of my body in a traditional manner. Thus my renewed interest in creating abstract art paintings in physical form.

This essay describes my first experiments in using algorithmic approaches to handle audio and video media content in order to augment abstract paintings created with acrylic on cardboard.

Re-activating the mind-to-hand connection_

Despite the fact I've been drawing and sketching on paper since my teen years, it was only in 2013 that I first experimented with painting on canvas, exhibiting my work (15 paintings) at OQ Coffee gallery in Highland Park, New Jersey (my local town) during the same year. After that I've only painted sporadically, fully focusing instead on digital, algorithmic and multimedia art from 2016 onward. In 2017 though I reflected on the importance to keep the mind-to-hand connection alive [see references] and published some hand-made drawings and sketches. For the reasons explained above, I reassumed abstract painting during January-February of 2019 and produced 5 acrylic-on-cardboard artworks (Figure 1).

Figure 1. Photographs of acrylic on cardboard abstract paintings created during January-February of 2019. Each artwork is proposed to be 'augmented' with the narrative of their own creation through the use of interactive audio and video content. These works are early attempts at integrating augmented reality technology into the creation of physical abstract paintings. Click on images to enlarge them for complete view.

One of the artworks shown on Figure 1 (center-left) was selected as proof-of-concept to be augmented with the narrative of its own creation through the use of interactive audio and video content. Under this light, technology serves the purpose to complement the output of a physical action taken by the artist, that of engaging his body with the process of drawing, painting and handling of cardboard as support material.

Interactive audio-visual augmentation_

User interaction with a digital representation of the physical painting (a photographic image of the painting) takes place when audio and video events are triggered in response to mouse clicks at certain visual elements within the frame of the image. Specifically, when mouse clicks occur within solid black squares a unique audio file (one for each solid black square) is triggered containing a verbal explanation from the author about one or several aspects of the creation process that led to the completion of the work. Furthermore, when a mouse click occur within the solid red circle on the bottom-right of the image, a video file is triggered displaying the author's explanation of the original sketch for the painting drawn on paper (Video 1).

The objective of using technology to augment the painting is to provide the user with information about the context in which the author created the artwork. Augmentation should not compete or replace the emotional potential of the painting attained through content, form, texture and color.

Video 1. Quick video phone recording of Processing Sketch running on my desktop that augmented the photographic image of acrylic on cardboard painting by triggering video and audio files in response to mouse clicks by the user. Media content complemented the digital representation of the artwork by providing a narration of the context in which it was created. This prototype serves as proof-of-concept to keep improving upon.

By prototyping the interaction via mouse clicks to trigger media content associated with visual elements within the image, the 'augmentation' concept came to life in its most basic form. This approach could be incorporated into an exhibit in which a laptop computer is placed beneath the physical painting and let the user wear headphone and interact with the digital image to listen to the audio stories being told relating to the physical painting in front of him (Figure 2).

Figure 2. Sketch depicting possible installation to interactively augment a physical painting with audio and video content triggered by mouse clicks on the digital image of the physical painting being showcased. The visitor wear headphones to listen to the audio and video content about the physical painting she/he is experiencing.

Next steps_

I would like to keep exploring additional possibilities for interaction using the same technology presented here. Also, to take the interaction to the browser so the visitor can interact with all the paintings in the exhibit by scrolling down the page on a browser. Without doubt the end goal is to incorporate augmented reality technology to complement my creation of physical paintings, combining digital and physical worlds.