Internarrative Installation at The Sanctuary Gallery

To test the installation I arranged to use a gallery space in a local artist studio collective – the Sanctuary, Northampton. This was an invaluable trial run as it gave an opportunity to iron out some glitches, document the interaction along with the audience reaction. It was clear from the start that the guests found the animations entertaining and, through the interactive process, began to understand their own creative role in this symbiotic collaboration. As the story unfolded some guests started to narrate what they saw on screen before them, identifying with the characters and their plights. This vocalisation was an unexpected but exciting result that seemed to affirm some initial success!

Animation Database

The following videos show the compiled variations for each style of animation. I wanted to use a variety of animation styles so it was clear when a narrative segment had changed and the story was progressing. I also feel it reflects the different types of media we are exposed to and highlights our instinctive ability to find a narrative thread.

Each video plays all of the potential audience combinations (shown in the top left) for that particular narrative segment.

Refining My Project & Technical Development

After further feedback I’ve decided to focus on my original idea of creating an interactive cinematic experience, harnessing Machine Learning as a creative partner. The feedback has highlighted two areas that need further consideration; The first is the potential size of the installation being difficult to test, document and fit into an exhibition. Its large scale could also be difficult to monitor from a Machine Learning perspective. The second addresses the relationship between audience and machine. I have come to realise that there needs to be direct feedback from the machine for the audience to understand they are having a collaborative effect on the narrative progression. With those elements in mind I have envisaged the following interactive installation:

updated_installation_clear

Being the conductor in this interactive experiment, there are three elements that need to be balanced in this collaborative system, each with their own technical intricacies.

Machine

Recently, there has been a lot of development in tools designed to make Machine Learning accessible to artists, designers and musicians. A good option for my project is the Wekinator created by Rebecca Fiebrink. It’s been really fun experimenting with the software and it clearly has a huge amount of potential for my proposed interaction. Here’s some of my initial learning models to give an idea of the user interface:

screenshot_collage_clear

The software uses Open Sound Control (OSC) to communicate with other programs. I plan on using Processing to control the database of narrative elements, so I will need to download and install the oscP5 library. The following is some sample code that I will need to modify and incorporate into my Processing sketch.

import oscP5.*;
import netP5.*;

OscP5 oscP5;
int x, y;

void setup() {
 size(1280, 720);
 oscP5 = new OscP5(this, 12001);
}

void draw() {
 background(255);
 rect(x, y, 100, 100);
}

void oscEvent(OscMessage theOscMessage) {
 if (theOscMessage.addrPattern().equals("/adress")) {
  x = theOscMessage.get(0).intValue();
  y = theOscMessage.get(1).intValue();
  }
}

I will be using a narrative progression system based on a gaming style decision tree, with the machine making decisions based on the current and previous position of the audience.

Narrative Elements

In order to keep the communication of the narrative as clear as possible, I will be using a series of shapes to act as narrative protagonists. This should allow for flexibility in the creation of the animated elements as well as offering a layer of interpretation back to the audience. I have taken inspiration from the following:

Audience

Once the prototype versions of the preceding elements are in place, I’ll need to test how understandable this system is from an audience perspective. To achieve this, I plan to make a paper maquette (small scale model) of the complete cinematic space, with representative audience members that can be moved around within the space. The benefit of this is it’s much quicker to make and as a working model it will allow me to gather lots of user feedback from different audience members. In terms of affordances, it’s also a lot more appealing to move a representative version of yourself, eliminating the element of self-consciousness with the decision making process.

Internarrative Concept

Context

My research up to this point has addressed our own understanding of digitally driven narratives and how that relates to the recent advances in machine learning. The key projects referenced within this study have been related to database driven narratives i.e. Soft Cinema by Lev Manovich and iAM by Quelic Berga, as well as the recent implementation of Machine Learning within the field of entertainment i.e. Sunspring by Ross Goodwin and artistic applications of Google’s TensorFlow.

ddn-rnn-context

My Position

My practice aims to build on the knowledge I’ve gained to discuss how our perception of Machine Learning is rooted in our own paradigm of narrative understanding. Our own ability to find connections, patterns and order the fragmented information that exists within digital formats is reflected in the Recurrent Neural Network systems that drive Machine Learning. The more a machine is able to demonstrate its ability to understand narrative constructs, the more we perceive it as intelligent, a concept that gives machines much more agency in the creative process.

This position is comparable to the ideas proposed by Bruno Latour in that of Actor-network theory – a concept that gives equal importance to every element or ‘actor’ within a particular network, all working together on the same stage. The emphasis within this theory falls on the ‘network’ as each actor (human or non-human) only takes shape according to the relationship that exist between one another. This theory changes the hierarchy in a system that presents a narrative to an audience, such as cinema, promoting a symbiotic relationship between audience, content and machine.

Cinema’s historic appeal is arguably built on an immersive environment that aims to hide the inner workings of its narrative delivery – simply a dark space with floating audiovisuals for a viewing audience. However, it’s no longer a one-way communication i.e. simply utalising technology to push out a predetermined narrative. Machines have the ability to assess and interpret the content being presented and the audience it is presented to – creating the potential for a feedback loop that affects the narrative experience. This demands an environment that offers immersion with technology, not in spite of it, and has the potential to address questions posed at the very beginning of the project (via Manovich’s Soft Cinema): What kind of cinema is appropriate for networked, internet age?

Proposed installation

The system I propose could be described as a ‘Homeostatic Cybernetic Feedback Loop’. Referring back to my original proposal:

“…it is hoped to transform a narrative experience from one of passive absorption to an active, anecdotal engagement where the act of viewing creates a co-authored narrative.”

I plan to communicate these ideas through a cinematic environment that includes three elements:

Content – creating a database of short animations (with accompanying meta data) using shapes as narrative protagonists that offers a diverse range of options in terms of plot development.

Audience – Assigning areas of the cinematic space to certain characters or plot themes will encourage the audience to move around within the space, aligning themselves with their individual preference. Using the audience’s position in the space offers a democratic way of gauging an opinion on the narrative and is a sensory input that can be used to shape its progression.

Machine – I will utalise Machine Learning through a ‘classification’ training model gathered through screen vision. This will ensure the machine has its own part to play in narrative selection.

I have illustrated these ideas through a patent style drawing (I’m a big fan of Jeremy Bailey so this is based on his parody patent drawings):

patent

My Project in a Sentence

My project discusses the creative application of Machine Learning through narrative interpretation.


Internarrative looks at the way we (as humans) understand and navigate narratives that exists within a digital context – and how that is reflected in recent experiments with machine learning (in particular Recurrent Neural Networks)

NN Contemporary Art – Project Space

I’ll be working in the NN Contemporary Art Project Space this week so feel free to pop in if you’re nearby. We’ll be there from Wednesday 10th to Saturday 13th August (11am to 6pm) with a wrap party on the Saturday from 4 to 6pm. We’ve got a working title of Changing Spaces, encompassing the work in progress of both Louise Bird and myself. If you wanted to find out more you can read the full press release here. We’re both really looking forward to it and I’ll be posting plenty of pictures on here and on twitter!

LB_JS_NN_Project_Space

Daydreaming with Stanley Kubrick

I’m a big fan of Stanley Kubrick’s work so finding out that LCC had an archive dedicated to his body of work was really exciting and play a huge part in my decision to study here. I’ve had the pleasure of visiting the archive, viewing artifacts from the vast majority of his films, including the one I was particularly excited to see – 2001: A Space Odyssey. The collected artifacts range from film props used on set to technical drawings created by leading technology companies of the time that speculated on possible future technologies. Most of the commissions didn’t even appear in the film and simply existed as source material for the overall visual aesthetic, indicating the level of detail that Kubrick required.

Having heard about the ‘Daydreaming with Stanley Kubrick’ exhibition at Somerset House, I was looking forward to seeing how other artists, musicians and filmmakers would respond to his films – and it did not disappoint! It was really inspiring to see such a range of work – sound, sculpture, moving image, even virtual reality – all adding a new dimension to Kubrick’s cinematic work.

One piece that I found particularly inspiring, so much so I didn’t want to leave the instillation, was ‘The Corridor’ by Toby Dye – a walk in cinematic music video playing out across four screens. The dramatic narrative and music lyrics were accentuated by a Kubrick-esque application of single point perspective cinematography. But what I enjoyed the most was its 360 degree presentation across four screens which allowed for an incredibly immersive and unique viewer experience.

The films instillation presented the viewers with an opportunity to find and follow their own preferred narrative within four simultaneous, interweaving narratives on a continuous loop. These characteristics can be seen in database driven narratives, offering sudo-infinite story lines within the control of the original directors vision. I intent to take these concepts forward into my own experimentation.

Final FMP & Thesis Proposal

The following extracts are taken from my final proposal that sets the path for my practical and theoretical research. I’m really happy with the direction I’ve taken and feel it clearly identifies the idea of non-linearity within digitally driven narratives. There is a tension between the recent development in Machine Learning – data driven control – and that of previous experiments in database driven narratives such as Soft Cinema. If you would like to read the full text, it can be viewed here.

Field of Study:

Within Interaction Design this project will work with database driven narratives and the more recent break throughs in Machine Learning. I intend to use this research led project as a way to challenge narrative timelines that exist within digital formats. Although secondary to my major focus – the potential of database driven narratives – the psychology of live performance (such as VJing, and cinema) is an important consideration in relation to audience participation and enjoyment. The wider theories of Generative art – art created either wholly or partially by an autonomous system – touch upon many aspects of this project and therefore it is expected that this will be referred to across both the contextual and practical areas of the project.

Predicted Resolutions:

There are plenty of possibilities for fun and entertaining interaction via a range of sensory inputs. This research project proposes to take these ideas and apply Machine Learning processes to engage the audience in co-authoured narratives that might include a series of tests combining a range of visual and audio content such as; audio/video shorts, VJ performance or game. The tests will mainly focus around storytelling through interactive technologies. Utilising software and hardware combined with the feedback loop from sensory inputs, for example; face detection, proximity sensors, audio input – it is hoped to transform a narrative experience from one of passive absorption to an active, anecdotal engagement where the act of viewing creates a co-authored narrative.

Knock Lock

The technical aspect of my concept requires the reading, storing and auto playing of audio files. I need to work out how I can get a device (Arduino or Raspberry pi) to play the latest mp3 (dated files in one folder) from a USB stick when it’s plugged in. After getting some advice from Gareth – I was pointed in the diresction of NodeJS running on a Raspberry pi (resource links: github.comnpmjs.com) which would involve some logic like this:
– detect USB inserted
– read all files (filtering in MP3s only) from USB
– sort files by alpha or date
– play file from top of this list.

So, whilst I was looking into that, I decided to put my Arduino skills to the test and create something that would add a cool layer of interaction to my object – a Knock Lock! This would lock the Geocache, only to be unlocked by someone who knew the secret knock.

The circuit I created uses a switch to turn a servo motor arm to a ‘lock’ position. Some LEDs to indicate status: Red – the box is locked, Green – the box is unlocked and Yellow that would indicate if a valid knock had been received.

I’m really pleased with the way it works so check it out:

Knock_Lock_Arduino

Geocaching – Development & Feedback

Today’s session was a feedback extravaganza! It was a good opportunity to get some final input from a number of tutors, as well as Riccie from IBM. Riccie works in the IBM Interactive Experience studio in London, one of the largest digital agencies in the world, so it was really valuable to get her input on my project. We had an interesting chat with the main ideas revolving around the user journey. Here are a few points.

“Where does the journey begin with this experience?”

“Do you have to bing anything with you in order for this to work? If so, do you get that when you buy into the service and what do you have to do with the object?”

“These are all these areas of exchange that need to be considered, and the easiest way to do that is to play it out – what suggestions need to be in place so its clear how to interact with the object: is part of the website, the object or word of mouth within the community.”

“Having a clear journey and level of anticipation helps in creating interest around the experience, rather than just pushing it out to people and hoping for the best.”

“So, moving forward, what is the essence of the audio collection – is it to impart wisdom etc. and what are the touch points in the process that need to be played out?”

Geocaching_IBM_Feedback1

Geocaching_IBM_Feedback2

Similar to our previous session, we also had the opportunity to give our thoughts on everyone’s project in the class, using post-it-notes as we circulated. Here’s the feedback I received:

post-it-feedback

 

FMP & Thesis Proposal: First Draft Extract

This is an extract taken from the first draft of my proposal that I’ll be submitting for feedback later this month. If you would like to read the full text, it can be viewed here.

Context:

One of the key practitioners within my chosen area of research is Kurt Ralske – a visual artist and musician who has created experimental moving image with his own custom software. “Cultural artifacts – books, photographs, cinema – all these thing in their digital form have become dematerialised – they no longer take up space. At the same time they become detemporalised – they’re freed from time. They’ve become pure data. They’ve become an abstraction. They almost don’t even exist – they have no mass and no duration” (Kurt Ralske – LISA Conference 2012). Kurt Ralske’s work presents these concepts visually by, for example, condensing the entire duration of a film into a single omnipresent image or converting cinematic content into a data set that can be explored from a critical perspective.

Digital environments and networked technology has drastically changed the way we consume different types of narrative – allowing us the opportunity to follow our own line of enquiry. Lev Manovich and Andreas Kratky explore what kind of cinema is appropriate for the Internet age in the project ‘Soft Cinema: Navigating The Database’. The films that were created aimed to represent cinema in the software age and explore “a ‘cinema’ in which human subjectivity and the variable choices made by custom software combine to create films that can run infinitely without ever exactly repeating the same image sequences, screen layouts and narratives” (Manovich and Kratky, 2005).

References:

softwareandart (2012) Kurt Ralske – LISA2012. Available at:
https://www.youtube.com/watch?v=nrL7wiFyMy8 [Accessed: 11 May 2016].

Manovich, L. and Kratky, A. (2005) Soft cinema: Navigating the database. United States: MIT Press.

Research Proposal 04: Methodology

Defining a methodology within the design process is always tricky! In simple terms, it’s the larger system into which you fit your methods… therefore the methodology determines the methods you use. As an analogy I find it helps to think of it as a larger machine that your work runs through – the output will look very different depending on the type of machine.

As we’ve discussed during the course, there seems to be a lack of trusted approaches, ultimately meaning we adopt methodologies for other areas such as social science or engineering etc. We’ve looked at the characteristics of several scientific models, as well as participatory and speculative approached, however, the one that I keep coming back to is ‘Research Through Design’. I think I favor this approach due to its ability to adapt and react to designed outputs through an iterative process.

As a fun group task, we were asked to create manifestos in response to four different approached. It’s really interesting to see that running a simple idea though each set of ideals gave vastly different outputs.


Pragmatic Perspectives
Focus on the identification and resolution of common everyday needs encountered by people in their day-to-day lives.
Manifesto:
There is only one solution to a problem.
Observe – Record – Identify – Simplify – Define – Deconstruct.
Understands the parts that make the whole.
Create a hierarchy of needs.
Establish a system.
Design, prototype and build the solution.
Observe solutions in situation.
Analyse the results – measure the success.

Utopian Perspectives
Present an idealized vision of a possible future centered around some trend, artifact or service.
Manifesto:
We, the utopians stand for the ideal. The best, the most harmonious, the shining future.
We do not stand for chaos, destruction, anarchy.
We, the utopians build the road towards an imaginary, but possible future.
We, the utopians work towards a democratic, selfless community.

Critical Perspectives
Call attention to the complications or possible dystopian outcomes of possible futures.
Manifesto:
Must be aware of current problems to design solutions for the future.
Must be inquisitive – enquire and have a strong sense of purpose and ethics.
Must want to delve deeper into the meanings and workings of contemporary culture.
Must understand what has come before to understand what may come next.
Must be dissatisfied with the status quo and seek change.

Pataphysical Perspectives
Questions and challenges what is understood as real, imaginary and virtual through using the absurd in the context of science (pseudo scientific, “the science of imaginary solutions”).
Manifesto:
The ‘pataphysicist challenges the fact that design can provide an end-all solution to a problem.
The ‘pataphysist questions whether there is a problem in the first place?
The ‘papayaphyisician immerses himself within his imagination.
The ‘polophysician employs scientific methods to tackle imaginary problems.
The ‘patataphysician employs imaginary methods to tackle scientific problems.
It doesn’t have to work.
It is a fact that real pataphysicians can speak of countless things without really saying nothing at all.


This task was great fun and gave me a few more approached to add to my toolkit! Moving forward with my project, I think I will focus on a combination of ‘Research For’ & ‘Research Through Design’ and try to adapt parts of Critical and Speculative approaches.

Geocaching – Paper Prototype & Feedback

This was a great opportunity to get some valuable feedback from my fellow classmates. I’d created a paper prototype in order to communicate my ideas clearly and was intrigued to see everyone’s progress with their own projects. The concept that I was working on was based on developing an audio version of the surrealist game ‘Exquisite Corpse’.

Exquisite corpse is a method by which a collection of words or images is collectively assembled. Each collaborator adds to a composition in sequence, either by following a rule (e.g. “The adjective noun adverb verb the, adjective noun etc“, as in “The green duck sweetly sang the dreadful dirge”) or by being allowed to see only the end of what the previous person contributed.

This gif demonstrates the concept:

And here’s the feedback I received:

Geocaching_Written_feedback

Geocaching – Initial Ideas & Feedback

Here are a few initial ideas, pictured above, in reaction to my own Geocaching experience.

1 – The Geocache Companion – This devise sends you on a journey, pointing you in the direction of random caches while asking you to take photos along the way. The photos are secretly uploaded to a anonymous Twitter handle, only to be revealed when the journey has been completed, turning the object itself into a Geocache.

2 – Geocache Calendar – Designed to encourage regular visits to the cache. The discoverer might see a desirable keepsake in one of the compartments which requires several more visits to reach. This gives the cache an authority and transforms it into secret shrine.

3 – Fairy Tale Cache – This is focused on the fun and sense of magic and discovery that Geocaching offers, particularly when families with children enjoy the experience together.

I had a good initial feedback session with Nicolas. I’m pleased with my initial thought process, however, I’ve been encouraged to think a bit more about the tribe themselves and what they like about hobby. Up to this point I’ve only been considering the interaction aspect of the process and I need to look at the bigger picture. I’ve also been pointed in the direction of Bruno Latour, a philosopher and sociologist who has influenced the development of ‘Actor-network theory’ – treating objects as part of a social network. It’s an idea that gives equal importance to every element, or ‘actor’, within a particular network, all working together on the same stage.