AI Design for Civic Engagement: XR Intelligent Protests 

Challenge:  In this project we asked, How we can we make the protest process more inclusive? 
Outcome: A Unity app for protests, which uses machine learning tools to strengthen the experience of protest, while allowing access for many people unable to participate in protest.
Our Process Site:  posthumanai.com

This new form of civic engagement is called "intelligent protest", which enables users to remotely participate in protests across realities, using computer vision and Dr. Rebecca Feibrink's "Wekinator" machine learning tool. Using Wekinator, we created an intelligent interface to engage in protest using facial movements instead of voices, and implemented our protest in both physical and virtual environments. 

OUR RESEARCH:

This year long research project was created using our collaborative AI Design Toolkit- poieto with unheard voices across the United States. We held 14 workshops, each addressing different points of machine learning, and included talks and research with data, models, materials and outputs. A main focus of our workshops was how we can create, label and control our own data. 

We then asked participants to imagine how they would create their own AI Design using Twitter, Google Forms, audio recordings, and physical design, wherein the participants proposed:

• data

• models

• outputs & materials

Participants submitted their design proposals and collectively voted on them using Google Forms. Drawing on elements from the top proposals (embodiment, protest, civic engagement) we designed a tool using the game engine Unity, that would allow users to collectively engage in civic protest in a virtual space, which could then be represented in a browser, VR headset or displayed in the physical environment. We also prototyped a face and sound-based protest using Wekinator and Processing. 

WHAT WE MADE:

MATERIALS / PROTOTYPING

ELECTROMAGNETIC MICROPHONE PROTOTYPE

In our data workshops we created several prototypes focusing on unrealized (not immediately seen or heard) data. In one workshop, we created electromagnetic microphones to explore our environment and detect preciously unheard sounds. 

PLANT SONIFICATION PROTOTYPE

Many of our participants wanted to design for intelligence using plants. We prototyped with the medical sensor kit BITalino, and printed a circuit board from Pulsu(m) Plantae, and used acupuncture needles and EEG and ECG sensors to digitize plant signals. This research informed how we designed our protests, using interconnected signals to amplify presence. 

FACE PROTEST PROTOTYPE

Our approach to AI Design was to prototype with hardware and software. Many individuals who felt disconnected to AI concepts stated that the physical materials provided a more comfortable entry point. Using wekinator, face tracking software, an FM synthesis processing sketch, participants created faces, inspired by the behavior of protest, from the raising of the eyebrows, to the puffing of the cheeks, and used these movements to engage in a collective protest. 

PROTEST USE CASE (Tree Removal in Alhambra, CA)

Using the Intelligent Protest prototype, participants were able to remotely login and participate in the virtual protest space. This virtual space was visualized and displayed at an actual physical protest site, using AI Design and physical movement to bridge physical and virtual worlds. 

A virtual sit-in was created by using Wekinator with Open-Frameworks’ detailed facial feature tracking software. We setup Wekinator to receive 14 input values and compute 5 continuous output values which were mapped to an avatar in the Unity game engine. We selected Wekinator’s default neural network algorithm and used 5 collaboratively designed facial movements to train the neural network for the face protest. These outputs were used in a designed Unity environment, where each individual who logged in had an avatar of a tree with roots. 

Our first test case was in connection with protesting tree removal in Alhambra, CA. The city parks department proposed the removal of a number of trees from city sidewalks. Many local residents protested this proposed action and we were invited to use our tool to help activists communicate their message to the city council.

PHYSICAL / DIGITAL DESIGN OF THE TREE PROTEST

We used the AI Design Tool - poietio, prototypes and materials of the protest to create the digital protest environment. In our “Tree Protest”, we used the sound file taken from the recordings of trees being watered on the physical site of the protest. 

The roots were created by a simulation of a Lindenmayer System (L-System) and the outputs affected the individual avatars in the collaborative protest space were: 

outputs: Rotation of tree canopy, Root color, Root network growth rate, Level of audio distortion

Cultural AI Design Frames

• One of the themes that emerged in our community research was that of discouragement with the current political system.

• We performed research using processing and Wekinator with our community. We advocated for open source tools that can work across platforms, so all individuals who have access to a computer can use the files. We provided files and directions to individuals who wanted to use some type of code on publicly available computers. Most importantly, specific community members were accessible through phone, email and in person meetings to answer any questions, so people new to the experience did not feel alone.

• We took materials from existing protests (sounds from the environment, images, etc) to include in the design created in Unity. We also used physical materials to prototype thinking around AI (such as physical blocks for pattern matching and quilts for neural networks).

This collaboratively designed experience of presence was designed for a collective experience. The more people who participated, the more audio and visual information individual users would receive from the environment. Having multiple forms, in physical, virtual and augmented spaces resulted in new design opportunities. Additionally, reimagining the form of protest in the digital space (though checkin in at standing rock, to #metoo) explores new thinking in this space. 

This design approach generated new ideas and conversations within communities typically excluded from the AI Design space (such as individuals working multiple jobs, or those with no tech background). In our approach, we used AI Design to address areas of inequality. Our prototype allowed for participants without time or financial means: parents, caretakers, individuals working multiple jobs, to be digitally present in an embodied form for 24 hours.

REAL WORLD APPLICATION

We simulated a protest around “trees” in the Unity environment. In the virtual protest, participants produced branching nodes in the environment in response to activity in the system. We attempted to deploy the virtual protest at a city council hearing, and although city ordinances did not allow “virtual” participants to speak at the hearing, the council expressed interest in using our tool in future applications.

GENEARLIZABILITY 

This embodied protest thinking was then applied to five protest topics that were deemed important by the group. Groups came up with their own intelligent design using wekinator, unity and/or processing. 

RESEARCH INSIGHTS

• Participants wanted to be present in multiple spaces.

• Participants wanted the ability to create different behaviors across different environments based off of one input.

• Participants wanted to connect machine learning approaches to individual body movements.

• Participants wanted the design to offer more feedback/reward, with the presence of more people.

• Participatory AI Design approaches, rather than product driven making can result in multiple types of projects, such as a new interface for protesting, designing for embodiment in AI.

• The process of the project not only resulted in rich work, but provided the social good of modeling inclusivity, while creating literacy in the AI Design space.

• The machine learning design allowed for multiple bodies to strengthen the experience of the protest.

OUTCOMES

We used machine learning tools to engage in collaborative, civic engagement/protest. This project led to new thinking in AI Design, protest, designing across realities and prototyping AI Design thinking without code. The outcome of the project speaks to new ways of engaging in civic discourse, and multiple presences and behaviors across realities. Our co-creative research and XR prototypes in Unity, led to the creation of new experiences for civic participation.

RELEVANCE

Intelligent protests emerged from a desire to have increased access to the protest space. Not all bodies are able to engage in protest due to a multitude of reasons, such as concerns about immigration status, care taking responsibilities, health related issues or having to work multiple jobs. This project allows people to be digitally present in physical and digital protests. 

Project Date: July 2016 - May 2017

Teams: Feminist AI Projects Research group; Cultural AI Design Tool - poieto