Post-Internet Cities Conference, MAAT Lisbon (2017)
NETWORKED COLLUDING IN THE INTERNET OF THINGS began as a design fiction (inspired by noir, mafia archetypes, and the classic whodunnit mystery) that investigates a ring of "AI-gents." This fiction developed into a toolkit for potential workshops, as tools for making transparent how smart IoT devices collect data and talk to each other.
Networked Colluding tackles the tricky problems of visualizing the behaviors and activity of artificially intelligent devices. As a contemporary topic, AI is often shrouded in mystery; this project is a design for the demystification of ubiquitous AI, and asks, what if we could use solving a mystery as a teaching tool to understand and democratize AI-embedded tech?
The mystery to be solved is as follows: The neighborhood watch has discovered a leak of personal, household information: floor plans and layouts of intimate spaces are now available to the public. Who—or what—is responsible for this?
We created four characters (the "AI-gents"), each representing different types of machine learning and intelligences, that together become a network of household smart devices. These characters include: a video doorbell (like Ring), a cylindrical companion (modeled after CUIs like Amazon Echo/Alexa), a speculative smart broom, and a "boss" (later revealed to be a Roomba).
Each device has a persona/profile, showing where they live in a domestic space, as well as the kinds of data they collect, see, or listen to. This allows us to approach learning about and dissecting artificial intelligence through the lens of forensics.
↑ Devices taken out from the illustrated scene (header image). Click to zoom in on the different characters and data.
In our research process, we prototyped with existing artificial intelligence and machine learning tools to understand the behaviors of these IoT devices. As part of a toolkit, we envision this prototyping process as interrogation techniques—how can we play with available machine learning tools as a way to understand how AI works?
We broke this down into two interrogation stages, using a combination of physical computing and ML to question our suspects: the video doorbell and speculative smart broom.
PART I: Video doorbell
laptop with Xcode
↑ The prototype uses a basic webcam inside a "crystal ball."
Instead of using a consumer Ring, we created a "Crystal Ball Webcam" as an accessible, cheap tool. This also plays off the trope of surveillance cameras, crime prediction, and future-predicting psychics.
A webcam was connected to openFrameworks in order to get a real time data. As the webcam rotated on a 180º servo, it would see and identify—through object recognition—objects and potential "threats" in its field of view.
When testing the crystal ball in different environments, the classify algorithm in openFrameworks would identify various objects and humans through the camera, usually making mistakes. In one instance (window on the right), computer vision identified a mosque which was actually an outdoor amphitheatre. This confirms an emerging critique of AI: are devices only as smart as the scope of data that is fed to them?
We designed a panorama as an "interrogation tool," embedded with clues to provoke the crystal ball to reveal who the Roomba is:
↑ Laundry and vacuum point to other household chores and cleaning items, suggesting that the secret Boss might also be a cleaning item. The household cat is not only disruptive to a Roomba trying to negotiate the same space, but is also a nod to machine learning tools like edges2cats.
PART II: Speculative smart broom
broom (or stick-like object)
↑ TinyTile button attached to broom handle; the button would be pressed while the gesture was performed, so that the accelerometer and gyroscope could record the gesture.
Intel's TinyTile was used to train this broom to recognize two distinct sweeping gestures. The device generates a neural net in code, which appears as a string of numbers. We drew from this for designing coded language.
We approached code on two levels: secrets as code, and the acts of coding and decoding. Inspired by the neural net, we used the numbers—literally a part of the code—to create a coded language that contained clues about who the mafia boss is (the Roomba).
↑ Screenshot highlighting two sweeping gestures.
The setup of this mystery is quite simple—the answer/culprit is clearly a Roomba. However, we want to use this opportunity of creating a toolkit to prototype with accessible, common items: what this reveals is that machine learning and some degree of "intelligence" can be achieved with a few tools and a computer. AI is not (always) fancy, indecipherable technology.
We hope that this prompts questions about what constitutes "intelligence," imparts knowledge about how information and data are collected, and advocates for transparency and democratizing of tools in future development of aritifically intelligent devices.
We wrote and presented a paper in collaboration with Godiva Veliganilao Reisenbichler and Xiaoxuan Sally Liu, "Right to the Post-Internet City: An Internet of Enlightened Things" at the Post-Internet Cities conference. The paper considers questions of agency, transparency, and citizenship in the area of living, co-existing, and making decisions for/along with AI devices. It can be read: here.
↑ Installation image courtesy of Ars Electronica/Florian Voggeneder.
Networked Colluding was presented as part of a group under "The Internet of Enlightened Things" at Ars Electronica 2017, a festival for arts, technology, and society. The festival's topic was artificial intelligence, and our projects were installed in the POSTCITY location, using a variety of techniques from poster design to projections and interactive objects. See our projects on Ars Electronica's website here, too.