Hypertext Swarm

Mixed Reality
Artificial Intelligence
UX Design
Research Project
Animistic Collaborators in Mixed Reality

Part of a suite of projects exploring new affordances, interactions, and forms of artificial intelligence in 3D space that conceive of AI as collaborators rather than servants. My project in this collection, Hypertext Swarm, imagines collaborators in augmented reality that use AI to scan and analyze artwork. If these animistic collaborators are trained in art history, curation, and criticism, what connections would they draw if they could "see" art?

These AI agents operate as a collective, utilizing plurality to their advantage. They collaborate with art curators by generating various cultural, historical, and formal connections around the work from their database, leading to new discoveries and threads of inquiry.

Research: Hypertext

The knowledge creation of the AI collaborators is based on the idea that information is embedded with a system and is non-linear.

↑ On left: example diagram of how hypertext functions across textual media; on right: online, hyperlinks using similar logic for expanding information scope and path.

The AI collaborators are designed with these ideas in mind: networked knowledge, non-linearity, and free (but relevant) association.

User + Context

A curator typically has extensive knowledge of a period or area of work, and is in charge of the institution's collection. They also often assume other knowledge-sharing and engagement roles such as public speaking and writing. In other words, they are the cultural mediators between art, artists, and audience/context.

Hypertext Swarm is designed for an art curator in an speculative future context, where mixed reality and artificial intelligence are seamlessly integrated into everyday work.

↑ A curator stands in a gallery, adjusting her AR HMD (in this case, a Microsoft Hololens).


The curator is preparing herself for a new show based on the existing collection of the institution. As she browses the current show, she notices a standalone sculpture in the middle of the gallery space, and wants to use that work as inspiration for the next show. Given that she is still in the brainstorming stages, she enlists the help of the Hypertext Swarm through her AR device.

(The Hypertext Swarm is by no means performing the work of the curator, but is, in a sense, conversing with her through idea generation.)

↑ Storyboarding demo video, animation and character designs, and sketches of AI behavior.

User Journey

Swarm Design + Behavior Testing

Hypertext Swarm is designed with future AI functionality in mind, but is a "wizard-of-oz'ed" version of what a true AI-enabled project could be. With that in mind, I had to seek workarounds in code that could create similar behaviors as an AI.

Specifically, I needed to be able to allow the swarm agents to swim around an artwork, which means they needed to: 1. recognize boundaries of artwork surface, 2. autonomously swim around this boundary, and 3. determine their own radius of movement.

This is a test of code I adapted from a script of fish swimming in an aquarium:

↑ Early collages illustrating UI: zoomed in closely to work of art, references would pop up around AI, which are brainstorming as they swim.


This working prototype was created in Unity for the HTC VIVE as a proof of concept. Ideally, this would function in augmented reality so that the curator can view both the real artwork and the digital scan and interface simultaneously.

↑ Swarm leaving the pod to swim around 3d scan.

↑ Bubbles containing semiotic, thematic, and historical references trail behind each AI agent, and are generated throughout the process.

↑ The bubbles are returned to the pod and arranged by height according to how confident the AI feels about its relevance to the curator's objective.

↑ On left: curator can pop each bubble and move the information about their workspace; on right: these references can be grouped according to thematic, visual, and other groupings.

To demonstrate potential functionality, for this prototype I collected some references and notes relevant to this particular sculpture, Suzanne. For instance, relevant sources include "primitivism" as a theme, followed by examples of primitivism by modern artists like Henri Matisse and Paul Gauguin, as well as other visual parallels like the Venus of Willendorf.

↑ Comp of prototype as fully AR: the virtual swarm and bubbles exist as an overlay on top of the artwork, in the gallery space.

Special thanks to the Norton Simon Museum, Pasadena, CA for use of their space. The featured artwork is Suzanne (1909) by Alexander Archipenko.