Nicci Yin is a designer
of interactions and critical media, engaging design to bring together art and emerging technologies. She has contributed to curatorial projects, most notably with Space Caviar and Creative Time Reports, and produced feminist media while a fellow at Barnard Center for Research on Women. Most recently, Nicci's work has been shown at Ars Electronica, the Post-Internet Cities Conference, and Microsoft Design Expo. She is currently based in Seattle.

CV, Instagram, Email
hello [at] nicciyin [dot] com

Hypertext Swarm

Part of Animistic Collaborators in Mixed Reality

Media Design Practices Summer Research 2018 led by Phil Van Allen
Team: Stephanie Cedeño, Godiva Veliganilao Reisenbichler, and Nan Tsai

Hypertext Swarm

is part of a research project, Animistic Collaborators in Mixed Reality, exploring new affordances, interactions, and forms of artificial intelligence (AI) in 3D space that conceive of AI as collaborators rather than servants. My project in this suite, Hypertext Swarms, imagines collaborators in augmented reality (AR) that use artificial intelligence (AI) to scan and analyze artwork. If these animistic collaborators are trained in art history, curation, and criticism, what connections would they draw if they could "see" art?

Art history and criticism, as well as curation, are creative activities in that ideas need to be inferred from visual forms, and connections need to be drawn across disciplines to create compelling arguments. These arguments are not always through text, but through the juxtaposition of texts and art, or art with other art.

The Hypertext Swarm pulls and collects various references, associations, and interpretations in order to help an art historian or curator create a thesis around an art object. The analyzed work exists as a scan in augmented reality, able to be accessed anywhere. The AI itself would be trained on art historical data.

The Swarm involves individual AI creatures (seen below) that swim around a virtual scan of an artwork. These creatures then begin to pull semiotic, thematic, and historical references, putting them into bubbles.

These bubbles can be arranged according to how confident the AI feels about these associations, and the bubbles can then be popped and rearranged in a VR or MR workspace by the curator (see video demo above).

Process, research, and sketches—

This sequence describes an initial idea of a "reef" where the fish would dock, as well as a more literal fish-like behavior. Also diagrammed below are various connections, patterns of the fish, as well as interactions between the AI and the artwork:

The following rough collage mock-ups are designs for potential UI in mixed or virtual realities. These initially explored how the user's view might show an overlay of text, computer vision-guided outlines, and image references on top of zoomed in sections of the artwork:

Testing swarm behavior (the pink capsules serving as the swarm, following a blue ball) in Unity3D:

The final project was created for the HTC VIVE, but speculatively for use in mixed reality environments.

All images and content
(c) Nicci Yin.