Reactile: Programming Swarm User Interfaces through Direct Physical Manipulation

Ryo Suzuki, Jun Kato, Mark D. Gross, Tom Yeh  

The ACM CHI Conference on Human Factors in Computing Systems (CHI 2018)

Links:     PDF   Poster   Slide   Video   30s Video   GitHub   ACM DL   IEEE   arXiv   Talk

Abstract

We explore a new approach to programming swarm user interfaces (Swarm UI) by leveraging direct physical manipulation. Existing Swarm UI applications are written using a robot programming framework: users work on a computer screen and think in terms of low-level controls. In contrast, our approach allows programmers to work in physical space by directly manipulating objects and think in terms of high- level interface design. Inspired by current UI programming practices, we introduce a four-step workflow—create elements, abstract attributes, specify behaviors, and propagate changes—for Swarm UI programming. We propose a set of direct physi- cal manipulation techniques to support each step in this work- flow. To demonstrate these concepts, we developed Reac- tile, a Swarm UI programming environment that actuates a swarm of small magnets and displays spatial information of program states using a DLP projector. Two user studies—an in-class survey with 148 students and a lab interview with eight participants—confirm that our approach is intuitive and understandable for programming Swarm UIs.

Introduction

In recent years, Swarm User Interfaces have emerged as a new paradigm of human-computer interaction. While the idea of coordinated miniature robots was originally proposed in the literature of swarm and micro-robotic systems, HCI researchers have explored the use of these robots as a user interface. However, this opportunity is currently limited to highly skilled programmers who are proficient in robot programming. For typical programmers inexperienced in robot programming who wish to build a Swarm UI application, it is unclear if the robot programming approach is the most appropriate for UI programming. To design interactive UI applications, pro- grammers often must think in terms of higher-level design for user interaction, whereas robot programming tends to focus on low-level controls of sensors and actuators. Historically, a novel UI platform is adopted only after the advent of an effective programming tool that empowers a larger developer community, and even end-users, to create many applications for the platform; for example, HyperCard for interactive hyper- media, Phidgets for physical interfaces, and Interface Builder for GUI applications. We stipulate that current approaches to programming Swarm UI are too robot-centric to be effec- tive for building rich and interactive applications. Then, what would be a better alternative?

Reactile

This paper introduces Reactile, a programming environment for Swarm UI applications. The goal of Reactile is to explore a new approach to programming Swarm UI applications. To design an appropriate workflow for Swarm UI programming, we look into existing UI programming paradigm for inspiration. The common workflow of UI programming can be decomposed into four basic steps: create elements, abstract attributes, specify behaviors, and propagate changes. Based on these insights, we propose the following four-step workflow for Swarm UI programming: 1) creates shapes, 2) abstracts shape attributes as variables, 3) specifies data-bindings be- tween dynamic attributes, and 4) the system changes shapes in response to user inputs. With this workflow, a programmer can think in terms of high-level interface and interaction design to build interactive Swarm UI appli- cations, compared to existing, low-level, robot programming approaches.

The workflow of Swarm UI programming is inspired by the existing UI programming paradigm. We first review the common workflow of UI programming and decompose it into four basic elements that represent high-level steps. Then we discuss how to apply this workflow to Swarm UI programming. As we see in well-known design patterns for interactive UI ap- plications such as reactive programming paradigm, the Model-View-Controller, and the observer pattern, they share a com- mon workflow consisting of four basic elements: 1) create elements, 2) abstract attributes, 3) specify behaviors, and 4) propagate changes.

Implementation

Reactile actuates a swarm of small magnetic markers to move on a 2D canvas with electromagnetic force. We designed and fabricated a board of electromagnetic coil arrays (3,200 coils), which covers an 80 cm x 40 cm area. Reactile tracks the marker positions and detects interactions between a user and swarm markers using a standard RGB camera and computer vision techniques. The system displays spatial information using a DLP projector to allow a programmer to see program states in the same physical context.

In Reactile, a user interface consists of a swarm of passive magnetic markers which move on a 2D workspace driven by electromagnetic forces. Reactile uses a grid of electromagnetic coils to actuate these magnetic markers. Running current through the circuit coils generates a local magnetic field so that each coil can attract a single magnet located within its area. Each coil is aligned with a certain offset in both horizontal and vertical direction with an effective area overlap, which allows the coil to attract the magnet located in the adjacent coil. We design electromagnetic coil arrays to be fabricated with a standard printed circuit board (PCB) manufacturing. This reduces the cost and fabrication complexity, making it easy for the actuation area to scale up.

Our PCB design is a 4-layer board, and each layer contains a set of coils, each of which has an identical circular shape with a 15 mm diameter and a 2.5 mm overlap between nearby coils. Each coil has 15 turns with 0.203 mm (8 mils) spacing between lines, and the distance between centers of two coils is approximately 10 mm, which makes a 10 mm grid for attractive points. The final prototype covers an 80 cm x 40 cm area with 80 x 40 coils by aligning five identical boards horizontally. The fabrication of each board costs approximately $80 USD, including manufacturing of PCB and electronic components.

Publication

Slide