Dynamic Shape Construction and Transformation with Collective Elements

Ryo Suzuki  

PhD Dissertation

Links:     PDF   Poster   Slide   Video   30s Video   GitHub   ACM DL   IEEE   arXiv   Talk

Abstract

This thesis explores dynamic and collective shape construction as a new way to physicalize digital information for interactive physical displays --- i.e., shape-changing displays enabled by a swarm of collective elements. Through physical form of digital objects, the user can directly touch, grasp, and manipulate digital information through rich tangible and embodied interactions, but at the same time, such physical objects can dynamically change their shape for an interactive computer display and interface through collective shape construction and transformation with a swarm of elements. The goal of this thesis is to envision and illustrate how such an interface might support human activities by transforming physical forms at various sizes, from millimeter to meter scale.

To achieve this goal, this thesis introduces collective shape-changing interface, a new type of shape-changing interfaces constructed by discrete, collective, physical elements. This proposed approach promises to address the current limitations of shape-changing interfaces --- wherein a swarm of modular elements enables us to decompose the large, monolithic shape-changing objects into a set of simple, distributed elements. At the same time, their swarm behaviors enable us to make an unbounded shape transformation for expressive representation. This thesis contributes to the first exploration of this new class of shape-changing interfaces and proposes two approaches: active and passive shape construction. In active shape construction, collective elements can dynamically move and reconfigure themselves to construct a three-dimensional shape. Passive shape construction instead leverages external actuation to assemble and transform collective passive objects for dynamic shape creation. I explore and demonstrate how active and passive collective shape construction can be used as a future of computer interfaces, by developing various prototypes built on top of novel hardware and software platforms. Given these investigations, I discuss the design implications and possible research directions towards the future of collective shape-changing user interfaces.

Introduction

What if computer displays can represent information not only graphically but also physically? What if such physical forms of information could be as malleable and programmable as the pixels on a computer screen? If so, it could be used as a dynamic physical medium to interact with the digital world.

Ivan Sutherland, a founder of virtual and augmented reality, once envisioned that the future of computer displays would be "a room within which the computer can control the existence of matter" This radical vision has inspired many researchers over the decades, as such technologies could open up a new paradigm of human-machine interfaces.

However, we are still far from this exciting future. Today's computer interfaces mostly focus on screen-based interaction, where the screens serve as a looking window'' of the digital world --- the user can see digital information through the glass, but a barrier between what is inside (digital world) and what is outside (physical world) confines how we interact with the digital world. Current technologies do not allow us to directly touch, feel, grasp, and manipulate digital objects, in the same way that humans have done with physical objects for hundreds of thousands of years.

Thesis Statement

The goal of this thesis is to bring Sutherland's vision closer to reality by developing a new form of interactive and dynamic physical displays, and to illustrate how such an interface might support human activities by transforming physical forms and environments at various scales.

As a step toward this vision, this thesis explores dynamic and collective shape construction as a new way to physicalize digital information for interactive physical displays --- i.e., shape-changing displays enabled by a swarm of collective elements. Collective elements refer to discrete physical objects that can construct a physical, three-dimensional shape.

Each individual element can dynamically change its shape, position, and other physical properties through internal or external actuation, as to collectively construct and transform the overall physical shape. This enables a new way of representing digital information. Such physical shapes allow the user not only to see information, but to touch, feel, grasp, construct, and manipulate it, in the same way that interact with physical objects. At the same time, these physical objects must also embody dynamic computation. Collective elements can dynamically and programmatically reconfigure themselves, as if they are rendered in an interactive computer display and interface.

In contrast to shape changes made of monolithic materials, constructing shapes out of discrete elements enables rich expressiveness in representing information. Like pixels on a screen, they make shapes by collectively transforming the overall structure. Additionally, their components can be simple and interchangeable, thus allowing for scale. These elements can also interact with existing environments, and they make everyday objects and environments more dynamic, adaptive, and interactive by collectively actuating and reconfiguring them in a programmable fashion.

Making shapes out of discrete collective elements is not a new idea. There is a long history of modular self-reconfigurable robots and swarm robotics. These areas of research have explored the idea of collective and general-purpose shape transformation for robotic applications, such as space exploration, rescue, and navigation. However, there are many critical challenges when we apply this approach for interactive interfaces.

For example, the speed of transformation needs to be much faster than for robotic applications, as the interactive system must change and respond to the user in real-time (e.g., in seconds, not minutes or hours). Another consideration is scalability. To display meaningful information, it may require a relatively large number of elements, which often introduces implementation problems. Finally, unlike autonomous systems, interactive systems must consider the interaction between humans and objects --- there remains work to be done in understanding how we might interact with such collective elements and how these interfaces could support everyday human activities.

This thesis addresses these questions by investigating how collective shape construction and transformation can be used for interactive computer interfaces. To this end, this thesis introduces collective shape-changing interfaces, a new class of shape-changing interfaces constructed by a swarm of discrete physical elements. The main contribution of this thesis is the first exploration of this new class of shape-changing interfaces in the following four domains:

1. Shape representation:
   explore what types of shape representations are possible to display information

2. Reconfiguration methods:
   explore how both active and passive elements can be used to construct a shape for interactive interfaces

3. Interaction:
   explore how the user can interact with many collective elements through direct physical manipulation,

4. Applications:
   explore, illustrate, and demonstrate what kind of applications are achievable for human-computer interaction.

This new class of shape-changing interfaces promises to address some of the limitations of the current shape-changing interfaces. For example, a swarm of modular elements enables us to decompose the large, monolithic shape-changing objects into a set of simple, distributed elements. This significantly contributes to the deployability of the system in everyday environments. In addition, the swarm behaviors enable us to make unbounded shape transformations for expressive representation. Through my explorations of various proof-of-concept prototypes, I demonstrate how we can push the boundary of the current shape-changing interfaces by leveraging the collective behaviors of both active and passive elements. I also demonstrate how these dynamic shapes can support a range of application scenarios, such as interactive displays, adaptive environments, dynamic data physicalization, and accessibility support for people with visual impairments. Finally, I discuss the challenges and opportunities for using this approach towards the future of dynamic physical media.

Thesis Contributions

This thesis makes contributions to the field of Human-Computer Interaction in the following areas:

1. A design space exploration of dynamic shape construction with collective elements

2. A new taxonomy and investigation of active and passive shape construction and transformation with collective elements

3. A novel technique for creating a dynamic shape with active shape-transformable swarm robots (e.g., ShapeBots, LiftTiles)

4. A novel technique for constructing 3D shapes with an assembly of passive magnetically connectable blocks (e.g., Dynablock)

5. A novel technique for actuating passive magnetic markers with scalable electro-magnetic actuation (e.g., FluxMarker, Reactile)

6. A novel technique for actuating existing objects to reconfigure spatial layouts (e.g., RoomShift)

7. A novel interaction technique for programming the dynamic shape construction on a 2D surface with direct physical manipulation (e.g., Reactile)

Future Vision

PhD Dissertation Defense

Thesis Committee:

• Daniel Leithinger (CU Boulder, chair)

• Mark Gross (CU Boulder)

• Tom Yeh (CU Boulder)

• Hiroshi Ishii (MIT Media Lab)

• Takeo Igarashi (The University of Tokyo)

Defense Date: May 13th, 2020

Publication

   Download PDF

Slide