betaFeedback

WorksComputer Science

MaterialVellum: Steering Generative AI by Leveraging Expressive Material Affordances to Form Design Constraints

Abstract

We present MaterialVellum, an interaction approach for steering generative AI through material-inspired manipulation of design constraints. In urban design, for instance, zoning bounds can be defined by the stretch of elastic bands, building density redistributed by molding clay, and functional relationships organized by the pull of magnets. MaterialVellum departs from current interfaces that rely on natural language prompts, and instead leverages users' existing knowledge of physical materials to represent and manipulate design constraints. We introduce a framework that maps material affordances—such as elasticity, malleability, or attraction—to spatial, relational, and proportional constraints that underpin design. These mappings are operationalized in user interfaces through familiar direct manipulation multi-touch gestures, including stretching, sculpting, and clustering, whose effects feed as control parameters into generative AI pipelines. We demonstrate this approach through technology probes, including an urban design, a flower arrangement design, and a graphical design scenario.A qualitative study (N=10) shows that material-based interactions improve accessibility of constraint expression, support exploratory workflows, and lower barriers for non-experts, while effectiveness depends on alignment with users' mental models.

MaterialVellum framework mapping material affordances to spatial and relational constraints in design

Figure 1. MaterialVellum: (left) Framework mapping material affordances to spatial and relational constraints in design, (right) which is demonstrated in urban design and other use case scenarios.

CCS Concepts

• Human-centered computing → Interactive systems and tools; Interaction paradigms; • Computing methodologies → Artificial intelligence.

Keywords

generative AI, human-AI co-creation, material metaphors

ACM Reference Format:

Chu Li, Nicolai Marquardt, Hugo Romat, Ken Hinckley, Nathalie Riche, Kwon Ko, and Fanny Chevalier. 2026. MaterialVellum: Steering Generative AI by Leveraging Expressive Material Affordances to Form Design Constraints. In The 39th Annual ACM Symposium on User Interface Software and Technology (UIST ’26), November 02–05, 2026, Detroit, MI, USA. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3830398.3830504

This work is licensed under a Creative Commons Attribution 4.0 International License. UIST ’26, Detroit, MI, USA

© 2026 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2856-3/2026/11

https://doi.org/10.1145/3830398.3830504

Introduction

Design constraints are central to the creative process. They define the creators’ arena by specifying both the boundaries of the solution space and identifying the open spaces where artistic interpretation can flourish [43]. Such constraints—hard or soft—take many forms: spatial grids that organize web interfaces, height regulations that shape urban skylines, and proportional rules that define fashion design. Creative professionals then shape solutions within this bounded space and iterate over solutions, balancing tradeoffs against given constraints.

When creators co-design with generative AI (genAI), design constraints further function as an expressive way for communicating the guardrails for steering a model’s creative expression. Still, while natural language is effective for many expressions of intent, the articulation of constraints—from finding the right terminology to express, to ways of setting their scope—can be tedious [59]. This is especially true of non-textual media where sketching (e.g., a grid line), modeling (e.g., a building’s shape), fitting (e.g., pieces of fabric), might come as more direct and natural means of expression. In this work, we explore the potential of leveraging physical material affordances to shape design constraints within co-design workflows with genAI, as an alternative to the classic textual prompts.

Reasoning about constraints (e.g., flexible-rigid, soft-hard) aligns conceptually with reasoning about material behavior (e.g., elastic-solid, porous-dense). We exploit this alignment by proposing a conceptual mapping that links material qualities—such as elasticity, density, attraction—to the direct expression of constraints. For instance, consider an urban designer planning for 20 to 25 acres of green space. This constraint can be conceptualized as a rubber band: its ideal tension represents the target area, while slackness or snapping indicates a failure to meet that value. Here, the physical properties of the rubber band embody the constraint’s tolerance zone. We conceptualize constraints as interactions between resources (the elements being constrained) and a substrate (the space within which constraints operate). Resources are categorized by their amount, malleability, and divisibility, whereas substrates are defined by their dimensionality and their active or passive nature. These mappings are operationalized in user interfaces through familiar direct-manipulation multi-touch gestures, including stretching, sculpting, and clustering, whose effects translate into control parameters for genAI pipelines.

Our approach builds on prior work in direct manipulation for AI systems [39], extending instrumental interaction frameworks to generative models [50], treating text as material for co-generation [54], and developing specialized ideation canvases that move beyond linear chat-based interactions [38]. We extend these efforts by grounding AI constraint expression in physical material qualities, aligning with a growing body of HCI research (e.g., [20, 51, 64]) that recognizes materiality as a powerful lens for understanding design constraints and possibilities. A key advantage of this material framing is that materials naturally prompt intuitive gestures, and research demonstrates that gesture-based interactions enable users to encounter, interpret, and sustain abstract meanings through direct, embodied engagement [41, 54, 63].

To demonstrate the feasibility and potential of our approach, we developed MaterialVellum, a fictional direct-manipulation interactive system where users manipulate digital material representations using familiar gestures: pulling and stretching elastic bands, pinching and modeling clay, manipulating attracting and repelling magnet pairs, and piling up and shaping sand. These interactions then serve as inputs for genAI pipelines (using Stable Diffusion (SD) and ControlNet modules), generating responsive and controllable design outputs. We illustrate MaterialVellum through technology probes spanning urban design, flower arrangement, and graphic design scenarios. Using these probes, we conducted a qualitative study with ten participants of varying design experience to gain insights into both the value and limitations of material metaphor manipulation for AI-assisted design. The study reveals how our approach addresses key challenges in current genAI interfaces. For designers, our approach offers an expressive method that integrates constraints directly into the design process, while for non-designers, the physicality of materials makes design constraints more accessible and inviting.

In summary, our contributions include:

  1. an approach mapping materials to design constraints that enables manipulation through familiar material interactions (Section 3);

  2. a demonstration of this approach through three technology probes, with design implications for interactive systems that support material metaphor manipulation of design constraints (Section 4);

  3. findings from a preliminary user study showing that the effectiveness of material metaphors depends on whether digital representations and gestural interactions match users’ expectations and mental models (Section 5).

Related Work

MaterialVellum sits at the intersection of design constraints, material affordances, and genAI interaction. We review each of these areas in turn, situating our contribution within the broader landscape of embodied approaches to human-AI co-creation.

The Role of Design Constraints

Design constraints are the boundaries (spatial, structural, relational, or proportional) that define the solution space within which creative work takes shape [43, 57]. Constraints play a dual role in creativity, acting as limitations and enablers of creative thought [3, 32, 58].

Biskjaer et al. [4] formalize a constraint-based understanding of design spaces, offering a schema for how designers navigate and manipulate constraints to shape what is possible. Biskjaer and Halskov [5] show that highly limiting creative decisions can accelerate the design process and propose decisive constraints as an analytical concept for understanding creative turning points.

Externalizing and communicating constraints, however, remains a persistent challenge. Chevalier and Ivory [10] show that the ability to articulate and respect constraints varies with expertise and whether constraints are explicitly prescribed, with both novice and professional designers needing better support. Within HCI, tools like Scout [61] address this by making constraints explicit and manipulable, enabling rapid exploration of design alternatives through high-level constraint specification. Magnusson [36] argues that designing constraints is itself a compositional act: defining a system of constraints encapsulates a space for potential expression, establishing the identity and expressive scope of the instrument. This framing extends naturally to genAI, where the user is similarly composing constraints for the system to follow. Yet current genAI interfaces offer limited means for doing so, requiring users to translate rich, multidimensional constraints into linear text prompts [59]. This gap between the centrality of constraints in design practice and the lack of support for communicating them to AI systems motivates MaterialVellum’s approach of encoding design constraints through material representations and gestural interaction.

Figure 2. Parallel dimensions of constraint qualities (left) and material qualities (right) reveal the cognitive alignment between how constraints and materials are described.

Material Affordances & Gesture Interactions

HCI scholars increasingly recognize materiality as a lens for understanding the possibilities and constraints of interactive design [20, 51, 64]. Giaccardi and Karana [20] proposed a materials experiences framework, and Karana et al. [26] operationalized this by quantifying materials not only for what they are, but for what they do, express, and make us do. Dourish [15] provides theoretical grounding through embodied interaction, the idea that tangible computing draws not on everyday artifacts as metaphorical decoration, but on how we experience and act within our environments.

Several systems demonstrate materials as creative digital media, including SandCanvas [27] for multi-touch sand animation and SandTouch [33] for VR sand art, both showing that material-inspired interactions support rapid, expressive creative work.

Materials are particularly relevant to interaction design because they naturally prompt specific gestures. Dövencioglu et al. [17] show that exploratory hand movements vary systematically across materials (i.e., from silk to sand), driven by how materials deform under force, confirming that people possess ingrained gestural knowledge tied to material behaviors. Gibson’s theory of affordances [21] provided the ecological basis for this observation: the action possibilities of an object are perceived directly through its physical properties. Sun et al. [60] formalize how such embodied gestures can carry abstract meanings through a framework for mapping metaphorical meanings to gesture-based interactions. Renom et al. [48] further demonstrate that the technical reasoning people use for physical tool use transfers to digital environments, suggesting

that material metaphors allow users to leverage existing physical reasoning rather than learning new interaction vocabularies.

Recent work explores the pairing of materials and gestures as an interaction paradigm for genAI. Shen et al. [54] reimagine text as a material that can be sculpted, grown, and transformed through embodied gestures for LLM-mediated writing. This work showed that material metaphors shape users’ mental models, foster non-linear workflows, and invite creative exploration beyond text prompting. We extend this material-metaphor approach to visual designs, mapping everyday materials to design constraints and connecting gestural manipulation to image generation pipelines.

Interacting with GenAI beyond Text

Text-to-image prompting has become the dominant paradigm for steering generative models, but its limitations are well documented (e.g., [59, 65]). Users struggle to articulate visual intent [6, 34, 55], iterate through trial-and-error [35], and often lack understanding of the models’ capabilities [37, 59]. The challenge is particularly acute for design constraints; their multidimensional, interconnected nature makes them difficult to describe and manipulate with text.

A growing body of work addresses these limitations through alternative interaction paradigms. One approach operationalizes prompt-based interaction as general-purpose tools: Riche et al. [50] introduce AI-Instruments, embodying prompts as graphical interface objects through reification, reflection, and grounding. In a complementary direction, Masson et al. [39] map prompts to direct manipulation actions, and Shi et al. [55] enable design intent specification through direct manipulation on design tokens. Other approaches leverage spatial and trajectory-based controls: TaleBrush [12] and PromptPaint [11] let users manipulate generation trajectories, while DesignPrompt [46] and WorldSmith [13] use spatial layout as a steering modality. Marquardt et al. [38] introduced ImaginationVellum, a freeform ideation canvas where the canvas itself serves as the prompt, modulating AI generation through spatial proximity, generative strokes, and intent tags. Together, these systems extend the interaction vocabulary available for steering genAI through spatial arrangement [38], instrumental interaction [50], and design tokens [55].

In fact, direct manipulation has offered non-textual control of design intent long before generative models. Constraint-based drawing maintains relation limits as the user drags [22]; soft selection spreads a local edit outwards with decreasing influence [24]; force fields pull objects together or push them apart [1]; and particle canvases support granular layout {citep}[27, 33]. Sketch- and mask-based conditioning has since carried this tradition into diffusion models, including the ControlNet pipelines we build on {citep}[66]`.

Where most direct manipulation approaches define or extend a set of interaction mechanisms, MaterialVellum instead leverages the embodied knowledge people already possess about everyday materials (elastic bands, clay, magnets, sand) and the gestures these materials naturally invite. While prior approaches offer tool-mediated, more precise and granular control, material-elicited gesture interactions are inherently approximate and whole-handed [17, 27, 33]. We argue this quality is not incidental but productive: it lowers the barrier to entry [2], supports exploratory engagement [25], and affords open-ended creative expression aligned with how people naturally approach early-stage design [15].

Mapping Materials to Design Constraints

Design constraints are the boundaries—spatial, structural, relational, or proportional—that define the solution space within which creative work takes shape. Surveying how constraints are discussed across design and cognitive psychology, we observed that researchers characterize them using qualitative language that mirrors how people describe physical materials. Constraints are spoken of as flexible or rigid [19], soft or hard [57], ambiguous or precise [47]. We identify six such dimensions—flexibility [19], enforcement [57], clarity [47], temporality [53], interaction [47, 57], and abundance [40]—along which both constraints and materials can be described (Figure 2). This overlap suggests a natural cognitive alignment: a text block that must stay integral shares the same quality language as a hard, rigid material; while decorative graphic elements that can be cropped, stretched, or rearranged to fill empty spaces evoke the fluidity of sand.

This cognitive alignment motivates the use of materials as metaphors for constraints. However, these qualitative descriptions alone are not readily usable for systematically selecting an appropriate material mapping. Describing a constraint as “flexible” does not specify how it is flexible, nor does it capture the spatial or interactive context in which the constraint operates. Inspired by prior work on constraint typologies [5, 57, 58], we propose a complementary lens that conceptualizes constraints as the interaction between resources—the elements being constrained—and substrates—the space within which constraints operate. We characterize resources along dimensions such as amount, divisibility, and malleability, and substrates along dimensions such as dimensionality and engagement (Figure 3). These dimensions are not intended to be exhaustive; rather they offer a practical vocabulary and an approach for reasoning about the structural properties of constraints, which in turn can inform the selection of materials whose physical behaviors best embody them. Similarly, the values within each dimension admit further granularity: flexible, for instance, can be further decomposed into whether the constraint enforces an upper bound, a lower bound, or permits variation within a tolerance range (Figure 3). Our goal is not a definitive taxonomy, but a descriptive vocabulary that makes the structural properties of constraints legible and informs material selection.

Figure 4 illustrates six representative pairings produced by applying this mapping approach. For four of these examples, we show visual representations and a gesture vocabulary: rubber bands for flexible zoning boundaries, sand for fluid graphic design hierarchy, magnets for building-use compatibility, and clay for building density distribution. Together, these examples show how differences across constraints translate into materially distinct interactions. These examples illustrate how the material-inspired lens generates principled mappings across different constraint types and use cases, and how each material naturally elicits gestures that enact the corresponding constraint behavior.

Figure 4. Representative material-to-constraint mappings derived by our framework, showing resource and substrate properties, constraint and material examples, visual representations, and associated gestures.

MaterialVellum: Applying Material Metaphors to Design Constraints

To explore the conceptual framework presented in Section 3, we developed MaterialVellum as a proof of concept that applies material metaphors to design constraints through a gestural multi-touch interface. Our design leverages the inherent affordances of familiar materials—elastic bands, clay, magnets and sand—to elicit natural gestural responses rooted in users’ embodied experience. Users stretch, mold, push, and scatter digital representations in ways that mirror how they would physically handle these materials, while the system translates these manipulations into meaningful design constraints for a genAI pipeline. We present urban design as our primary demonstration scenario, followed by two additional application domains that illustrate the generalizability of the approach.

Urban Design: Constraints as Material

Urban design involves the planning and organization of physical spaces within cities, balancing multiple competing constraints to create functional, livable environments. We selected urban design as our primary demonstration scenario because it involves several distinct yet interacting constraint types—such as zoning boundaries, building density, and functional compatibility—that lend themselves well to material metaphor mappings.

Rather than developing a professional urban design tool to replace existing systems, this scenario serves as a technology probe to explore how material affordances can make design constraints more accessible and manipulable through embodied interaction.

Urban Design Elements and Constraints

The MaterialVellum urban design probe focuses on three design elements that exemplify different constraint types commonly found across design domains: flexible boundaries (zoning), resource allocation constraints (building density), and relational constraints (functions).

Zoning defines permissible land use within designated areas, sometimes expressed as flexible ranges rather than rigid boundaries (e.g., 20–30% residential allocation for a new development). Modern zoning regulations can also be modified over time as development needs change [18]. Such constraints exhibit elasticity: they accommodate variation within bounds but fail when limits are exceeded.

Building density represents the distribution of built volumes, or how concentrated development is within an area [49]. While total volume remains fixed, designers can redistribute density—concentrating height in some zones while keeping lower profiles elsewhere—to create varied urban experiences.

Functions dictate the spatial relationships between building types and uses. Some are naturally compatible (e.g., shopping and dining) while others require separation (e.g., industrial and residential), creating attraction and repulsion constraints that shape urban layout.

Material Metaphor Mapping

We map design constraints to material behaviors that exhibit analogous properties:

Elastic Bands for Zoning. Elastic bands embody flexibility within limits, maintaining form when appropriately tensioned but snapping when overstretched. This behavior mirrors zoning constraints, which accommodate variation within prescribed bounds but fail beyond acceptable thresholds. In terms of our constraint anatomy (Figure 3), the zoning area is the resource—flexible in amount and indivisible—while the land parcel serves as the 2D passive substrate.

Figure 3. Constraint Anatomy. We conceptualize constraints as the interaction between resources and substrates. Resources are characterized by amount, malleability, and divisibility; substrates by dimensionality and engagement.

Clay for Building Density. Clay maintains constant mass during shape transformation, embodying urban design’s constraint of fixed total building volume that designers can redistribute. When interacting with clay, users can redistribute the same material volume to create varying density patterns; concentrating clay in one area naturally reduces it elsewhere, reflecting real-world trade-offs.

Magnets for Functions. Magnetic attraction and repulsion provide intuitive representation of functional compatibility constraints. Compatible urban functions behave like attracting magnet pairs, naturally clustering together, while incompatible ones act like repelling pairs, maintaining appropriate separation distances. Mapping this to our constraint anatomy (Figure 3), the functions are the resources, and the urban layout acts as a 2D active substrate, exerting compatible forces that govern where resources can be placed.

Interface & Interaction

The urban design probe interface consists of five main parts (Figure 5A): a top toolbar with three design element tabs (Zoning, Building, Function), a submenu displaying element-specific interaction options, a central interaction canvas, an SD image preview on the top left, and utility controls on the top right. Each design element tab transforms the interface to support corresponding material interactions.

Zoning as elastic bands

Figure 5. Zoning as elastic bands. (A) Users create zone boundaries via four-finger touch; real-time area percentages are displayed. (B) Bands snap back when stretched beyond allowable limits, providing visual constraint feedback. (C) Different zoning configurations produce varied urban layouts, with buildings auto-populated according to density level.

Zoning as Elastic Bands. For Zoning elements, users create elastic band loops representing zones through four-finger touch gestures on the canvas (Figure 5A). As users move their fingers, the elastic loop updates dynamically, displaying real-time calculations to indicate zoning coverage. When users lift their fingers, the system converts the loop into a defined zone boundary. The elastic bands enforce zoning constraints through material behavior: when while others require separation (e.g., industrial and residential), creating attraction and repulsion constraints that shape urban layout. a zone exceeds maximum allowable area, the band snaps back to the permitted threshold, providing visual feedback about the design constraint limits (Figure 5B). Created zones can be edited via long press or deleted via double-tap (Figure 5C). These user-defined zones are automatically populated with pre-generated building typologies based on established urban design practices [30]. Each density level has contextually appropriate architectural forms: low-density zones include single-family houses and low-rise apartments, mid-density neighborhoods feature mixed-use buildings and townhouses, and high-density regions resemble urban downtowns.

Buildings as Clay. For Building elements, the canvas transforms into a grid of dots representing malleable clay, where dot density corresponds to the zoning constraints established in the previous tab (Figure 6A). Low-density zones display sparse dot patterns while high-density zones show tightly packed grids, visually encoding developable volume across areas. Users manipulate building density through familiar pinching gestures: pinching inward increases building density while pinching outward reduces it (Figure 6A & B), with changes immediately reflected in the 3D urban model preview (bottom right). As users interact with the clay, the system sends the updated 3D model to SD, generating photorealistic renderings in the preview panel.

Buildings as clay, showing pinching gestures on a dot grid and corresponding 3D building previews

Figure 6. Buildings as clay. (A) Pinching inward on a dot grid increases building density, reflected in the 3D preview. (B) Pinching outward lowers buildings, redistributing density while preserving total volume. The 3D model serves as input to SD.

The interface supports two viewing modes: side-by-side (default) with the generated image in the top left corner, and overlay mode where the generated image appears directly on the canvas. A transparency slider allows users to adjust the opacity of the generated image, enabling examination of relationships between height manipulations and the resulting imagery. The dot grid employs a sunset color gradient indicating building density. This creates a heat map effect when overlaid with SD-generated images, helping users understand the correlation between their manipulations and the resulting urban forms.

The interaction technique focuses on manipulating clusters of buildings rather than individual buildings, reflecting urban design’s emphasis on neighborhood-scale density patterns. The clay metaphor encourages manipulation of the overall building massing distribution across urban blocks, while the material properties implicitly encode the necessary constraints.

Urban Functions as Magnets. Urban Functions are represented as colored magnets corresponding to four building types: amusement park, shopping mall, wastewater treatment plant, and a textile factory (Figure 7A). The magnetic behaviors are based on functional compatibility—compatible programs (e.g., parks and malls) attract when moved near each other, while incompatible pairs (e.g., parks and factories) actively repel (Figure 7B & C).

Functions as magnets, showing compatible and incompatible urban functions and selected blocks

Figure 7. Functions as magnets. (A) Colored circles represent urban functions with attraction/repulsion based on compatibility. Compatible functions attract when placed near each other. (B) Incompatible programs repel when dragged near each other. (C–E) Selected blocks are inpainted with function-specific imagery to produce the final result.

As users drag magnets across the canvas, the prototype identifies the nearest urban block and updates its functionality visually. Target blocks pulse to provide visual feedback during updates, ensuring users understand which areas will be affected. Users can toggle overlay mode to view the generated imagery at full canvas size, with an optional visual aid that highlights urban blocks with corresponding colored boxes, providing precise feedback about which block each magnet will affect.

The four programs are a scoping choice for the probe; future users can easily extend to others such as restaurants, schools, or concert halls by specifying compatibility relationships.

Extending to Other Applications

We illustrate MaterialVellum’s generalizability with two additional scenarios.

Using Magnets for Ikebana

Ikebana is the Japanese art of flower arrangement, a practice that treats negative space as equally significant to the floral components, emphasizing that the voids between elements carry as much compositional weight as the materials [28]. The art form focuses on achieving equilibrium and harmony through careful relationships between organic forms [28]. Considering these spatial and relational dynamics of ikebana, the magnetic metaphors are one possible material-to-constraint mapping, where attraction represents compatible element relationships and repulsion enforces appropriate spacing. MaterialVellum’s magnetic approach allows users to explore ikebana’s complex positional rules through intuitive push-pull interactions that mirror the natural tensions and harmonies central to this art form.

Design Constraints. The three structural elements are known as Shin (heaven), Soe (humanity), and Hikae (earth), each governed by distinct constraints (Figure 9A). Shin is positioned 1010–15∘15^{\circ} from vertical, Soe at 45∘45^{\circ}, and Hikae at 75∘75^{\circ} [52]. Proportionally, Shin’s length should equal three times the combined width and height of the Utsuwa (container), Soe should be 2/32/3–3/43/4 of Shin’s length, and Hikae should be 1/21/2–2/32/3 of Soe’s length [52].

Ikebana as magnets

Figure 9. Ikebana as magnets. (A) Traditional angular and proportional rules for Shin, Soe, and Hikae relative to the Utsuwa. (B) The interface encodes these rules in a magnetic system that restricts placement to valid positions. (C) Resulting generated arrangements.

Interface & Interaction. The ikebana interface reuses the magnet material, with three magnetic circles for structural elements and a rectangle for the container (Figure 9A). The magnetic system restricts placement to valid positions, enforcing traditional proportional and angular requirements.

Users can add supplementary elements like leaves and flowers, which exhibit attraction and repulsion behaviors based on composition principles—compatible elements attract while similar types

repel to encourage visual variety (Figure 9B). As the user interacts with the magnetic circles, the arrangement is sent to SD using scribble ControlNet with prompts specific to ikebana aesthetics (Figure 9C).

Using Sand for Graphic Design

Sand has a long history as an artistic medium, from traditional sand drawing practices to contemporary sand animation performances [27, 62]. The material has also been appropriated for digital creative applications including SandCanvas [27] and SandTouch [33]. Sand offers an expressive vocabulary of physical gestures that engage the whole hand rather than discrete strokes, creating interactions that are intuitive to learn, quick to execute, and easy to modify—qualities that make sand particularly well-suited for design exploration and iteration [27].

Effective graphic design demands careful coordination of focal points, visual hierarchy, and spatial placement, yet communicating these visual relationships through text presents significant challenges, especially when design elements are closely integrated [55]. MaterialVellum’s sand metaphor addresses this challenge by allowing designers to directly manipulate spatial layouts through these natural, expressive interactions that support rapid experimentation with compositional arrangements.

Design Constraints. Graphic design involves spatial and hierarchical constraints that govern element placement and relationships, including visual hierarchy, adequate spacing, and balanced composition weight. These constraints require simultaneously considering interdependent factors—font sizes must relate to image scales, spacing must accommodate content density, and position must support visual flow while maintaining readability. Separating spatial layout decisions from content generation allows designers to focus on overall composition before AI systems populate specific elements, enabling rapid exploration of layout variations while maintaining control over spatial relationships. Following our constraint anatomy (Figure 3), the graphic design elements are the resources: flexible in amount, divisible without constraints, and fluid in malleability, while the canvas itself serves as the 2D passive substrate.

Interface & Interaction. Users begin with a square canvas and can emit sand clusters using multi-touch gestures in the default “add” mode (Figure 8A). Switching to “move” mode allows reshaping and repositioning (Figure 8B). Once positioned, users assign semantic roles through voice input—title, main image, logo, description—with each cluster changing color to reflect its function (Figure 8C). After all clusters have been tagged, users can trigger AI generation to produce the complete graphic design (Figure 8D).

Sand for graphic design

Figure 8. Sand for graphic design. (A) Multi-touch gestures emit sand clusters in “add” mode. (B) “Move” mode enables reshaping and repositioning. (C) Voice input assigns semantic roles (title, image, logo) with color coding. (D) AI generates the complete poster. (E–F) Cluster size and density control element scale and font weight/opacity.

Sand cluster properties directly control generated content through size and density. Cluster density determines font weight for text elements and opacity for images, while cluster size controls the scale for both text and images (Figure 8E & F). Text content is generated via GPT-4 based on design themes and semantic tags (e.g., the title of a concert poster), while images are created via SD with transparency support (e.g., the main image of a concert poster).

Implementation and Pipelines

MaterialVellum is implemented as a web-based application with a React.js [42] frontend and Flask [45] backend. The frontend uses Matter.js [7] for physics-based material interactions, GSAP [16] for animations, and Three.js [8] for 3D rendering. Image generation runs on a locally hosted SD server (NVIDIA RTX 4090) using multiple ControlNet models (Depth and Scribble) [66], with model selection depending on the active material metaphor. Text generation uses OpenAI’s GPT-4 API [44].

We designed technical pipelines for each of the use case applications to leverage multi-touch interactions for manipulation of material representations, which then in turn feed as control parameters into genAI generation (Table 1). For example, the pipeline for the urban design use case operates through three sequential stages corresponding to each material interaction. Zoning Stage: User-created elastic band inputs are converted to template-based urban blocks and buildings. This 2D urban design plan serves as the foundation for generating a 3D urban model, where building heights are determined by established typologies [30]. Building Stage: When users manipulate clay to adjust building density, the system captures the updated 3D model and sends it to SD. The pipeline first generates a depth map from the image capture of the 3D model; then uses this depth map as ControlNet input to produce photorealistic renderings of the modified perspective. Urban Function Stage: The generated building imagery serves as the base layer for function-based inpainting. When users position function magnets, the system calculates bounding boxes for the closest urban blocks, accounting for block dimensions, building density, and camera perspective to ensure accurate coverage of both block dimensions and extruded buildings. These bounding boxes then form masks that drive targeted inpainting with function-specific prompts (e.g., “Drone photography of water amusement park, top down view”). Inpainting results are applied as overlay layers rather than direct edits, allowing blocks to automatically revert to their original appearance when magnets are moved to new locations.

MaterialConstraintInteraction ParameterEnforcement Mechanism
ClayBuilding density in urban designBuilding densityDepth map from 3D model → ControlNet (Depth) → SD
MagnetsFunctions in urban designMagnet positionNearest block’s bounding box → inpainting mask → SD inpainting
MagnetsAngular and proportional rules in ikebanaMagnet positionElement positions + connecting lines → ControlNet (Scribble) → SD
SandVisual hierarchy in graphic designCluster densityCluster density → image alpha via LayerDiffuse → SD
ElasticZoning in urban designBand enclosed areaBand input → template-based urban blocks
SandElement scale in graphic designCluster sizeCluster size → post-processing scales generated element

Table 1. Constraints implemented in MaterialVellum, grouped by the layer in which each is enforced.

Qualitative User Study

We conducted a qualitative study with ten participants to gather preliminary feedback from both novices and professional designers about the concepts and techniques within MaterialVellum.

Study Design

Procedure. Sessions took place with participants seated at a Microsoft Surface Studio (28-inch 4.5K display) configured in drafting table mode. At the beginning of each session, participants signed consent forms agreeing to audio and video recording and completed a brief pre-study questionnaire on demographics and design experience. Each session comprised three parts: Introduction. The experimenter began by asking participants about their prior experience with image generation tools, then introduced the agenda, and provided a brief overview of design constraints and MaterialVellum’s material metaphor approach. Design Activities. Participants completed two guided design activities using the urban design (Section 4.1) and the graphic design probes (Section 4.2.2). For urban design, participants were asked to imagine themselves in a participatory urban design session, and were required to create three new neighborhood designs: (a) a vibrant downtown with skyscrapers, (b) an elderly-friendly neighborhood, and (c) a neighborhood they would personally want to live in. For graphic design, participants were asked to create a concert poster using sand. At the beginning of each activity, the experimenter provided instructions on interaction techniques and UI controls. The experimenter clarified that there was no evaluation of design output quality, emphasizing that the researcher’s goal was to gather genuine reactions and honest feedback about their experiences. After completing each design activity, the participants were asked to provide immediate reactions to each probe. Feedback. After completing both design activities, participants filled out a post-study questionnaire with Likert-scale questions assessing their experiences across multiple dimensions. For each material probe (elastic, clay, magnets, sand), participants rated five dimensions: naturalness of the constraint-to-material mapping, naturalness of the metaphor-prompted gestures, ease of manipulation, whether the system generated what they intended, and their confidence in having adhered to the specified constraints (Figure 10).

Figure 10. Likert-scale responses (N=10) for each material metaphor across five dimensions, shown top to bottom: constraint adherence, controllability, ease of manipulation, material-to-gesture mapping, and constraint-to-material mapping.

The questionnaire also incorporated questions from the Creativity Support Index (CSI) to evaluate MaterialVellum’s overall support for creative activities (Figure 11). Sessions concluded with the experimenter gathering participants’ impressions of the most salient strengths and weaknesses of the experience, perceived value and use case of the approach, suggestions for other design scenarios and materials, and desired improvements.

Self-reported Creativity Support Index

Figure 11. Self-reported Creativity Support Index.

Participants

Following institutional review board approval, we recruited ten participants (four men, six women) via organizational mailing lists. Participants spanned a range of design experience, though the sample skews toward novices (Appendix Table 3). Study sessions were conducted in-person, lasting approximately one hour, with participants receiving $50 USD gift cards as compensation.

Data Collection and Analysis

Each session was video recorded with audio transcription and screen capture. We collected all generated images via MaterialVellum’s built-in logging. We also collected quantitative data via post-study questionnaires.

Our qualitative analysis focused on identifying high-level themes from the collected data. One researcher conducted qualitative open coding [9] based on video transcripts and questionnaire responses to develop an initial set of themes, then systematically coded all responses according to these themes. To ensure clarity and protect participant privacy, quotes have been slightly modified for concision, grammar, and anonymity.

Findings

Our qualitative analysis, supported by Likert-scale data, revealed six key insights about how participants experienced MaterialVellum’s approach of using material metaphors for forming design constraints to steer generative AI.

T1 Embodied material knowledge supports natural gestures. Material metaphors effectively prompted natural gestural interactions that participants found intuitive to learn and remember. Likert-scale responses on whether the gestures prompted by material metaphors felt natural showed strong agreement for magnets and elastic, with more mixed responses for sand and clay (Figure 10). After the tutorial, only one participant needed interaction gestures re-explained, suggesting the material affordances effectively communicated their associated gestures. P6 explained: “We’re so familiar with sand and clay from childhood […] To a point where I feel like [the approach] is quite intuitive.” The naturalness of the gestures meant that participants could focus on their design goals rather than learning interface mechanics. P5 commented: “For brainstorming, this [gestural interaction] feels much more natural than text prompting.”

Not all material mappings felt equally natural. P6 identified a disconnect between the pinch gesture and their embodied understanding of sand and clay: “In the real world, you don’t pinch clay or sand. You use your whole hand to morph it sideways. [Sand and clay] should be more of a whole-hand interaction.”

T2 Material metaphor effectiveness varies across participants and contexts. Participants demonstrated distinct preferences across the four material metaphors, with both Likert-scale responses (Figure 10) and qualitative feedback revealing differences in how each metaphor was experienced. While participants like P1 appreciated the overall approach, “I like the metaphor a lot, it’s a proper way to illustrate the relationship between physical engagement and design,” individual mappings resonated differently with different users.

Magnets received the most uniformly positive ratings (Figure 10). P9 explained: “My favorite feature was the magnets. I really like the concept of different city areas going well together or not […] it’s obvious and effective.” P6 added: “I enjoyed the magnet interaction most because I could clearly map my changes to image changes.” P2 found them engaging for creating realistic environments: “The magnets are the most fun because it feels like designing a real world environment.”

Elastic was rated positively on most dimensions, with participants appreciating the conceptual mapping but expressing some frustration with implementation limitations. P2 captured this duality: “I thought the rubber band was really fun […] but frustrating because it mapped to a template rather than my exact drawings.”

Clay and Sand showed more polarized responses. P8 said: “Clay was what I enjoyed most because the translation from gesture to results has the clearest feedback.” While P9 felt “the clay visualization had no correlation to actual clay.” For Sand, P10: “I like the sand concept, I’ve never thought of using sand this way, but it reminds me of how kids draw in sand, which feels intuitive.” P8: “It’s satisfying to put sand onto the canvas. It gives me control and lets me insert my opinion.” P7 said “I like this abstract approach to poster design because it helps me think about visual hierarchy and prominence.” However, P7 also

mentioned sand’s limits: “Sand is too abstract for controlling text, which has many visual parameters: boldness, font size, line spacing.”

P4 articulated the underlying tension between clarity and metaphorical richness “This is a paradoxical design challenge: if you make everything crystal clear with one-to-one mapping, you lose the metaphor. If it’s too vague, people struggle with the mental model. The beauty lies somewhere in between [. . .] you give people some guidance while leaving room for discovery.”

T3 Controllability for generative AI differs across design contexts. Participants’ satisfaction with AI generation outputs varied considerably across materials and design contexts (Figure 10), highlighting tensions between constraint-based design and creative flexibility. Some participants found constraints helpful for making decisions without domain expertise. P6: “Urban design is something I have no knowledge of, but magnets helped me be less aware of my lack of knowledge yet still design correctly.” P6 also valued the balance: “This system hits a nice middle ground where I can specify that I want a particular area to be a park, but leave everything else up to the AI’s imagination.” However, others wanted more flexibility. P2 noted: “I tried to draw specific designs but got templates that were only somewhat similar.” P5 echoed this desire for greater precision: “The ability to interact directly is really nice, though I find myself wanting finer control.”

T4 Novice-friendly approach that supports early-stage ideation and educational purposes. MaterialVellum proved particularly effective for novice users and educational contexts. P7: “The tool has educational value for learning design constraints through interaction. For example, I learned that parks and shopping malls complement each other.” The staged design process was especially beneficial for building novice confidence. P7: “I appreciate the three-step guided process and how each step builds on the previous one. This naturally leads me to consider how density relates to building heights and their corresponding functions.” P7 further noted how the constraints reduced cognitive load: “The rubber band gives me confidence that I could focus on my design goals while the system handles the constraints.”

Participants repeatedly identified educational applications, particularly for children and community engagement. P4: “This is perfect for community engagement with non-technical participants who have ideas but lack technical skills. They can experiment without needing precision.” P7 envisioned specific educational contexts: “This tool is ideal for early-stage conceptualization—understanding the general vibe and what the design looks like from a distance. It’s particularly good for non-professionals and novices who don’t need to think about constraints intentionally. I could see this being used educationally, even with kids in museums.”

T5 Playful engagement through embodied interaction. The hands-on nature of the interface resulted in high engagement levels. All ten participants strongly agreed with the statement “I was very engaged/absorbed in the activity—I enjoyed it and would do it again”. P5: “Being able to use your hands to make things is innately satisfying [. . .] I want to keep playing with it.” P6: “It’s a bit more physically demanding than using a mouse [. . .] but the embodied nature and the fact that you’re having a ton of fun makes the extra effort worth it.” P3 appreciated the sensory dimension: “It provides visual sensory stimulation that’s soothing.” P5, a former landscape architect, contrasted MaterialVellum with professional tools: “I really like this kind of physical, embodied interaction. It’s much more fun than using AutoCAD.”

T6 Example applications and potential beyond demonstrated scenarios. Participants readily identified application domains beyond urban design and graphic design (Table 2). The breadth and specificity of these suggestions indicate that the material framing transferred conceptually: participants could describe constraints in other domains in material terms and imagine gestures for manipulating them. As P6 expressed: “This notion of embodied interactions could work for just about any genAI scenario, because how I use genAI today is still mostly text-based, and it’s very limiting. Creativity is always bigger than whatever text can describe.”

Application DomainSuggested MaterialsParticipant Examples
Interior DesignElastic band for layout“When people are moving and need to rearrange their furniture.” (P6)
Fashion DesignClay, elastic for garment flow“Specify the flow of fabric for shirts or pants.” (P2)
Graphic DesignMagnets for color palettes“It would be great if [the magnets] can tell me I’m designing something wrong, like putting colors together that are colorblind-unfriendly or culturally insensitive.” (P6)
Game DesignMagnets for character and maps“Character optimization [. . .] making fantasy maps for tabletop games.” (P5)
Scientific VisualizationSand for layout“Figure design for research, faster than Figma for iterations.” (P8)
NutritionMagnets for nutrient interactions“Iron and calcium would repel since they affect absorption.” (P5)
SchedulingMagnets for time conflicts“Determine what things go together or conflict with each other.” (P5)
Traffic SeparationMagnets for pedestrian safety“You can also represent how close pedestrians are to cars.” (P7)

Table 2. Suggested materials for different application domains, with participant examples.

Discussion

Material Metaphor Effectiveness and Its Limits. Our findings reveal that no single material metaphor is universally effective. Magnets received consistently high ratings, likely because the attract/repel logic maps cleanly onto compatibility relationships with minimal interpretive ambiguity. Clay and sand, by contrast, elicited polarized responses. Our findings suggest that the effectiveness of a material metaphor depends not only on the structural fit between material affordances and constraint type, but also on individual users’ embodied experiences with those materials.

A core design tension underlies these differences: the more literal the mapping between physical material and digital behavior, the easier users can form a mental model, but overly faithful mappings risk introducing false equivalences. Physical clay, for example, cannot be unsculpted, yet reversibility is expected and necessary in a digital design tool. As P4 reflected, a perfect one-to-one correspondence eliminates the metaphor’s generative power, while excessive vagueness leaves users without a workable mental model. This reflects what Smith [56] called the “tension between literalism and magic”—the design challenge of deciding when the metaphorical vehicle should give way to capabilities of the digital system [15]. Our results suggest that the most productive material metaphors are those that preserve enough physical fidelity to be immediately graspable while deliberately departing from literal behavior where the design task demands it.

Material Metaphors Complement Rather Than Replace Other Interaction Paradigms. Several participants expressed a desire to combine material interactions with other modalities (drawing, speech, and text input), particularly when they wanted finer-grained control over generated outputs. Participants observed that material metaphors are best suited for the early, exploratory phases of a design process, where approximate, whole-handed gestures align well with the goal of rapidly sketching out constraint relationships and spatial arrangements. This positioning resonates with Texterial [54], which similarly found that material-based interactions complement rather than replace conventional writing interfaces. In our case, material metaphors excel at externalizing high-level constraint structures (e.g., which zones attract or repel, how density distributes across a site) but are inherently imprecise for specifying exact values. The same holds against the direct manipulation techniques discussed in § 2.3. Soft selection, force fields, and mask-based conditioning are conventionally exposed through explicit parameters that users can set, whereas we deliberately trade that precision for approximate, whole-handed gestures. Future systems could therefore position material interactions as one layer within a multi-modal toolkit, allowing users to transition fluidly from gestural exploration to text- or drawing-based refinement as their design intent advances.

Technical Challenges and Open Questions. Translating gestural input into genAI control parameters remains highly scenario-specific in the current system. Each material-constraint pairing required manual mappings from gesture input to system prompts and ControlNet parameters. Systematizing this translation through a declarative grammar that maps gesture input to prompt modifiers is a promising future work direction. The reverse direction, propagating model state back into material behavior, remains unexplored: clay could stiffen where sampling variance is high, or sand density shift with a diffusion model’s cross-attention maps. Whether such feedback clarifies model state or undermines the metaphor’s predictability is an open question.

Resources, Substrates, and Material Decomposition. Our framework conceptualizes constraints as interaction between resources and substrates; this decomposition then guides material selection. This lens applies well to spatial and proportional constraints, where resources are tangible—but constraints with abstract resources (temporality, logical exclusions) lack obvious physical substrates. Yet, the resource-substrate vocabulary is not inherently spatial, and offers structured ways to reason about constraint anatomy, inviting experimentation with non-obvious mappings leveraging materials for abstract constraints.

Our constraint anatomy admits substrates of one to three dimensions (Figure 3), but every substrate in our probes is two-dimensional. Building height is volumetric, yet users express it by pinching a planar grid whose depth is conveyed through color. Richer surfaces could align a material’s dimensionality with its constraints; however, manipulating 3D content through 2D input is a long-standing problem [23] and depth carries its own perceptual costs [31].

Study Design Limitations. Our themes were developed by one researcher and reflect one interpretive perspective. Single-session studies of a novel tactile interface are subject to novelty [29] and demand [14] effects, so our engagement ratings should be read as initial reactions rather than durable evidence. We also did not include a baseline condition: MATERIALVELLUM is a technology probe intended to surface reactions to a design space, and we make no claim that material metaphors outperform text prompting or established direct manipulation techniques. Empirical comparison against these techniques is an important next step, and should evaluate not only task performance but also qualities like enjoyability and memorability, which are where we expect the value of a familiar material metaphor to lie.

Conclusion

MATERIALVELLUM demonstrates how physical material affordances can function as an expressive lens for articulating design constraints when working with genAI. Our framework offers an initial approximation of how material properties may be mapped to constraint qualities through resources and substrates. As genAI becomes increasingly central in creative practice, material metaphors suggest one path toward interactions grounded in embodied human experience. Looking forward, we hope this work inspires exploration of alternative material-to-AI interaction mappings, advancing more creative material-grounded forms of human-AI co-creation.

Acknowledgments

We thank the participants of our user study for taking the time to provide feedback on MATERIALVELLUM as well as the reviewers of this submission for concrete suggestions to improve this paper. We also thank Jocelyn Shen, Rima Cao, Ryan Yen, Jude Rayan, Arnavi Chedha-Kothary, Ann Paradaso, Andrew D. Wilson and Jen Fox for their feedback throughout this project.

References

PIDAge RangeGenderOccupationDesign ExperienceImage Generation Tools
P125 to 34WomanSoftware Engineer (prev. Architect, 6+ years)IntermediateDALL-E, Midjourney
P225 to 34ManSoftware EngineerIntermediateDALL-E, Stable Diffusion
P335 to 44WomanExecutive AssistantNo design experienceNone
P425 to 34WomanPhD StudentBasic/BeginnerNone
P525 to 34ManResearcher (prev. Landscape Architect, 6+ years)IntermediateMidjourney, Adobe Firefly, Stable Diffusion
P625 to 34ManPhD StudentNo design experienceDALL-E, Stable Diffusion
P725 to 34WomanPhD StudentBasic/BeginnerStable Diffusion, DALL-E
P818 to 24WomanPhD StudentBasic/BeginnerDALL-E, Midjourney, Stable Diffusion
P955 to 64ManTechnical Program ManagerNo design experienceBing, Copilot
P1018 to 24WomanStudentBasic/BeginnerDALL-E, Midjourney, Stable Diffusion

Table 3. Participant demographics, design experience, and prior use of image generation tools.

Technical pipeline for the urban design probe

Figure 12. Technical pipeline for the urban design probe. Zoning inputs generate template-based 3D models; clay-based height manipulation feeds depth maps to ControlNet for photorealistic rendering; magnet-placed functions create inpainting masks for targeted image generation.

References

  1. [1]David Ahlström. 2005. Modeling and Improving Selection in Cascading Pull-Down Menus Using Fitts’ Law, the Steering Law and Force Fields. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’05). Association for Computing Machinery, New York, NY, USA, 61–70. doi:10.1145/1054972.1054982DOI
  2. [2]Saskia Bakker, Alissa N. Antle, and Elise van den Hoven. 2012. Embodied Metaphors in Tangible Interaction Design. Personal and Ubiquitous Computing 16, 4 (April 2012), 433–449. doi:10.1007/s00779-011-0410-4DOI
  3. [3]Michael Mose Biskjaer. 2014. Self-Imposed Creativity Constraints. Ph.D. Dissertation. Aarhus University.
  4. [4]Michael Mose Biskjaer, Per Dalsgaard, and Kim Halskov. 2014. A Constraint Based Understanding of Design Spaces. In Proceedings of the 2014 Conference on Designing Interactive Systems (DIS ’14). Association for Computing Machinery, New York, NY, USA, 453–462. doi:10.1145/2598510.2598533DOI
  5. [5]Michael Mose Biskjaer and Kim Halskov. 2014. Decisive Constraints as a Creative Resource in Interaction Design. Digital Creativity 25, 1 (Jan. 2014), 27–61. doi:10.1080/14626268.2013.855239DOI
  6. [6]Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). Association for Computing Machinery, New York, NY, USA, 1–14. doi:10.1145/3586183.3606725
  7. [7]Liam Brummitt. 2014. Matter.js. https://brm.io/matter-js/
  8. [8]Ricardo Cabello. 2010. Three.js – JavaScript 3D Library. https://threejs.org/
  9. [9]Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis. SAGE, Los Angeles London.
  10. [10]Aline Chevalier and Melody Y Ivory. 2003. Web Site Designs: Influences of Designer’s Expertise and Design Constraints. International Journal of Human-Computer Studies 58, 1 (Jan. 2003), 57–87. doi:10.1016/S1071-5819(02)00126-XDOI
  11. [11]John Joong Chung and Eytan Adar. 2023. PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). Association for Computing Machinery, New York, NY, USA, 1–17. doi:10.1145/3586183.3606777DOI
  12. [12]John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–19. doi:10.1145/3491102.3501819DOI
  13. [13]Hai Dang, Frederik Brudy, George Fitzmaurice, and Fraser Anderson. 2023. World-Smith: Iterative and Expressive Prompting for World Building with a Generative AI. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). Association for Computing Machinery, New York, NY, USA, 1–17. doi:10.1145/3586183.3606772DOI
  14. [14]Nicola Dell, Vidya Vaidyanathan, Indrani Medhi, Edward Cutrell, and William Thies. 2012. "Yours Is Better!": Participant Response Bias in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). Association for Computing Machinery, New York, NY, USA, 1321–1330. doi:10.1145/2207676.2208589DOI
  15. [15]Paul Dourish. 2001. Where the Action Is: The Foundations of Embodied Interaction. The MIT Press. doi:10.7551/mitpress/7221.001.0001DOI
  16. [16]Jack Doyle. 2024. GSAP: GreenSock Animation Platform. https://gsap.com/
  17. [17]Dicle N. Dövencioğlu, Feybun Üstün, Katja Doerschner, and Knut Drewing. 2022. Hand Explorations Are Determined by the Characteristics of the Perceptual Space of Real-World Materials from Silk to Sand. Scientific Reports 12, 1 (Aug. 2022), 14785. doi:10.1038/s41598-022-18901-6
  18. [18]Donald L. Elliott. 2008. A Better Way to Zone: Ten Principles to Create More Livable Cities. Island Press, Washington, DC.
  19. [19]Jon Elster. 2000. Ulysses Unbound: Studies in Rationality, Precommitment, and Constraints. Cambridge University Press, New York.
  20. [20]Elisa Giaccardi and Elvin Karana. 2015. Foundations of Materials Experience: An Approach for HCI. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). Association for Computing Machinery, New York, NY, USA, 2447–2456. doi:10.1145/2702123.2702337DOI
  21. [21]James J. Gibson. 2014. The Theory of Affordances. In The People, Place, and Space Reader. Routledge, 56–60.
  22. [22]Michael Gleicher and Andrew Witkin. 1994. Drawing with Constraints. The Visual Computer 11, 1 (Jan. 1994), 39–51. doi:10.1007/BF01900698DOI
  23. [23]Mark Hancock, Sheelagh Carpendale, and Andy Cockburn. 2007. Shallow-Depth 3D Interaction: Design and Evaluation of One-, Two- and Three-Touch Techniques. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07). Association for Computing Machinery, New York, NY, USA, 1147–1156. doi:10.1145/1240624.1240798DOI
  24. [25]Giulio Jacucci and Ina Wagner. 2007. Performative Roles of Materiality for Collective Creativity. In Proceedings of the 6th ACM SIGCHI Conference on Creativity & Cognition (C&C ’07). Association for Computing Machinery, New York, NY, USA, 73–82. doi:10.1145/1254960.1254971DOI
  25. [26]Elvin Karana, Bahareh Barati, Valentina Rognoli, and Anouk Zeeuw Van Der Laan. 2015. Material Driven Design (MDD): A Method to Design for Material Experiences. International Journal of Design 9, 2 (2015), 35–54. https://re.public.polimi.it/handle/11311/979536
  26. [27]Rubaiat Habib Kazi, Kien Chuan Chua, Shengdong Zhao, Richard Davis, and Kok-Lim Low. 2011. SandCanvas: A Multi-Touch Art Medium Inspired by Sand Animation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). Association for Computing Machinery, New York, NY, USA, 1283–1292. doi:10.1145/1978942.1979133DOI
  27. [28]Rachel Kelly. 2017. Ikebana: A Collaborative Design Pedagogy. The Design Journal 20, sup1 (July 2017), S1093–S1105. doi:10.1080/14606925.2017.1353053DOI
  28. [29]Michael Koch, Kai von Luck, Jan Schwerz, and Susanne Draheim. 2018. The Novelty Effect in Large Display Deployments – Experiences and Lessons-Learned for Evaluating Prototypes. In Proceedings of the 16th European Conference on Computer-Supported Cooperative Work – Exploratory Papers. European Society for Socially Embedded Technologies (EUSSET). doi:10.18420/ecscw2018_3
  29. [30]Kohn Pedersen Fox Associates. 2014. Ideal Block & Master Plan. https://uk.kpf.com/london-block
  30. [31]Joseph J. LaViola Jr, Ernst Kruijff, Ryan P. McMahan, Doug Bowman, and Ivan Poupyev. 2017. 3D User Interfaces: Theory and Practice. Addison-Wesley Professional.
  31. [32]Bryan Lawson. 2006. How Designers Think. Routledge.DOI
  32. [33]Long Liu, Junbin Ren, Zeyuan Fan, Chenhui Li, Gaoqi He, Changbo Wang, Yang Gao, and Chen Li. 2025. SandTouch: Empowering Virtual Sand Art in VR with AI Guidance and Emotional Relief. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, 1–21. doi:10.1145/3706598.3714275DOI
  33. [34]Vivian Liu. 2023. Beyond Text-to-Image: Multimodal Prompts to Explore Generative AI. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA ’23). Association for Computing Machinery, New York, NY, USA, 1–6. doi:10.1145/3544549.3577043DOI
  34. [35]Vivian Liu and Lydia B Chilton. 2022. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–23. doi:10.1145/3491102.3501825DOI
  35. [36]Thor Magnusson. 2010. Designing Constraints: Composing and Performing with Digital Musical Systems. Computer Music Journal 34, 4 (2010), 62–73. https://www.jstor.org/stable/40962941
  36. [37]Atefeh Mahdavi Goloujeh, Anne Sullivan, and Brian Magerko. 2024. Is It AI or Is It Me? Understanding Users’ Prompt Journey with Text-to-Image Generative AI Tools. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 1–13. doi:10.1145/3613904.3642861
  37. [38]Nicola Marquardt, Asta Rosezaw, Hugo Romat, Payod Panda, Michel Pahud, Gonzalo Ramos, Steven M. Drucker, Andrew D. Wilson, Ken Hinckley, and Nathalie Riche. 2025. ImaginationVellum: Generative-AI Ideation Canvas with Spatial Prompts, Generative Strokes, and Ideation History. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST ’25). Association for Computing Machinery, New York, NY, USA, 1–19. doi:10.1145/3746059.3747631DOI
  38. [39]Damien Masson, Sylvain Malacria, Géry Casiez, and Daniel Vogel. 2024. Direct-GPT: A Direct Manipulation Interface to Interact with Large Language Models. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 1–16. doi:10.1145/3613904.3642462DOI
  39. [40]Ravi Mehta and Meng Zhu. 2016. Creating When You Have Less: The Impact of Resource Scarcity on Product Use Creativity. Journal of Consumer Research 42, 5 (Feb. 2016), 767–782. doi:10.1093/jcr/ucw051
  40. [41]Elisa D. Melker and Kasper Hornbæk. 2019. A Framework for the Experience of Meaning in Human-Computer Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–15. doi:10.1145/3290605.3300455
  41. [42]Meta Open Source. 2024. React. https://react.dev/
  42. [43]Balder Onarheim and Stefan Wiltschnig. 2010. Opening and Constraining: Constraints and Their Role in Creative Processes. In Proceedings of the 1st DESIRE Network Conference on Creativity and Innovation in Design. 83–89. https://innovationenglish.sites.ku.dk/files/2013/12/creativity-constraints.pdf
  43. [44]OpenAI. 2023. GPT-4 Model | OpenAI API. https://developers.openai.com/api/docs/models/gpt-4
  44. [45]Pallets. 2024. Flask. https://flask.palletsprojects.com/en/
  45. [46]XiaoHan Peng, Janin Koch, and Wendy E. Mackay. 2024. DesignPrompt: Using Multimodal Interaction for Design Exploration with Generative AI. In Proceedings of the 2024 ACM Designing Interactive Systems Conference (DIS ’24). Association for Computing Machinery, New York, NY, USA, 804–818. doi:10.1145/3643834
  46. [47]Walter R. Reitman. 1964. Heuristic Decision Procedures, Open Constraints, and the Structure of Ill-Defined Problems. Human Judgments and Optimality 282 (1964), 283–315.
  47. [48]Miguel A. Renom, Baptiste Caramiaux, and Michel Beaudouin-Lafon. 2022. Exploring Technical Reasoning in Digital Tool Use. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–17. doi:10.1145/3491102.3501877DOI
  48. [49]Eirik Resch, Rolf André Bohne, Trond Kvamsdal, and Jardar Lohne. 2016. Impact of Urban Density and Building Height on Energy Use in Cities. Energy Procedia 96 (Sept. 2016), 800–814. doi:10.1016/j.egypro.2016.09.142DOI
  49. [50]Nathalie Riche, Anna Offenwanger, Frederic Gmeiner, David Brown, Hugo Romat, Michel Pahud, Nicolai Marquardt, Kori Inkpen, and Ken Hinckley. 2025. AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, 1–18. doi:10.1145/3706598.3714259DOI
  50. [51]Daniela Roszner, Jean-François Blanchette, Leah Buchley, Paul Dourish, and Melissa Mazmanian. 2012. From Materials to Materiality: Connecting Practice and Theory in HCI. In CHI ’12 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’12). Association for Computing Machinery, New York, NY, USA, 2787–2790. doi:10.1145/2212776.2212721
  51. [52]Shozo Sato. 2012. Ikebana: The Art of Arranging Flowers. Tuttle Publishing.
  52. [53]Paul Scudieri and Blaine Lilly. 2011. A Constraint-Based Model of the Design Process. DS 69: Proceedings of E&PDE 2011, the 13th International Conference on Engineering and Product Design Education, London, UK, 08.-09.09.2011 (2011), 205–210.
  53. [54]Jocelyn Shen, Nicolai Marquardt, Hugo Romat, Ken Hinckley, Nathalie Riche, and Fanny Chevalier. 2026. Texterial: A Text-as-Material Interaction Paradigm for LLM-Mediated Writing. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI ’26). Association for Computing Machinery, New York, NY, USA, 1–18. doi:10.1145/3772318.3790330DOI
  54. [55]Xinyu Shi, Yinghuo Wang, Ryan Rossi, and Jian Zhao. 2025. Brickify: Enabling Expressive Design Intent Specification through Direct Manipulation on Design Tokens. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, 1–20. doi:10.1145/3706598.3714087DOI
  55. [56]Randall B. Smith. 1987. Experiences with the Alternate Reality Kit: An Example of the Tension between Literalism and Magic. In Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface, Vol. 17. 61–67. doi:10.1145/29933.30861DOI
  56. [57]Martin Stacey and Claudia Eckert. 2010. Reshaping the Box: Creative Designing as Constraint Management. International Journal of Product Development 11, 3/4 (2010), 241–255. doi:10.1504/IJPD.2010.033960
  57. [58]Patricia D. Stokes. 2005. Creativity from Constraints: The Psychology of Breakthrough. Lawrence Erlbaum Associates.
  58. [59]Hari Subramonyam, Roy Pea, Christopher Pondoc, Maneesh Agrawala, and Colleen Seifert. 2024. Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24). Association for Computing Machinery, New York, NY, USA, 1–19. doi:10.1145/3613904.3642754DOI
  59. [60]Zhida Sun, Sintong Wang, Chengxong Liu, and Xiaojian Ma. 2022. Metaphor-action: Support Gesture-Based Interaction Design with Metaphorical Meanings. ACM Transactions on Computer-Human Interaction 29, 5 (Oct. 2022), 1–33. doi:10.1145/3511892DOI
  60. [61]Amanda Swearngin, Chenglong Wang, Alannah Oleson, James Fogarty, and Amy J. Ko. 2020. Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. doi:10.1145/3313831.3376593DOI
  61. [62]David I. Tafler. 2019. Drawing Spirits in the Sand: Performative Storytelling in the Digital Age. Religions 10, 9 (Aug. 2019), 492. doi:10.3390/rel10090492DOI
  62. [63]Liang Tan and Kenny K. N. Chow. 2018. An Embodied Approach to Designing Meaningful Experiences with Ambient Media. Multimodal Technologies and Interaction 2, 2 (June 2018), 13. doi:10.3390/mti2020013
  63. [64]Mikael Wiberg, Hiroshi Ishii, Paul Dourish, Anna Vallgårda, Tobie Kerridge, Petra Sundström, Daniela Rosner, and Mark Rolston. 2013. Materiality Matters—Experience Materials. Interactions 20, 2 (March 2013), 54–57. doi:10.1145/2427076.2427087
  64. [65]J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23). Association for Computing Machinery, New York, NY, USA, 1–21. doi:10.1145/3544548.3581388
  65. [66]Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Paris, France, 3813–3824. doi:10.1109/ICCV51070.2023.00355

Paper details

vlnc:W_6rcj2yg8
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents