publications

2026

1. 

   DeltaDorsal: Enhancing Hand Pose Estimation with Dorsal Features in Egocentric Views

   William Huang, Siyou Pei, Leyi Zou, Eric J. Gonzalez, Ishan Chatterjee, and Yang Zhang

   CHI ’26

   The proliferation of XR devices has made egocentric hand pose estimation a vital task, yet this perspective is inherently challenged by frequent finger occlusions. To address this, we propose a novel approach that leverages the rich information in dorsal hand skin deformation, unlocked by recent advances in dense visual featurizers. We introduce a dual-stream delta encoder that learns pose by contrasting features from a dynamic hand with a baseline relaxed position. Our evaluation demonstrates that, using only cropped dorsal images, our method reduces the Mean Per Joint Angle Error (MPJAE) by 18% in self-occluded scenarios (fingers (ge 50%) occluded) compared to state-of-the-art techniques that depend on the whole hand’s geometry and large model backbones. Consequently, our method not only enhances the reliability of downstream tasks like index finger pinch and tap estimation in occluded scenarios but also unlocks new interaction paradigms, such as detecting isometric force for a surface “click” without visible movement while minimizing model size. Our codebase is found at https://github.com/hilab-open-source/deltadorsal.

2. 

   World Mouse: Exploring Interactions with a Cross-Reality Cursor

   Esen K. Tütüncü, Mar Gonzalez-Franco, Khushman Patel, and Eric J Gonzalez

   CHI Workshops ’26

3. 

   Sticky and Magnetic: Evaluating Error Correction and User Adaptation in Gaze and Pinch Interaction

   Jazmin Collins, Prasanthi Gurumurthy, Eric J Gonzalez, and Mar Gonzalez-Franco

   CHI LBW ’26

   The gaze-and-pinch framework offers a high-fidelity interaction modality for spatial computing in virtual reality (VR), yet it remains vulnerable to coordination errors—timing misalignments between gaze fixation and pinch gestures. These errors are categorized into two types: late triggers (gaze leaves a target before pinch) and early triggers (pinch before gaze arrival on target). While late triggers are well-studied, early triggers lack robust solutions. We investigate two heuristics—Sticky selection (temporal buffer) and Magnetic selection (spatial field)—to mitigate these errors. A within-subjects study (N = 9) on the Samsung Galaxy XR evaluated these heuristics against a baseline. Findings indicate that while throughput and selection time remained stable, the heuristics fundamentally shifted user behavior and significantly reduced errors during selection. Notably, Magnetic selection induced an “offloading” effect where users traded precision for speed. Additionally, the heuristics reclassified ambiguous failures as explainable coordination errors. We provide recommendations for selection heuristics that enhance interaction speed and cognitive agency in virtual reality.

4. 

   Navig-AI-tion: Navigation by Contextual AI and Spatial Audio

   Mathias N. Lystbæk, Haley Adams, Ranjith Kagathi Ananda, Eric J Gonzalez, Luca Ballan, Qiuxuan Wu, Andrea Colaço, Peter Tan, and Mar Gonzalez-Franco

   CHI LBW ’26

   Audio-only walking navigation can leave users disoriented, relying on vague cardinal directions and lacking real-time environmental context, leading to frequent errors. To address this, we present a novel system that integrates a Vision Language Model (VLM) with a spatial audio cue. Our system extracts environmental landmarks to anchor navigation instructions and, crucially, provides a directional spatial audio signal when the user faces the wrong direction, indicating the precise turn direction. In a user study (n=12), the spatial audio cue with VLM reduced route deviations compared to both VLM-only and Google Maps (audio-only) baseline systems. Users reported that the spatial audio cue effectively supported orientation and that landmark-anchored instructions provided a better navigation experience over audio-only Google Maps. This work serves as an initial look at the utility of future audio-only navigation systems for incorporating directional cues, especially real-time corrective spatial audio.

5. 

   Break the Window: Exploring Spatial Decomposition of Webpages in XR

   Chenyang Zhang, Tianjian Wei, Haoyang Yang, Mar Gonzalez-Franco, Yalong Yang, and Eric J Gonzalez

   CHI LBW ’26

   Most XR web browsers still present webpages as a single floating window, carrying over desktop design assumptions into immersive space. We explore an alternative by breaking the browser window and distributing a webpage into spatial UI chunks within a mixed-reality workspace. We present Break-the-Window (BTW), an exploratory prototype that spatially decomposes live, fully functional webpages into movable panels supporting mid-air and surface-attached placement, as well as direct touch and ray-based interaction. Through a formative study with XR practitioners and an exploratory qualitative study with 15 participants, we observed how spatial decomposition supports distributed attention and spatial meaning-making, while also surfacing challenges around coordination effort, interaction precision, and the lack of shared spatial UI conventions. This work invites discussion on how web interfaces might be reimagined for spatial computing beyond the single-window paradigm.

6. 

   SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions

   Vasco Xu, Brian Chen, Eric J. Gonzalez, Andrea Colaço, Henry Hoffmann, Mar Gonzalez-Franco, and Karan Ahuja

   IEEE VR ’26 (TVCG Special Issue)

2025

1. 

   HandOver: Enabling Precise Selection & Manipulation of 3D Objects with Mouse and Hand Tracking

   Esen K. Tütüncü, Mar Gonzalez-Franco, and Eric J Gonzalez

   UIST ’25

   We present HandOver, an extended reality (XR) interaction technique designed to unify the precision of traditional mouse input for object selection with the expressiveness of hand-tracking for object manipulation. With HandOver, the mouse is used to drive a depth-aware 3D cursor enabling precise and restful targeting — by hovering their hand over the mouse, the user can then seamlessly transition into direct 3D manipulation of the target object. In a formal user study, we compare HandOver against two ray-based techniques: traditional raycasting (Ray) and a hybrid method (Ray+Hand) in a 3D docking task. Results show HandOver yields lower task errors across all distances, and moreover improves interaction ergonomics as highlighted by a RULA posture analysis and self-reported measures (NASA-TLX). These findings illustrate the benefits of blending traditional precise input devices with the expressive gestural inputs afforded by hand-tracking in XR, leading to improved user comfort and task performance. This blended paradigm yields a unified workflow allowing users to leverage the best of each input modality as they interact in immersive environments.

2. 

   ForcePinch: Force-Responsive Spatial Interaction for Tracking Speed Control in XR

   Chenyang Zhang, Tiffany S Ma, John Andrews, Eric J Gonzalez, Mar Gonzalez-Franco, and Yalong Yang

   UIST ’25

   Spatial interaction in 3D environments requires balancing efficiency and precision, which requires dynamic tracking speed adjustments. However, existing techniques often couple tracking speed adjustments directly with hand movements, reducing interaction flexibility. Inspired by the natural friction control inherent in the physical world, we introduce ForcePinch, a novel force-responsive spatial interaction method that enables users to intuitively modulate pointer tracking speed and smoothly transition between rapid and precise movements by varying their pinching force. To implement this concept, we developed a hardware prototype integrating a pressure sensor with a customizable mapping function that translates pinching force into tracking speed adjustments. We conducted a user study with 20 participants performing well-established 1D, 2D, and 3D object manipulation tasks, comparing ForcePinch against the distance-responsive technique Go-Go and speed-responsive technique PRISM. Results highlight distinctive characteristics of the force-responsive approach across different interaction contexts. Drawing on these findings, we highlight the contextual meaning and versatility of force-responsive interactions through four illustrative examples, aiming to inform and inspire future spatial interaction design.

3. 

   Programmable Reality

   Ryo Suzuki, Parastoo Abtahi, Chen Zhu-Tian, Mustafa Doga Dogan, Andrea Colaco, Eric J. Gonzalez, Karan Ahuja, and Mar Gonzalez-Franco

   Frontiers in Virtual Reality, 2025

   Innovations in spatial computing and artificial intelligence (AI) are making it possible to overlay dynamic, interactive digital elements on the physical world. Soon, every object might have a real-time digital twin, enabling the “Internet of Things” so as to identify and interact with even unconnected items. This programmable reality would enable computational manipulation of the world around us through alteration of its appearance or functionality, similar to software, but for reality itself. Advances in AI language models have enabled zero-shot segmentation and understanding of the world, making it possible to query and manipulate objects with precision. However, this vision also demands natural and intuitive ways for humans to interact with these models through gestures, gaze, and existing devices. Augmented reality (AR) provides the ideal bridge between AI output and human input in the physical world. Moreover, diffusion models and physics simulations offer exciting possibilities for content generation and editing, allowing us to transform everyday activities into extraordinary experiences. As AR devices become ubiquitous and indistinguishable from reality, these technologies blur the lines between reality and simulations. This raises profound questions about how we perceive and experience the world while having implications for memory, learning, and even behavior. Programmable reality enabled by AR and AI has vast potential to reshape our relationships with the digital realm, ultimately making it an extension of the physical realm.

4. 

   RestfulRaycast: Exploring Ergonomic Rigging and Joint Amplification for Precise Hand Ray Selection in XR

   Hongyu Mao, Mar Gonzalez-Franco, Vrushank Phadnis, Eric J Gonzalez, and Ishan Chatterjee

   DIS ’25

   Hand raycasting is widely used in extended reality (XR) for selection and interaction, but prolonged use can lead to arm fatigue (e.g., "gorilla arm"). Traditional techniques often require a large range of motion where the arm is extended and unsupported, exacerbating this issue. In this paper, we explore hand raycast techniques aimed at reducing arm fatigue, while minimizing impact to precision selection. In particular, we present Joint-Amplified Raycasting (JAR) – a technique which scales and combines the orientations of multiple joints in the arm to enable more ergonomic raycasting. Through a comparative evaluation with the commonly used industry standard Shoulder-Palm Raycast (SP) and two other ergonomic alternatives—Offset Shoulder-Palm Raycast (OSP) and Wrist-Palm Raycast (WP)—we demonstrate that JAR results in higher selection throughput and reduced fatigue. A follow-up study highlights the effects of different JAR joint gains on target selection and shows users prefer JAR over SP in a representative UI task.

5. 

   Online-EYE: Multimodal Implicit Eye Tracking Calibration for XR

   Baosheng James Hou, Lucy Abramyan, Prasanthi Gurumurthy, Haley Adams, Ivana Tosic Rodgers, Eric J Gonzalez, Khushman Patel, Andrea Colaço, Ken Pfeuffer, Hans Gellersen, Karan Ahuja, and Mar Gonzalez-Franco

   CHI ’25

   Unlike other inputs for extended reality (XR) that work out of the box, eye tracking typically requires custom calibration per user or session. We present a multimodal inputs approach for implicit calibration of eye tracker in VR, leveraging UI interaction for continuous, background calibration. Our method analyzes gaze data alongside controller interaction with UI elements, and employing ML techniques it continuously refines the calibration matrix without interrupting users from their current tasks. Potentially eliminating the need for explicit calibration. We demonstrate the accuracy and effectiveness of this implicit approach across various tasks and real time applications achieving comparable eye tracking accuracy to native, explicit calibration. While our evaluation focuses on VR and controller-based interactions, we anticipate the broader applicability of this approach to various XR devices and input modalities.

6. Practical Input Considerations for Wearable AI Assistants in XR

   Eric J. Gonzalez

   CHI Workshops ’25

7. 

   H2E: Hand, Head, Eye a Multimodal Cascade of Natural Inputs

   Khushman Patel, Vrushank Phadnis, Eric J Gonzalez, Hans Gellersen, Ken Pfeuffer, and Mar Gonzalez-Franco

   IEEE VRW ’25

8. 

   Beyond the Phone: Exploring Phone-XR Integration through Multi-View Transitions for Real-World Applications

   Fengyuan Zhu, Xun Qian, Daniel Kalmar, Mahdi Tayarani, Eric J Gonzalez, Mar Gonzalez-Franco, David Kim, and Ruofei Du

   IEEE VR ’25

9. 

   EmBARDiment: an Embodied AI Agent for Productivity in XR

   Riccardo Bovo, Steven Abreu, Karan Ahuja, Eric J Gonzalez, Li-Te Cheng, and Mar Gonzalez-Franco

   IEEE VR ’25

   </>

10. 

    Revisiting Put-That-There: Context Aware Window Interactions via LLMs

    Riccardo Bovo, Daniele Giunchi, Pasquale Cascarano, Eric J Gonzalez, and Mar Gonzalez-Franco

    ISMAR Workshops ’25

2024

1. 

   PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos

   Steven Abreu, Tiffany D. Do, Karan Ahuja, Eric J. Gonzalez, Lee Payne, Daniel McDuff, and Mar Gonzalez-Franco

   ICLR Workshops ’25

2. 

   Hovering Over the Key to Text Input in XR

   Mar Gonzalez-Franco, Diar Abdlkarim, Arpit Bhatia, Stuart Macgregor, Jason A Fotso-Puepi, Eric J Gonzalez, Hasti Seifi, Massimiliano Di Luca, and Karan Ahuja

   IEEE ISEMV ’24

3. 

   Augmented Object Intelligence with XR-Objects

   Mustafa Doga Dogan, Eric J Gonzalez, Karan Ahuja, Ruofei Du, Andrea Colaço, Johnny Lee, Mar Gonzalez-Franco, and David Kim

   UIST ’24

   </> 🏅 SXSW Innovation Award Finalist

4. 

   Hands-on, Hands-off: Gaze-Assisted Bimanual 3D Interaction

   Mathias N. Lystbæk, Thorbjørn Mikkelsen, Roland Krisztandl, Eric J Gonzalez, Mar Gonzalez-Franco, Hans Gellersen, and Ken Pfeuffer

   UIST ’24

5. 

   Virtual Games, Real Interactions: A Look at Cross-reality Asymmetrical Co-located Social Games

   Alexandra Kitson, Sun Joo (Grace) Ahn, Eric J Gonzalez, Payod Panda, Katherine Isbister, and Mar Gonzalez-Franco

   CHI LBW ’24

6. 

   Intent-driven Input Device Arbitration for XR

   Eric J Gonzalez, Ishan Chatterjee, Mar Gonzalez-Franco, Andrea Colaço, and Karan Ahuja

   CHI LBW ’24 / ISMAR Demo ’24

7. 

   WindowMirror: An Opensource Toolkit to Bring Interactive Multi-Window Views into XR

   Riccardo Bovo, Eric J. Gonzalez, Li-Te Cheng, and Mar Gonzalez-Franco

   IEEE VRW ’24

   </>

8. 

   XDTK: A Cross-Device Toolkit for Input & Interaction in XR

   Eric J. Gonzalez, Khushman Patel, Karan Ahuja, and Mar Gonzalez-Franco

   IEEE VRW ’24 / ISMAR Demo ’24

   </>

2023

1. 

   Sensorimotor Simulation of Redirected Reaching using Stochastic Optimal Feedback Control

   Eric J Gonzalez and Sean Follmer

   CHI ’23

   🏆 Best Paper

2022

1. 

   A Model Predictive Control Approach for Reach Redirection in Virtual Reality

   Eric J Gonzalez, Elyse D. Z. Chase, Pramod Kotipalli, and Sean Follmer

   CHI ’22
2. Redirected Reaching in Virtual Reality : Modeling, Control & Applications

   Eric J Gonzalez

   PhD Thesis (2022)

2021

1. 

   X-Rings: A Hand-mounted 360° Shape Display for Grasping in Virtual Reality

   Eric J Gonzalez, Eyal Ofek, Mar Gonzalez-Franco, and Mike Sinclair

   UIST ’21

   🏅 Honorable Mention

2. 

   Hybrid Actuation With Unidirectional Clutches for Handheld Haptic Controllers

   Inrak Choi, Eric J. Gonzalez, and Sean Follmer

   IEEE RAL, 2021
3. Haptic Guidance to Support Design Education and Collaboration for Blind and Visually Impaired People

   Alexa F. Siu, Elyse D. Z. Chase, Gene S.-H. Kim, Abena Boadi-Agyemang, Eric J. Gonzalez, and Sean Follmer

   Design Thinking Research (Book Chapter)

   Designers create sketches, diagrams, and other visual media to both externalize a specific design concept as well as to explore design spaces. The largely visual and spatial nature of these diagrams used to support design activities poses several challenges for blind and visually impaired (BVI) designers to participate along sighted peers. This challenge includes the creation of tools to support the teaching of tactile graphics and their collaborative use in the context of design education. In efforts to address several of these challenges, we present PantoGuide, a low-cost system that provides audio and haptic guidance, via skin-stretch feedback to the dorsum of a user’s hand while the user explores a tactile graphic overlaid on a touchscreen. This system allows programming of haptic guidance patterns and cues for tactile graphics that can be experienced by students learning remotely or that can be reviewed by a student independently.

4. 

   Augmenting Perceived Softness of Haptic Proxy Objects Through Transient Vibration and Visuo-Haptic Illusion in Virtual Reality

   Inrak Choi, Yiwei Zhao, Eric J. Gonzalez, and Sean Follmer

   IEEE TVCG, 2021

2020

1. 

   PantoGuide: A Haptic and Audio Guidance System To Support Tactile Graphics Exploration

   Elyse D. Z. Chase, Alexa Fay Siu, Abena Boadi-Agyemang, Gene S-H Kim, Eric J Gonzalez, and Sean Follmer

   ASSETS ’20

2. 

   REACH+: Extending the Reachability of Encountered-type Haptics Devices through Dynamic Redirection in VR

   Eric J. Gonzalez, Parastoo Abtahi, and Sean Follmer

   UIST ’20

2019

1. 

   Investigating the Detection of Bimanual Haptic Retargeting in Virtual Reality

   Eric J. Gonzalez and Sean Follmer

   VRST ’19

   🏆 Best Short Paper

2. 

   Evaluating the Minimum Jerk Motion Model for Redirected Reach in Virtual Reality

   Eric J. Gonzalez, Parastoo Abtahi, and Sean Follmer

   UIST ’19 Adjunct

3. 

   Design and Analysis of High-Resolution Electrostatic Adhesive Brakes Towards Static Refreshable 2.5D Tactile Shape Display

   Kai Zhang, Eric J. Gonzalez, Jianglong Guo, and Sean Follmer

   IEEE TOH, 2019

2018

1. 

   shapeShift: 2D Spatial Manipulation and Self-Actuation of Tabletop Shape Displays for Tangible and Haptic Interaction

   Alexa F. Siu, Eric J. Gonzalez, Shenli Yuan, Jason B. Ginsberg, and Sean Follmer

   CHI ’18

   🏅 Best Demo Honorable Mention

2. 

   Influence of Elbow Flexion and Stimulation Site on Neuromuscular Electrical Stimulation of the Biceps Brachii

   Eric J. Gonzalez, Ryan J. Downey, Courtney A. Rouse, and Warren E. Dixon

   IEEE TNSRE, 2018

2017

1. 

   The Time-Varying Nature of Electromechanical Delay and Muscle Control Effectiveness in Response to Stimulation-Induced Fatigue

   Ryan J. Downey, Manelle Merad, Eric J. Gonzalez, and Warren E. Dixon

   IEEE TNSRE, 2017