https://nels.dev/Recent content on Nels NumanHugo -- gohugo.ioen-usFri, 13 Mar 2026 00:00:00 +0000https://nels.dev/updates/2026-03-13/Fri, 13 Mar 2026 00:00:00 +0000https://nels.dev/updates/2026-03-13/I passed my PhD viva with no corrections! My thesis is titled “Constructing Shared Space: Designing for Asymmetry in Collaborative Mixed Reality”. Thank you to everyone who has supported me throughout this journey, including my advisors, examiners, mentors, collaborators, friends, and family.https://nels.dev/updates/2025-10-09/Thu, 09 Oct 2025 00:00:00 +0000https://nels.dev/updates/2025-10-09/XR Blocks, a framework designed to accelerate human-centered AI + XR innovation on the web, which I worked on as a core contributor during my internship at Google, has been released! Check out the Google Research blog article for more info.https://nels.dev/publication/sensibleagent/Wed, 01 Oct 2025 02:00:00 +0100https://nels.dev/publication/sensibleagent/ABSTRACT Proactive AR agents promise context-aware assistance, but their interactions often rely on explicit voice prompts or responses, which can be disruptive or socially awkward. We introduce Sensible Agent,a framework designed for unobtrusive interaction with these proactive agents. Sensible Agent dynamically adapts both “what” assistance to offer and, crucially, “how” to deliver it, based on real-time multimodal context sensing. Informed by an expert workshop (n=12) and a data annotation study (n=40), the framework leverages egocentric cameras, multimodal sensing, and Large Multimodal Models (LMMs) to infer context and suggest appropriate actions delivered via minimally intrusive interaction modes.https://nels.dev/publication/adjustar/Wed, 01 Oct 2025 01:00:00 +0100https://nels.dev/publication/adjustar/View AR poster Vote for UIST 2025 Best Poster Award (function () { try { // Parse URL query parameters var params = new URLSearchParams(window.location.search); var refParam = (params.get('ref') || '').toLowerCase(); // Fallback to document.referrer if the 'ref' parameter is not present var ref = refParam || (document.referrer || '').toLowerCase(); console.debug('Ref source:', ref); // Check for 'uist-poster' if (ref.indexOf('uist-poster') !== -1) { var elements = document.https://nels.dev/publication/xrblocks/Mon, 29 Sep 2025 00:00:00 +0100https://nels.dev/publication/xrblocks/ABSTRACT We are on the cusp where Artificial Intelligence (AI) and Extended Reality (XR) are converging to unlock new paradigms of interactive computing. However, a significant gap exists between the ecosystems of these two fields: while AI research and development is accelerated by mature frameworks like JAX and benchmarks like LMArena, prototyping novel AI-driven XR interactions remains a high-friction process, often requiring practitioners to manually integrate disparate, low-level systems for perception, rendering, and interaction.https://nels.dev/updates/2025-08-09/Sat, 09 Aug 2025 00:00:00 +0000https://nels.dev/updates/2025-08-09/Sensible Agent, a full paper led by Geonsun Lee during our internships at Google, and AdjustAR, a poster led by me in collaboration with Niantic Spatial, have been accepted to ACM UIST ‘25!https://nels.dev/updates/2025-05-27/Tue, 27 May 2025 00:00:00 +0000https://nels.dev/updates/2025-05-27/Continuing my work with the Blended Intelligence Research & Devices (BIRD) lab at Google in Mountain View for the summer! An exciting time for Project Moohan and the rest of the growing AndroidXR family!https://nels.dev/publication/cocreatar/Sun, 27 Apr 2025 02:00:00 +0100https://nels.dev/publication/cocreatar/ABSTRACT Authoring site-specific outdoor augmented reality (AR) experiences requires a nuanced understanding of real-world context to create immersive and relevant content. Existing ex-situ authoring tools typically rely on static 3D models to represent spatial information. However, in our formative study (n=25), we identified key limitations of this approach: models are often outdated, incomplete, or insufficient for capturing critical factors such as safety considerations, user flow, and dynamic environmental changes. These issues necessitate frequent on-site visits and additional iterations, making the authoring process more time-consuming and resource-intensive.https://nels.dev/publication/decoupledhands/Tue, 01 Apr 2025 02:00:00 +0100https://nels.dev/publication/decoupledhands/ABSTRACT When collaborating relative to a shared 3D virtual object in mixed reality (MR), users may experience communication issues arising from differences in perspective. These issues include occlusion (e.g., one user not being able to see what the other is referring to) and inefficient spatial references (e.g., “to the left of this” may be confusing when users are positioned opposite to each other). This paper presents a novel technique for automatic perspective alignment in collaborative MR involving co-located interaction centered around a shared virtual object.https://nels.dev/updates/2025-03-27/Thu, 27 Mar 2025 00:00:00 +0000https://nels.dev/updates/2025-03-27/CoCreatAR received a Best Paper Honorable Mention recognition at ACM CHI ‘25 (top 5% of 5014 submissions)!https://nels.dev/updates/2025-01-16/Thu, 16 Jan 2025 00:00:00 +0000https://nels.dev/updates/2025-01-16/Our paper describing my internship project at Niantic, CoCreatAR, has been accepted to ACM CHI ‘25!https://nels.dev/updates/2024-12-02/Sun, 01 Dec 2024 00:00:00 +0000https://nels.dev/updates/2024-12-02/Started my internship at the Blended Intelligence Research & Devices (BIRD) lab at Google in Seattle under the mentorship of Mar Gonzalez-Franco, Eric Gonzalez, Ruofei Du, and Harish Kulkarni!https://nels.dev/publication/spaceblender/Tue, 01 Oct 2024 02:00:00 +0100https://nels.dev/publication/spaceblender/ABSTRACT There is increased interest in using generative AI to create 3D spaces for virtual reality (VR) applications. However, today’s models produce artificial environments, falling short of supporting collaborative tasks that benefit from incorporating the user’s physical context. To generate environments that support VR telepresence, we introduce SpaceBlender, a novel pipeline that utilizes generative AI techniques to blend users' physical surroundings into unified virtual spaces. This pipeline transforms user-provided 2D images into context-rich 3D environments through an iterative process consisting of depth estimation, mesh alignment, and diffusion-based space completion guided by geometric priors and adaptive text prompts.https://nels.dev/publication/blendscape/Tue, 01 Oct 2024 01:00:00 +0100https://nels.dev/publication/blendscape/ABSTRACT Today’s video-conferencing tools support a rich range of professional and social activities, but their generic meeting environments cannot be dynamically adapted to align with distributed collaborators’ needs. To enable end-user customization, we developed BlendScape, a rendering and composition system for video-conferencing participants to tailor environments to their meeting context by leveraging AI image generation techniques. BlendScape supports flexible representations of task spaces by blending users’ physical or digital backgrounds into unified environments and implements multimodal interaction techniques to steer the generation.https://nels.dev/updates/2024-09-23/Mon, 23 Sep 2024 00:00:00 +0000https://nels.dev/updates/2024-09-23/BlendScape received an Honorable Mention award at ACM UIST ‘24 (top 1.5% of 608 submissions)!https://nels.dev/updates/2024-08-02/Fri, 02 Aug 2024 00:00:00 +0000https://nels.dev/updates/2024-08-02/SpaceBlender and BlendScape (both co-first authored) were accepted to ACM UIST ‘24. Two corresponding patent applications were filed!https://nels.dev/updates/2024-06-25/Tue, 25 Jun 2024 00:00:00 +0000https://nels.dev/updates/2024-06-25/Supported by the In2Research program, we are excited to welcome our interns, Rowan Meng and Aiman Sohail, to the VECG lab! Rowan and Aiman will be working on projects related to generative AI and virtual reality, based on Ubiq-Genie.https://nels.dev/updates/2024-06-14/Fri, 14 Jun 2024 00:00:00 +0000https://nels.dev/updates/2024-06-14/Finished my internship at Niantic Labs. It was wonderful to work with Jessica van Brummelen and Gabriel Brostow – looking forward to continuing our collaboration and sharing our work soon!https://nels.dev/updates/2024-03-27/Wed, 27 Mar 2024 00:00:00 +0000https://nels.dev/updates/2024-03-27/Our paper on DreamCodeVR and two posters (AIsop and StreamSpace) were presented at IEEE VR ‘24 (led by Daniele Giunchi). I had a great time as a student volunteer at the conference!https://nels.dev/publication/ieeevr2024-dreamcodevr/Sat, 16 Mar 2024 01:00:00 +0100https://nels.dev/publication/ieeevr2024-dreamcodevr/ABSTRACT Virtual Reality (VR) has revolutionized how we interact with digital worlds. However, programming for VR remains a complex and challenging task, requiring specialized skills and knowledge. Powered by large language models (LLMs), DreamCodeVR is designed to assist users, irrespective of their coding skills, in crafting basic object behavior in VR environments by translating spoken language into code within an active application. This approach seeks to simplify the process of defining behaviors and visual changes through speech.https://nels.dev/publication/ieeevr2024-streamspace/Sat, 16 Mar 2024 01:00:00 +0100https://nels.dev/publication/ieeevr2024-streamspace/ABSTRACT We introduce StreamSpace as a framework for exploring screen-based collaborative mixed-reality (MR) experiences, focusing on the streaming, integration, and layout of screen content in MR environments. Utilizing Unity and Ubiq, this framework allows users to engage with, reposition, and resize uniquely identified screens within a user-centric virtual space. Building on Ubiq’s WebRTC capabilities, our framework enables real-time streaming and transformations through peer-to-peer communication. Key features of StreamSpace include distributed streaming, automated screen layout, and flexible privacy settings for virtual screens.https://nels.dev/publication/ieeevr2024-aisop/Fri, 15 Mar 2024 01:00:00 +0100https://nels.dev/publication/ieeevr2024-aisop/ABSTRACT We introduce AIsop, a system that autonomously generates VR storytelling experiences using generative artificial intelligence. AIsop crafts unique stories by leveraging state-of-the-art large language models (LLMs) and employs text-to-speech technology for narration. Further enriching the experience, a visual representation of the narrative is produced through a pipeline that pairs LLM-generated prompts with diffusion models, rendering visuals for clusters of sentences in the story. Our evaluation encompasses two distinct use cases: the narration of pre-existing content and the generation of entirely new narratives.https://nels.dev/updates/2023-10-09/Mon, 09 Oct 2023 00:00:00 +0000https://nels.dev/updates/2023-10-09/Our paper on extending Ubiq to the web was presented at ACM Web3D ‘23 by Sebastian Friston.https://nels.dev/updates/2023-10-13/Mon, 09 Oct 2023 00:00:00 +0000https://nels.dev/updates/2023-10-13/Our student competition entry on Reviving the Euston Arch: A Mixed Reality Approach to Cultural Heritage Tours was awarded an Honorable Mention for the Best Design Award at the ISMAR 2023 student competition.https://nels.dev/publication/web3d2023/Mon, 09 Oct 2023 00:00:00 +0100https://nels.dev/publication/web3d2023/ABSTRACT Social VR (SVR) systems are VR systems with a common subset of features facilitating unstructured social interaction. In the real world, social situations have many purposes, each with a different set of requirements, and roles its participants take - creator, moderator, performer, visitor, etc. Yet, common SVR systems typically offer only a single client to users. Even if there are versions for different platforms, there is a one-size-fits-all approach to the user experience.https://nels.dev/publication/ismar2023/Mon, 09 Oct 2023 00:00:00 +0100https://nels.dev/publication/ismar2023/ABSTRACT Augmented Reality (AR) and Virtual Reality (VR) users have distinct capabilities and experiences during collaborations: while AR users benefit from real-time contextual information due to physical presence, VR users enjoy the flexibility to transition between locations rapidly, unconstrained by physical space. Our research aims to utilize these spatial differences to facilitate engaging, shared MR experiences. Using Google Geospatial Creator, we enable large-scale outdoor authoring and precise localization to create a unified environment.https://nels.dev/updates/2023-09-15/Fri, 15 Sep 2023 00:00:00 +0000https://nels.dev/updates/2023-09-15/Completed my internship at the EPIC Group at Microsoft Research in Redmond. It was a fantastic experience working with Bala Kumaravel, Nicolai Marquardt, and Andy Wilson – looking forward to sharing our work soon!https://nels.dev/updates/2023-03-27/Mon, 27 Mar 2023 00:00:00 +0000https://nels.dev/updates/2023-03-27/Presented our workshop paper on Ubiq-Genie at the OAT workshop, while Ziwen Lu presented another workshop paper on our outdoor collaborative MR prototype at the ReDigiTS workshop during IEEE VR ‘23. Presenting my ongoing research at the Doctoral Consortium was also a valuable experience!https://nels.dev/updates/2023-03-15/Wed, 15 Mar 2023 00:00:00 +0000https://nels.dev/updates/2023-03-15/Our paper on immersive competence and literacy was published in Frontiers in Virtual Reality, led by my advisor, Anthony Steed.https://nels.dev/publication/oat2023/Wed, 01 Mar 2023 03:00:00 +0100https://nels.dev/publication/oat2023/ABSTRACT This paper describes the Ubiq-Genie framework for integrating external frameworks with the Ubiq social VR platform. The proposed architecture is modular, allowing for easy integration of services and providing mechanisms to offload computationally intensive processes to a server. To showcase the capabilities of the framework, we present two prototype applications: 1) a voice- and gesture-controlled texture generation method based on Stable Diffusion 2.0 and 2) an embodied conversational agent based on ChatGPT.https://nels.dev/publication/redigits2023/Wed, 01 Mar 2023 02:00:00 +0100https://nels.dev/publication/redigits2023/ABSTRACT Most research on collaborative mixed reality (CMR) has focused on indoor spaces. In this paper, we present our ongoing work aimed at investigating the potential of CMR in outdoor spaces. These spaces present unique challenges due to their larger and more complex nature, particularly in terms of reconstruction, tracking, and interaction. Our prototype system utilises a photorealistic model to facilitate collaboration between remote virtual reality (VR) users and a local augmented reality (AR) user.https://nels.dev/publication/ieeevr2023/Wed, 01 Mar 2023 01:00:00 +0100https://nels.dev/publication/ieeevr2023/ABSTRACT Mixed reality (MR) has the potential to transform the way we communicate and collaborate. However, there is a lack of knowledge about the collaborative use of different MR devices, such as collaboration between local augmented reality (AR) and remote virtual reality (VR) users. This position paper outlines research focused on deepening our understanding of the impact of asymmetry on collaboration and to use this knowledge to mitigate its negative effects and leverage its positive potential, leading to improved collaboration in MR environments.https://nels.dev/publication/frvir2023/Wed, 01 Mar 2023 00:00:00 +0100https://nels.dev/publication/frvir2023/ABSTRACT While immersive experiences are still a relatively immature medium, there are millions of consumer devices in use. The level of awareness of the forms of the interface and media will vary enormously across the potential audience. Users might own personal devices or might encounter immersive systems in various venues. We introduce the term immersive competence to refer to the general practical knowledge and skills that users accumulate about how typical immersive interfaces work – the ways in which buttons are used, main locomotion techniques, etc.https://nels.dev/updates/2022-12-15/Thu, 15 Dec 2022 00:00:00 +0000https://nels.dev/updates/2022-12-15/Presented my paper exploring user behavior in asymmetric collaborative mixed reality at VRST ‘22 in Tsukuba, Japan, written with my advisor Anthony Steed.https://nels.dev/publication/vrst2022/Fri, 14 Oct 2022 00:00:00 +0100https://nels.dev/publication/vrst2022/ABSTRACT A common issue for collaborative mixed reality is the asymmetry of interaction with the shared virtual environment. For example, an augmented reality (AR) user might use one type of head-mounted display (HMD) in a physical environment, while a virtual reality (VR) user might wear a different type of HMD and see a virtual model of that physical environment. To explore the effects of such asymmetric interfaces on collaboration we present a study that investigates the behaviour of dyads performing a word puzzle task where one uses AR and the other VR.https://nels.dev/updates/2022-10-12/Wed, 12 Oct 2022 00:00:00 +0000https://nels.dev/updates/2022-10-12/Our paper on our toolkit for running remote mixed reality experiments (Ubiq-Exp) was published in Frontiers in Virtual Reality, led by my advisor, Anthony Steed.https://nels.dev/updates/2022-09-12/Mon, 12 Sep 2022 00:00:00 +0000https://nels.dev/updates/2022-09-12/Had lots of fun at the Niantic Lightship VPS Hackathon in London! Our project, developed with Ziwen Lu, Kalila Shapiro, and Joe McFadden, won 2nd place!https://nels.dev/publication/frvir2022/Thu, 25 Aug 2022 00:00:00 +0100https://nels.dev/publication/frvir2022/ABSTRACT Developing mixed-reality (MR) experiments is a challenge as there is a wide variety of functionality to support. This challenge is exacerbated if the MR experiment is multi-user or if the experiment needs to be run out of the lab. We present Ubiq-Exp - a set of tools that provide a variety of functionality to facilitate distributed and remote MR experiments. We motivate our design patterns and tools from recent practice in the field and a desire to build experiments that are easier to reproduce.https://nels.dev/cv/Wed, 23 Feb 2022 14:46:38 +0000https://nels.dev/cv/Education Jul. 2021 — Mar. 2026London, UK Ph.D. Computer ScienceUniversity College London Research group: Virtual Environments and Computer Graphics (VECG). Thesis: “Constructing Shared Space: Designing for Asymmetry in Collaborative Mixed Reality”. Supervisors: Prof. Anthony Steed (primary) and Prof. Simon Julier (secondary). Passed viva with no corrections. Feb. 2018 — Oct. 2020Delft, Netherlands M.Sc. Computer ScienceDelft University of Technology Track: Data Science & Technology; Thesis: “Generative RGB-D Face Completion for Head-Mounted Display Removal”; Supervisors: Prof.https://nels.dev/updates/2021-09-25/Sun, 12 Sep 2021 00:00:00 +0000https://nels.dev/updates/2021-09-25/Moved to London to start my PhD at University College London with Anthony Steed and Simon Julier! I’m grateful to receive funding through the European Union’s Horizon 2020 Research and Innovation program under grant agreement No. 739578.https://nels.dev/updates/2021-03-27/Sat, 27 Mar 2021 00:00:00 +0000https://nels.dev/updates/2021-03-27/Presented a workshop paper on generative head-mounted display removal at the VHCIE workshop during IEEE VR ‘21. Also grateful for the opportunity to present my work at internal events at Microsoft and IBM!https://nels.dev/publication/vhcie2021/Sat, 27 Mar 2021 00:00:00 +0100https://nels.dev/publication/vhcie2021/ABSTRACT Head-mounted displays (HMDs) are an essential display device for the observation of virtual reality (VR) environments. However, HMDs obstruct external capturing methods from recording the user’s upper face. This severely impacts social VR applications, such as teleconferencing, which commonly rely on external RGB-D sensors to capture a volumetric representation of the user. In this paper, we introduce an HMD removal framework based on generative adversarial networks (GANs), capable of jointly filling in missing color and depth data in RGB-D face images.https://nels.dev/updates/2020-08-27/Thu, 27 Aug 2020 00:00:00 +0000https://nels.dev/updates/2020-08-27/Completed my research internship at the Intelligent Imaging group within the Netherlands Organization for Applied Scientific Research (TNO) in The Hague, where I worked on my master’s thesis research on generative head-mounted display removal with Frank ter Haar.https://nels.dev/publication/shs2019/Tue, 26 Mar 2019 00:00:00 +0100https://nels.dev/publication/shs2019/ABSTRACT Superhuman sports (SHS) is a field where technological augmentations of human abilities and environment are combined to play a new and exciting sport. SHS make use of artificial senses or virtual reality to create a new experience that involves physical fitness and skills. Star Tag aims to combine these aspects with an engaging audience experience. This augmented reality game uses the Microsoft HoloLens, making it possible to move through a mixed reality space effectively.https://nels.dev/publication/Mon, 01 Jan 0001 00:00:00 +0000https://nels.dev/publication/publications