Overview of Working in Extended Reality
This episode explores how virtual reality (VR), augmented reality (AR), and extended reality (XR) are evolving from experimental technologies into practical computing platforms. Host Kyle Polich speaks with Doug Bowman, a Virginia Tech computer science professor and XR/HCI researcher, about the history of VR, why interface design matters more than hardware alone, and how AI may shape the next generation of immersive experiences.
The central idea: the biggest challenge in XR is no longer just building the headset—it’s designing interactions that feel useful, intuitive, and adaptive across contexts.
Key Takeaways
-
VR is old, but now much more accessible
- The first working head-mounted display was built in 1968 by Ivan Sutherland.
- Modern consumer hardware, especially the Oculus Rift, dramatically lowered the cost of entry and accelerated the field.
-
Hardware is improving, but AR glasses are the bigger frontier
- Current VR headsets are already quite capable for focused use.
- The major long-term hardware challenge is creating all-day wearable AR glasses with good optics, battery life, heat management, and onboard compute.
-
The real research problem is interaction design
- Simple “natural” gestures, like grabbing virtual objects with your hands, are not always the best solution.
- XR interfaces often need to be “magical” rather than purely realistic—extending human capabilities instead of merely imitating the real world.
-
AI is becoming central to XR
- Bowman sees the future of XR as a combination of XR + AI.
- AI can reduce user effort by predicting intent, assisting with selection, and adapting interfaces based on gaze, head pose, and context.
-
Privacy is a major blocker
- XR devices raise concerns for both users and bystanders.
- Without strong privacy solutions—technical, policy-based, and social—widespread adoption may stall.
Doug Bowman’s Research Perspective
The Virginia Tech HCI Center
Bowman describes Virginia Tech’s Center for Human-Computer Interaction as a large interdisciplinary group spanning 20+ departments. The center works on:
- XR and immersive systems
- User experience design
- Technology’s effects on people
- Domain-specific applications, such as:
- history
- construction
- education
- public-facing immersive tours
One example he gives is using AR for a Civil War battlefield tour, helping users understand historical events in place.
Off-the-shelf Hardware, Custom Software
Bowman’s lab does not build hardware from scratch. Instead, it uses devices like:
- Meta Quest
- Valve Index
- HTC Vive
- HoloLens
- Apple Vision Pro-style systems for comparison and prototyping
Their focus is on:
- software prototypes
- interaction techniques
- user studies
- comparing experiences across devices and contexts
How XR Interaction Works
Why “Natural” Isn’t Always Best
A common assumption is that XR should mimic the real world as closely as possible. Bowman argues that this is often limiting:
- Real hands have limited reach
- Current hand tracking has limits
- Real-world metaphors do not always scale to digital environments
Instead, good XR design often blends familiar cues with new abilities. For example:
- Pointing can be used to select distant virtual objects
- 3D systems can give people abilities they don’t have in physical space
The Core Interaction Categories
Bowman highlights the standard framework used in 3D UI research:
- Selection — choosing an object in 3D space
- Manipulation — moving, rotating, or resizing objects
- Navigation — moving yourself through the environment
- System control — menus, settings, and commands
He notes that 2D UI patterns like buttons, sliders, and checkboxes still remain useful, even in XR.
XR for Everyday Work
More Than Gaming
A major point of the episode is that XR is not just for games or entertainment. One of its most promising uses is for knowledge work:
- documents
- multitasking
- virtual desktops
- flexible display space
- portable workspaces
Bowman argues that AR/VR can create nearly unlimited display space, letting users place and resize windows anywhere in their environment.
Studying Real-World Use
His lab studies XR in realistic settings by taking devices into places like:
- cafés
- libraries
- meeting rooms
- classrooms
Participants use the tools for their own work, which makes the research more ecologically valid than lab-only tasks.
HoloLens, Vision Pro, and the Future of AR
Bowman compares the HoloLens 2 and Apple Vision Pro:
- HoloLens: optical see-through, gesture-heavy interaction
- Vision Pro: passthrough video, gaze-centric interaction
His view:
- These devices are interesting but not yet the final form
- The biggest leap will come with true eyeglass-style AR
- The future matters less as a single device and more as a system that works across contexts
He’s especially interested in what happens when someone moves between contexts, such as:
- office → walking between buildings
- classroom → café
- home → airplane
Privacy and Social Norms
Bowman emphasizes that XR privacy is not just a technical issue.
Why Privacy Matters
XR devices may capture:
- personal information
- bystander data
- private environments
- unintended recordings
If that problem isn’t solved, the market may fail before it matures.
Possible Solutions
He describes a privacy-assistant concept that:
- detects sensitive information
- blurs or obscures it automatically
- or asks the user to confirm before sharing
User preference was split:
- some prefer automation
- others want manual confirmation
He also stresses the need for:
- policy solutions
- evolving social conventions
- norms similar to how society adapted to smartwatches in meetings
Research Methods: Quantitative and Qualitative
Bowman explains that XR research uses both forms of evaluation:
Quantitative methods
Used when the question is about:
- speed
- errors
- precision
- performance differences
- statistical effects
Qualitative methods
Used when the question is about:
- comfort
- social acceptability
- lived experience
- usefulness in context
- how people feel using the system in public
For his work on virtual displays in public spaces, interviews and observation were often more valuable than performance metrics.
Where the Field Is Going
Bowman believes the next frontier is adaptive, intelligent XR interfaces.
What “Adaptive” Means
A strong XR system should automatically adjust based on:
- location
- task
- context
- user intent
- environment
- whether the user is standing, walking, commuting, or working
He warns that bad adaptive systems can fail like Microsoft Clippy—trying to be helpful but becoming annoying.
The Goal
The ideal system would provide:
- the right help
- at the right time
- in the right format
- without requiring constant manual settings changes
Advice for Students Interested in HCI/XR
Bowman says there are many entry points into the field:
- Computer science: the most common route
- Psychology: especially perception, cognition, and neuroscience
- Design / visual arts: important for user experience and interfaces
- Industrial design: especially for hardware-oriented work
His advice for students:
- take HCI courses
- do undergraduate research
- learn by working on boundary-pushing projects
Closing Thoughts
Bowman’s main message is that XR is moving toward becoming a general-purpose computing platform. Just as computers became smaller, more personal, and more portable over time, XR devices may eventually become:
- wearable all day
- context-aware
- socially acceptable
- powered by AI
- capable of replacing or extending traditional screens
He sees that future as both technically challenging and highly promising.
Follow-Up Resources
- Doug Bowman: LinkedIn
- Virginia Tech Center for Human-Computer Interaction: organizational LinkedIn page
- Related talks and seminars: available through the Center for HCI’s online programming