XRUI and in-world widgets

overview xrui (extended reality user interface) and in world widgets provide the foundation for creating user interfaces that exist within the 3d environment rather than as traditional 2d overlays by rendering ui elements directly in the 3d scene, the system enables more immersive and spatially relevant interactions this chapter explores the implementation, components, and workflow of the xrui system within the ir engine client, demonstrating how web technologies like react are leveraged to create interactive interfaces in 3d space core concepts extended reality user interfaces traditional user interfaces in applications are typically rendered as 2d overlays on top of the viewport in contrast, xrui renders ui elements as part of the 3d scene positions interfaces in 3d space allows ui to be attached to specific objects or locations enables interaction through 3d input methods (vr controllers, gaze, etc ) this approach provides several benefits increased immersion by keeping users in the 3d environment spatial context for information and controls natural interaction with ui elements in the virtual space consistent visual language between the world and its interfaces in world widgets widgets are self contained ui elements that serve specific purposes within the 3d environment nameplates display user information above avatars control panels provide interactive controls for objects menus offer navigation and selection options information displays show contextual data in the environment notifications present alerts and updates in 3d space these widgets can be attached to objects, positioned relative to the user, or placed at fixed locations in the world widget management the ir engine client includes a comprehensive system for managing widgets registration define and register widget types visibility control show or hide widgets as needed state management track widget states (open/closed, enabled/disabled) positioning place widgets in appropriate locations interaction handle user input with widget elements this management system ensures consistent behavior and appearance across different widget types implementation xrui component architecture the xrui system is built on several key components // simplified architecture of xrui components interface xruisystem { xruicomponent ecscomponent; // attaches ui to entities weblayer3d renderer; // renders web content to textures createxrui function; // creates xrui instances interactionsystem system; // handles input with ui elements } this architecture uses the entity component system (ecs) pattern integrates web rendering with 3d graphics provides a bridge between react components and 3d entities handles input mapping between 3d and 2d spaces creating xrui elements the process of creating an xrui element involves several steps // simplified from various xrui creation functions import { createxrui } from '@ir engine/engine/src/xrui/createxrui'; import { setcomponent, getcomponent } from '@ir engine/ecs'; import { transformcomponent } from '@ir engine/spatial'; function createnameplatexrui(username) { // 1 define the react component for the ui const nameplateui = ({ username }) => ( \<div classname="nameplate"> {username} \</div> ); // 2 create the xrui instance with the react component const ui = createxrui(nameplateui, { username }); // 3 position the ui in 3d space const transform = getcomponent(ui entity, transformcomponent); transform position set(0, 0 2, 0); // position above target // 4 configure additional properties setcomponent(ui entity, visiblecomponent); return ui; } this function defines or imports a react component for the ui appearance creates an xrui instance using the createxrui function positions the resulting entity in 3d space configures additional components as needed returns the xrui instance for further use widget registration widgets are registered with the widget management system // simplified from src/systems/widgets ts import { widgets } from ' /widgets'; import { widgetappactions } from ' /widgetappservice'; import { dispatchaction } from '@ir engine/hyperflux'; import { createxrui } from '@ir engine/engine/src/xrui/createxrui'; function registercontrolpanelwidget() { // 1 create the xrui for the widget const controlpanelui = () => ( \<div classname="control panel"> \<h3>control panel\</h3> \<button onclick={() => console log('button clicked')}> toggle power \</button> \<div classname="slider container"> \<label>volume\</label> \<input type="range" min="0" max="100" defaultvalue="50" /> \</div> \</div> ); const ui = createxrui(controlpanelui); // 2 define the widget configuration const controlpanelwidget = { ui ui, label 'control panel', icon controlpanelicon, onopen () => console log('control panel opened'), onclose () => console log('control panel closed') }; // 3 register the widget with the system const widgetid = widgets registerwidget(ui entity, controlpanelwidget); return widgetid; } this function creates an xrui instance for the widget's visual representation defines a configuration object with properties like label, icon, and event handlers registers the widget with the widget system, receiving a unique id returns the widget id for later reference widget visibility control widgets can be shown or hidden programmatically // simplified widget visibility control import { widgetappservice } from ' /widgetappservice'; import { widgetappactions } from ' /widgetappservice'; import { dispatchaction } from '@ir engine/hyperflux'; // method 1 using the service directly function togglewidgetvisibility(widgetid, visible) { widgetappservice setwidgetvisibility(widgetid, visible); } // method 2 using hyperflux actions function showwidget(widgetid) { dispatchaction(widgetappactions showwidget({ id widgetid, shown true })); } function hidewidget(widgetid) { dispatchaction(widgetappactions showwidget({ id widgetid, shown false })); } these functions provide different methods to control widget visibility use either direct service calls or hyperflux actions allow widgets to be shown or hidden based on application state nameplate example a common use case for xrui is displaying nameplates above avatars // simplified nameplate implementation import { createxrui } from '@ir engine/engine/src/xrui/createxrui'; import { setcomponent, getcomponent } from '@ir engine/ecs'; import { transformcomponent } from '@ir engine/spatial'; import { entitytreecomponent } from '@ir engine/ecs'; function addnameplatetoavatar(avatarentity, username) { // 1 create the nameplate ui component const nameplateui = ({ username }) => ( \<div classname="nameplate"> \<div classname="username">{username}\</div> \<div classname="status indicator online">\</div> \</div> ); // 2 create the xrui instance const ui = createxrui(nameplateui, { username }); // 3 position it above the avatar const transform = getcomponent(ui entity, transformcomponent); transform position set(0, 1 8, 0); // 1 8 units above avatar base // 4 make it a child of the avatar entity so it moves with the avatar setcomponent(ui entity, entitytreecomponent, { parententity avatarentity }); return ui entity; } this function defines a react component for the nameplate appearance creates an xrui instance with the username as a prop positions the nameplate above the avatar makes the nameplate a child of the avatar entity so it follows the avatar's movement returns the nameplate entity for further reference technical implementation rendering web content in 3d the process of rendering web content in 3d involves several technical steps sequencediagram participant react as react component participant xrui as xrui system participant weblayer as weblayer3d participant texture as texture participant mesh as 3d mesh participant renderer as 3d renderer react >>xrui define ui appearance and behavior xrui >>weblayer initialize with react component weblayer >>weblayer render react to offscreen dom weblayer >>texture capture dom as texture xrui >>mesh create plane mesh xrui >>mesh apply texture to mesh mesh >>renderer add to 3d scene renderer >>renderer render in 3d environment this process takes a react component defining the ui renders it to an offscreen dom using weblayer3d captures the rendered dom as a texture creates a plane mesh in the 3d scene applies the texture to the mesh renders the mesh in the 3d environment input handling interaction with xrui elements requires mapping between 3d and 2d input sequencediagram participant user as user input participant controller as vr controller/mouse participant raycaster as 3d raycaster participant xrui as xrui component participant weblayer as weblayer3d participant react as react component user >>controller performs input action controller >>raycaster generates ray into scene raycaster >>xrui intersects with xrui plane xrui >>xrui converts 3d intersection to 2d coordinates xrui >>weblayer dispatches 2d event (click, hover) weblayer >>react triggers react event handlers react >>react updates component state react >>weblayer re renders ui weblayer >>xrui updates texture this process captures user input from controllers or mouse uses raycasting to detect intersections with xrui planes converts 3d intersection points to 2d coordinates on the ui dispatches appropriate events to the web layer triggers react event handlers in the component updates the ui based on the interaction renders the updated ui to the texture widget management system the widget management system uses hyperflux for state management // simplified from src/systems/widgetappservice ts import { definestate, defineaction } from '@ir engine/hyperflux'; // define the state structure for widgets export const widgetappstate = definestate({ name 'widgetappstate', initial { widgetsmenuopen false, widgets {} as record\<string, { enabled boolean; visible boolean; }> } }); // define actions for widget management export const widgetappactions = { registerwidget defineaction({ type 'ee client widgetapp register widget', id '' as string }), showwidget defineaction({ type 'ee client widgetapp show widget', id '' as string, shown false as boolean }), togglewidgetsmenu defineaction({ type 'ee client widgetapp toggle widgets menu' }) }; // service methods for widget management export const widgetappservice = { setwidgetvisibility (id string, visible boolean) => { dispatchaction(widgetappactions showwidget({ id, shown visible })); } }; this system defines a hyperflux state to track widget status provides actions for registering widgets and controlling visibility offers service methods for common widget operations maintains a consistent state for all widgets in the application widget types and use cases nameplates nameplates display user information above avatars // nameplate widget example function createnameplate(username, avatarentity) { const ui = createxrui(nameplateui, { username }); // position above avatar and make it a child const transform = getcomponent(ui entity, transformcomponent); transform position set(0, 1 8, 0); setcomponent(ui entity, entitytreecomponent, { parententity avatarentity }); // configure to always face the camera setcomponent(ui entity, billboardcomponent); return ui entity; } nameplates typically display usernames and status indicators follow avatar movement face the camera for readability scale based on distance from the viewer interactive panels interactive panels provide controls for objects or systems // interactive panel example function createcontrolpanel(targetobject) { const ui = createxrui(controlpanelui, { onpowertoggle () => targetobject togglepower(), onvolumechange (value) => targetobject setvolume(value) }); // position relative to the target object const transform = getcomponent(ui entity, transformcomponent); transform position copy(targetobject position); transform position y += 1 2; transform lookat(camera position); return ui entity; } interactive panels typically provide buttons, sliders, and other controls trigger actions on target objects position themselves contextually respond to user input in real time menu systems menu systems provide navigation and selection options // widget menu example function createwidgetmenu() { const ui = createxrui(widgetmenuui, { widgets registeredwidgets, onselectwidget (widgetid) => { widgetappservice setwidgetvisibility(widgetid, true); dispatchaction(widgetappactions togglewidgetsmenu()); } }); // position relative to the user const transform = getcomponent(ui entity, transformcomponent); transform position set(0, 1 4, 0 8); // in front of user // make it follow the user setcomponent(ui entity, followcameracomponent, { distance 0 8, locky false }); return ui entity; } menu systems typically display available options or widgets allow selection through various input methods position themselves for optimal visibility maintain consistent access to functionality integration with other components the xrui and widget systems integrate with several other components entity component system xrui elements are implemented as entities with components // example of xrui integration with ecs import { createentity, addcomponent } from '@ir engine/ecs'; import { xruicomponent } from '@ir engine/spatial/src/xrui/components/xruicomponent'; import { transformcomponent } from '@ir engine/spatial'; import { visiblecomponent } from '@ir engine/spatial'; function createxruientity(reactcomponent, props) { // create entity const entity = createentity(); // add transform component addcomponent(entity, transformcomponent); // add xrui component const xrui = new weblayer3d(reactcomponent, props); addcomponent(entity, xruicomponent, { xrui }); // make it visible addcomponent(entity, visiblecomponent); return { entity, xrui }; } this integration leverages the ecs architecture for consistent behavior allows xrui elements to use standard components like transform enables systems to process xrui entities like other entities provides a familiar pattern for developers working with the engine hyperflux state management widget state is managed through hyperflux // example of widget state in hyperflux import { getmutablestate, usemutablestate } from '@ir engine/hyperflux'; import { widgetappstate } from ' /widgetappservice'; // in a component that needs to check widget visibility function widgetcontroller() { const widgetstate = usemutablestate(widgetappstate); const iscontrolpanelvisible = widgetstate widgets\['control panel']? visible || false; // use the visibility state useeffect(() => { if (iscontrolpanelvisible) { // perform actions when control panel becomes visible } }, \[iscontrolpanelvisible]); // render component } // to update widget state function togglecontrolpanel() { const widgetstate = getmutablestate(widgetappstate); const currentvisibility = widgetstate widgets\['control panel']? visible || false; dispatchaction(widgetappactions showwidget({ id 'control panel', shown !currentvisibility })); } this integration uses hyperflux for reactive state management allows components to respond to widget state changes provides a consistent pattern for state updates enables widgets to be controlled from anywhere in the application avatar system xrui elements like nameplates integrate with the avatar system // example of xrui integration with avatars import { avatarcomponent } from ' /avatar/components/avatarcomponent'; import { createnameplate } from ' /createnameplate'; // system that adds nameplates to avatars function executenameplatesystem() { // query for avatars without nameplates const avatarswithoutnameplates = queryentities(\[ withcomponent(avatarcomponent), withoutcomponent(nameplatetagcomponent) ]); // add nameplates to avatars for (const entity of avatarswithoutnameplates) { const avatar = getcomponent(entity, avatarcomponent); const username = avatar username || 'unknown user'; // create nameplate createnameplate(username, entity); // mark avatar as having a nameplate addcomponent(entity, nameplatetagcomponent); } } this integration automatically adds nameplates to avatars updates nameplate content based on avatar properties ensures nameplates follow avatar movement creates a consistent visual identity for users benefits of xrui and in world widgets the xrui and widget systems provide several key advantages immersion keeps users in the 3d environment without breaking immersion spatial context places information and controls where they are most relevant familiar technologies leverages web technologies like react for ui development consistency provides a unified approach to ui across the application flexibility supports various widget types for different use cases extensibility allows new widget types to be easily created and integrated performance optimizes rendering through texture based approaches these benefits make xrui and in world widgets essential components for creating immersive and interactive 3d experiences next steps with an understanding of how user interfaces are created in 3d space, the next chapter explores how users are represented within the virtual environment through avatars next avatar management and customization docid\ b7 03bx2y8v1hohrsvr p