Visual scripting system

overview the visual scripting system is a powerful component of the ir engine's world editor that enables the creation of interactive behaviors without traditional programming it implements a node based interface where users connect visual elements representing actions, events, and data to define logic flows by providing an intuitive graphical alternative to text based coding, this system makes behavior creation accessible to users with varying technical backgrounds this chapter explores the implementation, workflow, and capabilities of the visual scripting system within the world editor core concepts node based programming the visual scripting system is built on a node based programming paradigm nodes visual blocks representing specific functions or operations edges connections between nodes that define data and execution flow sockets input and output points on nodes where connections attach graphs complete networks of connected nodes that define a behavior variables named data storage that can be accessed across a graph this paradigm provides an intuitive visual representation of program logic node types the system includes several categories of nodes for different purposes event nodes trigger execution based on specific conditions or inputs action nodes perform operations that affect entities or the environment logic nodes control the flow of execution with conditions and branches data nodes provide, manipulate, or store data values entity nodes reference and interact with scene objects math nodes perform mathematical calculations and operations custom nodes user defined functionality for specific requirements these node types provide a comprehensive toolkit for behavior creation script execution the script execution system processes visual scripts at runtime event detection monitors for conditions that trigger event nodes flow traversal follows execution paths through connected nodes data propagation passes values between nodes along data connections action invocation calls engine functions based on node configurations state management maintains script variables and execution context this execution system translates visual graphs into runtime behaviors implementation visual script panel the visual script panel provides the main interface for script creation // simplified from src/panels/visualscript/index tsx import react from 'react'; import { usehookstate } from '@hookstate/core'; import { paneldragcontainer, paneltitle } from '@ir engine/ui/src/components/editor/layout/panel'; import { tabdata } from 'rc dock'; import { usetranslation } from 'react i18next'; import { visualscriptstate } from ' / /services/visualscriptstate'; import { visualscriptflow } from ' /flow'; import { visualscriptsidepanel } from ' /sidepanel'; / visual script panel content component @returns visual script panel component / const visualscriptpanel react fc = () => { const { t } = usetranslation(); const visualscriptstate = usehookstate(visualscriptstate state); const currentscriptid = visualscriptstate currentscriptid value; // if no script is selected, show empty state if (!currentscriptid) { return ( \<div classname="visual script panel empty"> \<p>{t('editor\ visualscript noscriptselected')}\</p> \<button onclick={() => visualscriptstate createnewscript()}> {t('editor\ visualscript createnew')} \</button> \</div> ); } return ( \<div classname="visual script panel"> \<div classname="visual script panel content"> \<visualscriptflow scriptid={currentscriptid} /> \<visualscriptsidepanel scriptid={currentscriptid} /> \</div> \</div> ); }; / visual script panel title component @returns visual script panel title component / const visualscriptpaneltitle react fc = () => { const { t } = usetranslation(); return ( \<paneldragcontainer> \<paneltitle>{t('editor\ visualscript title')}\</paneltitle> \</paneldragcontainer> ); }; / visual script panel tab configuration / export const visualscriptpaneltab tabdata = { id 'visualscriptpanel', title \<visualscriptpaneltitle />, content \<visualscriptpanel />, closable false }; this component provides the main interface for creating and editing visual scripts shows an empty state when no script is selected renders the flow editor for node manipulation includes a side panel for script properties and variables integrates with the editor's tab system flow editor the flow editor manages the canvas where nodes are placed and connected // simplified from src/panels/visualscript/flow\ tsx import react, { usecallback, useeffect, usestate } from 'react'; import reactflow, { background, controls, minimap, addedge, usenodesstate, useedgesstate } from 'reactflow'; import { usehookstate } from '@hookstate/core'; import { visualscriptstate } from ' / /services/visualscriptstate'; import { nodetypes } from ' /node/types'; import { usenodespecgenerator } from ' /hooks/usenodespecgenerator'; import { visualtoflow } from ' /transformers/visualtoflow'; import { flowtovisual } from ' /transformers/flowtovisual'; / visual script flow editor component @param props component properties @returns visual script flow editor component / export const visualscriptflow react fc<{ scriptid string; }> = ({ scriptid }) => { // get script data const visualscriptstate = usehookstate(visualscriptstate state); const script = visualscriptstate scripts\[scriptid] value; // node and edge state const \[nodes, setnodes, onnodeschange] = usenodesstate(\[]); const \[edges, setedges, onedgeschange] = useedgesstate(\[]); // node specifications const nodespecs = usenodespecgenerator(); // initialize flow from script data useeffect(() => { if (script && script graph) { const { nodes flownodes, edges flowedges } = visualtoflow(script graph, nodespecs); setnodes(flownodes); setedges(flowedges); } }, \[script, nodespecs]); // handle edge connections const onconnect = usecallback((params) => { setedges((eds) => addedge(params, eds)); }, \[setedges]); // save changes to script const onsave = usecallback(() => { const graph = flowtovisual(nodes, edges); visualscriptstate updatescriptgraph(scriptid, graph); }, \[scriptid, nodes, edges]); // auto save on changes useeffect(() => { const timer = settimeout(onsave, 1000); return () => cleartimeout(timer); }, \[nodes, edges, onsave]); return ( \<div classname="visual script flow"> \<reactflow nodes={nodes} edges={edges} onnodeschange={onnodeschange} onedgeschange={onedgeschange} onconnect={onconnect} nodetypes={nodetypes} fitview \> \<background /> \<controls /> \<minimap /> \</reactflow> \</div> ); }; this component uses reactflow for the node based interface converts between visual script data and flow representation handles node and edge changes provides background, controls, and minimap for navigation auto saves changes to the script state node component the node component renders individual nodes in the flow editor // simplified from src/panels/visualscript/node/index tsx import react, { memo } from 'react'; import { handle, position, nodeprops } from 'reactflow'; import { nodespecjson } from ' /types'; / visual script node component @param props component properties @returns visual script node component / export const visualscriptnode react fc\<nodeprops<{ spec nodespecjson; data record\<string, any>; }>> = memo(({ id, data, selected }) => { const { spec, data nodedata } = data; return ( \<div classname={`visual script node ${spec category} ${selected ? 'selected' ''}`}> \<div classname="node header"> \<div classname="node title">{spec label}\</div> \</div> \<div classname="node content"> {/ input sockets /} {spec inputs map((input) => ( \<div key={input id} classname="node socket input socket"> \<handle type="target" position={position left} id={input id} classname={`socket ${input type}`} /> \<div classname="socket label">{input label}\</div> {input showcontrol && renderinputcontrol(input, nodedata\[input id], id)} \</div> ))} {/ output sockets /} {spec outputs map((output) => ( \<div key={output id} classname="node socket output socket"> \<div classname="socket label">{output label}\</div> \<handle type="source" position={position right} id={output id} classname={`socket ${output type}`} /> \</div> ))} \</div> \</div> ); }); / renders an input control based on type @param input input specification @param value current value @param nodeid node id @returns input control component / const renderinputcontrol = (input, value, nodeid) => { switch (input controltype) { case 'number' return ( \<input type="number" value={value || 0} onchange={(e) => updatenodedata(nodeid, input id, parsefloat(e target value))} /> ); case 'string' return ( \<input type="text" value={value || ''} onchange={(e) => updatenodedata(nodeid, input id, e target value)} /> ); case 'boolean' return ( \<input type="checkbox" checked={value || false} onchange={(e) => updatenodedata(nodeid, input id, e target checked)} /> ); // additional control types default return null; } }; this component renders a node with header and content sections creates input and output sockets based on node specification provides appropriate controls for editable inputs handles value changes and updates node data applies styling based on node category and selection state node specification generator the node specification generator defines available node types // simplified from src/panels/visualscript/hooks/usenodespecgenerator ts import { usecallback } from 'react'; import { nodespecjson, sockettype } from ' /types'; / hook for generating node specifications @returns node specification generator / export const usenodespecgenerator = () => { / generates specifications for all available node types @returns map of node specifications by type / const generatenodespecs = usecallback(() => { const specs = new map\<string, nodespecjson>(); // event nodes specs set('onstart', { type 'onstart', category 'event', label 'on start', description 'triggered when the scene starts', inputs \[], outputs \[ { id 'flow', label 'flow', type sockettype flow } ] }); specs set('onclick', { type 'onclick', category 'event', label 'on click', description 'triggered when the entity is clicked', inputs \[], outputs \[ { id 'flow', label 'flow', type sockettype flow }, { id 'entity', label 'entity', type sockettype entity } ] }); // action nodes specs set('playanimation', { type 'playanimation', category 'action', label 'play animation', description 'plays an animation on an entity', inputs \[ { id 'flow', label 'flow', type sockettype flow }, { id 'entity', label 'entity', type sockettype entity, defaultvalue 'self' }, { id 'animationname', label 'animation', type sockettype string, controltype 'string', showcontrol true } ], outputs \[ { id 'flow', label 'complete', type sockettype flow } ] }); // logic nodes specs set('branch', { type 'branch', category 'logic', label 'branch (if/else)', description 'branches flow based on a condition', inputs \[ { id 'flow', label 'flow', type sockettype flow }, { id 'condition', label 'condition', type sockettype boolean } ], outputs \[ { id 'true', label 'true', type sockettype flow }, { id 'false', label 'false', type sockettype flow } ] }); // data nodes specs set('number', { type 'number', category 'data', label 'number', description 'provides a number value', inputs \[ { id 'value', label 'value', type sockettype number, controltype 'number', showcontrol true, defaultvalue 0 } ], outputs \[ { id 'value', label 'value', type sockettype number } ] }); // additional node specifications return specs; }, \[]); return generatenodespecs(); }; this hook defines specifications for all available node types organizes nodes into categories (event, action, logic, data) specifies inputs and outputs with their types and properties provides default values and control types for editable inputs includes descriptions for documentation and tooltips script state management the script state service manages visual script data // simplified from src/services/visualscriptstate ts import { definestate, getmutablestate } from '@ir engine/hyperflux'; import { v4 as uuidv4 } from 'uuid'; import { graphjson } from ' /panels/visualscript/types'; / state management for visual scripts / export const visualscriptstate = definestate({ name 'visualscriptstate', // initial state initial () => ({ scripts {} as record\<string, { id string; name string; description string; graph graphjson; variables record\<string, any>; }>, currentscriptid null as string | null }), // create a new script createnewscript () => { const state = getmutablestate(visualscriptstate); const id = uuidv4(); // create empty script state scripts\[id] set({ id, name 'new script', description '', graph { nodes \[], edges \[] }, variables {} }); // set as current script state currentscriptid set(id); return id; }, // update script graph updatescriptgraph (scriptid string, graph graphjson) => { const state = getmutablestate(visualscriptstate); if (state scripts\[scriptid] value) { state scripts\[scriptid] graph set(graph); } }, // set current script setcurrentscript (scriptid string) => { const state = getmutablestate(visualscriptstate); state currentscriptid set(scriptid); }, // get script by id getscriptbyid (scriptid string) => { const state = getmutablestate(visualscriptstate); return state scripts\[scriptid] value; }, // delete script deletescript (scriptid string) => { const state = getmutablestate(visualscriptstate); // remove script state scripts\[scriptid] set(undefined); // clear current script if it was the deleted one if (state currentscriptid value === scriptid) { state currentscriptid set(null); } } }); this service defines a state structure for storing visual scripts provides methods for creating, updating, and deleting scripts manages the currently selected script stores script metadata, graph structure, and variables uses hyperflux for reactive state management script execution the script execution system runs visual scripts at runtime // simplified from src/runtime/visualscriptrunner ts import { entity, getcomponent, addcomponent } from '@ir engine/ecs'; import { visualscriptstate } from ' /services/visualscriptstate'; import { nodeexecutors } from ' /nodeexecutors'; / runs a visual script on an entity @param entity entity to run the script on @param scriptid id of the script to run @param eventtype type of event that triggered the script @param eventdata additional event data / export const runvisualscript = ( entity entity, scriptid string, eventtype string, eventdata any = {} ) => { // get the script const script = visualscriptstate getscriptbyid(scriptid); if (!script) return; // create execution context const context = { entity, variables { script variables }, eventdata }; // find event nodes matching the event type const eventnodes = script graph nodes filter(node => node type === eventtype ); // execute each matching event node for (const eventnode of eventnodes) { executenode(eventnode id, script graph, context); } }; / executes a node and follows the execution flow @param nodeid id of the node to execute @param graph script graph @param context execution context / const executenode = (nodeid string, graph any, context any) => { // find the node const node = graph nodes find(n => n id === nodeid); if (!node) return; // get the node executor const executor = nodeexecutors\[node type]; if (!executor) return; // execute the node const outputs = executor(node data, context); // follow execution flow if (outputs && outputs flow) { // find edges connected to the flow output const edges = graph edges filter(edge => edge source === nodeid && edge sourcehandle === 'flow' ); // execute connected nodes for (const edge of edges) { executenode(edge target, graph, context); } } }; this system retrieves script data from the script state creates an execution context with entity and variable information finds event nodes that match the triggering event executes nodes and follows the execution flow passes data between nodes according to connections script workflow the complete visual scripting workflow follows this sequence sequencediagram participant user participant visualscriptpanel participant nodespecgenerator participant floweditor participant visualscriptstate participant scriptrunner user >>visualscriptpanel opens visual script panel visualscriptpanel >>visualscriptstate creates or loads script visualscriptstate >>visualscriptpanel returns script data visualscriptpanel >>nodespecgenerator requests node specifications nodespecgenerator >>visualscriptpanel returns available node types visualscriptpanel >>floweditor initializes with script and nodes user >>floweditor adds event node (e g , "on click") user >>floweditor adds action node (e g , "play animation") user >>floweditor connects nodes user >>floweditor configures node properties floweditor >>visualscriptstate saves updated script graph visualscriptstate >>user script saved confirmation user >>scriptrunner runs scene with script scriptrunner >>visualscriptstate loads script data scriptrunner >>scriptrunner waits for event (e g , click) scriptrunner >>scriptrunner executes nodes in sequence scriptrunner >>user action performed (animation plays) this diagram illustrates the user creates or loads a visual script the panel initializes with available node types the user builds the script by adding and connecting nodes the script is saved to the script state at runtime, the script runner executes the nodes when events occur integration with other components the visual scripting system integrates with several other components of the world editor entity component system visual scripts interact with the ecs to manipulate entities // example of ecs integration import { entity, getcomponent, addcomponent, removecomponent } from '@ir engine/ecs'; / node executor for the "add component" action @param data node data @param context execution context @returns node outputs / export const executeaddcomponentnode = (data any, context any) => { const { entity } = context; const { componenttype, componentdata } = data; // add component to entity addcomponent(entity, componenttype, componentdata); // return flow output return { flow true }; }; / node executor for the "get component property" data node @param data node data @param context execution context @returns node outputs / export const executegetcomponentpropertynode = (data any, context any) => { const { entity } = context; const { componenttype, propertypath } = data; // get component const component = getcomponent(entity, componenttype); if (!component) return { value null }; // get property value const value = getpropertybypath(component, propertypath); // return value output return { value }; }; this integration allows scripts to add, remove, and modify entity components provides access to component properties for logic and calculations enables scripts to query entity state and relationships connects visual logic to the underlying entity architecture maintains consistency with the ecs programming model scene management visual scripts integrate with scene operations // example of scene management integration import { scenestate } from ' /services/scenestate'; / node executor for the "load scene" action @param data node data @param context execution context @returns node outputs / export const executeloadscenenode = (data any, context any) => { const { scenepath } = data; // load the scene scenestate loadscene(scenepath) then(() => { // continue execution if successful return { flow true }; }) catch(error => { console error('failed to load scene ', error); return { flow false }; }); }; / node executor for the "get current scene" data node @param data node data @param context execution context @returns node outputs / export const executegetcurrentscenenode = (data any, context any) => { const currentscene = scenestate state currentscenepath value; // return scene path output return { scenepath currentscene }; }; this integration enables scripts to load and manage scenes provides access to scene information for logic allows creation of scene transitions and level management connects visual scripting to the scene workflow supports building complete interactive experiences asset system visual scripts can reference and manipulate assets // example of asset system integration import { filesstate } from ' /services/filesstate'; / node executor for the "play sound" action @param data node data @param context execution context @returns node outputs / export const executeplaysoundnode = (data any, context any) => { const { soundassetid, volume, loop } = data; // get sound asset const soundasset = filesstate getfilebyid(soundassetid); if (!soundasset) { console error('sound asset not found ', soundassetid); return { flow true }; // continue execution despite error } // play the sound const audiosystem = getaudiosystem(); audiosystem playsound(soundasset url, { volume volume || 1 0, loop loop || false }); // return flow output return { flow true }; }; this integration allows scripts to reference and use assets like sounds, textures, and models provides access to asset metadata and properties enables dynamic asset loading and manipulation connects visual scripting to the asset management system supports creating rich multimedia experiences benefits of visual scripting the visual scripting system provides several key advantages accessibility enables non programmers to create complex behaviors visualization represents logic flows in an intuitive graphical format rapid iteration facilitates quick prototyping and testing of ideas modularity encourages creation of reusable script components integration connects seamlessly with other editor systems extensibility allows addition of custom nodes for specialized functionality debugging provides visual tracing of execution paths these benefits create a more inclusive and efficient development process for interactive content creation next steps with an understanding of the visual scripting system, the next chapter explores the editor control functions that provide core functionality for managing the world editor next editor control functions docid 9sqkzef8j2gm rco9 wd