Asset handling and pipeline

overview the asset handling and pipeline component is a critical element of the ir engine's world editor that manages the import, organization, and optimization of digital resources used in 3d worlds it provides a comprehensive system for bringing external files into projects, organizing them in a structured library, and processing them for optimal performance by implementing specialized preview and compression tools, this component ensures that creators can efficiently work with various asset types while maintaining high performance in their worlds this chapter explores the implementation, workflow, and optimization of assets within the world editor core concepts asset types the editor supports various asset types for world creation 3d models three dimensional object representations (glb, gltf, fbx) textures image files used for surface details (png, jpg, ktx2) materials surface property definitions that determine appearance audio sound effects and music files (mp3, wav) video motion picture files for dynamic content (mp4, webm) scripts code files that define behavior prefabs reusable object templates with predefined components these assets serve as the building blocks for creating interactive 3d environments asset management the asset management system provides tools for organizing resources file browser interface for navigating the project's asset structure categorization grouping assets by type and function search functionality finding assets by name, type, or metadata metadata handling storing and editing additional information about assets version tracking managing different iterations of the same asset this system creates an organized library of resources for efficient project development asset pipeline the asset pipeline processes resources for optimal use import processing converting external formats to engine compatible formats optimization reducing file sizes while maintaining quality compression applying specialized algorithms for different asset types preview generation creating thumbnails and preview representations validation checking assets for compatibility and potential issues this pipeline ensures assets are prepared for efficient use in the engine implementation asset panel the assets panel provides the main interface for asset management // simplified from src/panels/assets/index tsx import react from 'react'; import { usehookstate } from '@hookstate/core'; import { paneldragcontainer, paneltitle } from '@ir engine/ui/src/components/editor/layout/panel'; import { tabdata } from 'rc dock'; import { usetranslation } from 'react i18next'; import { filesstate } from ' / /services/filesstate'; import categorieslist from ' /categories'; import resources from ' /resources'; import topbar from ' /topbar'; / assets panel content component @returns assets panel component / const assetspanel react fc = () => { const { t } = usetranslation(); const filesstate = usehookstate(filesstate state); return ( \<div classname="assets panel"> \<topbar /> \<div classname="assets panel content"> \<categorieslist /> \<resources /> \</div> \</div> ); }; / assets panel title component @returns assets panel title component / const assetspaneltitle react fc = () => { const { t } = usetranslation(); return ( \<paneldragcontainer> \<paneltitle>{t('editor\ assets title')}\</paneltitle> \</paneldragcontainer> ); }; / assets panel tab configuration / export const assetspaneltab tabdata = { id 'assetspanel', title \<assetspaneltitle />, content \<assetspanel />, closable false }; this component provides the main interface for browsing and managing assets includes a topbar with actions like import and create contains a categories list for filtering assets by type displays resources (files and folders) in the current directory integrates with the editor's tab system file browser the file browser displays and manages project files // simplified from src/panels/files/filebrowser tsx import react, { useeffect, usestate } from 'react'; import { usehookstate } from '@hookstate/core'; import { filesstate } from ' / /services/filesstate'; import { fileitem } from ' /fileitem'; import { contextmenu } from ' /contextmenu'; / file browser component @param props component props @returns file browser component / export const filebrowser react fc<{ currentpath string; onfileselect (file filedata) => void; }> = ({ currentpath, onfileselect }) => { // get files state const filesstate = usehookstate(filesstate state); const files = filesstate files value; const selectedfileid = filesstate selectedfileid value; // load files for the current path useeffect(() => { filesstate loaddirectory(currentpath); }, \[currentpath]); // handle file selection const handlefileclick = (file filedata) => { filesstate selectfile(file id); onfileselect(file); }; // handle context menu const handlecontextmenu = (e react mouseevent, file filedata) => { e preventdefault(); filesstate selectfile(file id); // show context menu at cursor position contextmenu show(e clientx, e clienty, file); }; return ( \<div classname="file browser"> {files map(file => ( \<fileitem key={file id} file={file} isselected={file id === selectedfileid} onclick={() => handlefileclick(file)} oncontextmenu={(e) => handlecontextmenu(e, file)} /> ))} \</div> ); }; this component displays files and folders in the current directory handles file selection and context menu interactions loads directory contents when the path changes renders file items with appropriate icons and information integrates with the filesstate service for data management asset preview the asset preview system displays appropriate visualizations for different asset types // simplified from src/components/assets/assetspreviewpanel tsx import react, { useeffect } from 'react'; import { usehookstate } from '@hookstate/core'; import { filesstate } from ' / /services/filesstate'; import { audiopreviewpanel } from ' /assetpreviewpanels/audiopreviewpanel'; import { imagepreviewpanel } from ' /assetpreviewpanels/imagepreviewpanel'; import { modelpreviewpanel } from ' /assetpreviewpanels/modelpreviewpanel'; import { videopreviewpanel } from ' /assetpreviewpanels/videopreviewpanel'; import { previewunavailable } from ' /assetpreviewpanels/previewunavailable'; / asset preview panel component @returns asset preview panel component / export const assetspreviewpanel react fc = () => { // get selected file const filesstate = usehookstate(filesstate state); const selectedfileid = filesstate selectedfileid value; const selectedfile = filesstate files value find(f => f id === selectedfileid); // state for preview component const \[previewcomponent, setpreviewcomponent] = usestate\<react componenttype\<any> | null>(null); // determine preview component based on file type useeffect(() => { if (!selectedfile) { setpreviewcomponent(null); return; } // select appropriate preview component based on file type switch (selectedfile contenttype) { case 'image/png' case 'image/jpeg' case 'image/webp' setpreviewcomponent(() => imagepreviewpanel); break; case 'model/gltf binary' case 'model/gltf+json' setpreviewcomponent(() => modelpreviewpanel); break; case 'audio/mpeg' case 'audio/wav' setpreviewcomponent(() => audiopreviewpanel); break; case 'video/mp4' case 'video/webm' setpreviewcomponent(() => videopreviewpanel); break; default setpreviewcomponent(() => previewunavailable); break; } }, \[selectedfile]); // if no file is selected or no preview component is available if (!selectedfile || !previewcomponent) { return \<div classname="no preview">no file selected\</div>; } // render the appropriate preview component return ( \<div classname="asset preview panel"> \<h3>{selectedfile name}\</h3> \<previewcomponent file={selectedfile} /> \</div> ); }; this component determines the appropriate preview component based on file type renders specialized previews for different asset types displays file information alongside the preview handles cases where no file is selected or preview is unavailable updates dynamically when the selection changes asset import the asset import system handles bringing external files into the project // simplified from src/functions/assetfunctions ts import { filesstate } from ' /services/filesstate'; import { importsettingsstate } from ' /services/importsettingsstate'; / handles file upload to the project @param files files to upload @param directory target directory @returns promise resolving to uploaded file paths / export const handleuploadfiles = async ( files file\[], directory string ) promise\<string\[]> => { // get import settings const importsettings = importsettingsstate getsettings(); // process each file const uploadpromises = files map(async (file) => { // check if file should be processed before upload if (importsettings autooptimize) { // apply automatic optimization based on file type if (file type startswith('image/')) { file = await compressimage(file, importsettings imagecompression); } else if (file type includes('gltf') || file type includes('glb')) { file = await optimizemodel(file, importsettings modeloptimization); } } // upload the file to the server const response = await uploadfiletoserver(file, directory); // return the path to the uploaded file return response path; }); // wait for all uploads to complete const uploadedpaths = await promise all(uploadpromises); // refresh the directory to show new files await filesstate refreshdirectory(directory); return uploadedpaths; }; / uploads a file to the server @param file file to upload @param directory target directory @returns server response / const uploadfiletoserver = async (file file, directory string) => { // create form data for the upload const formdata = new formdata(); formdata append('file', file); formdata append('directory', directory); // send the upload request const response = await fetch('/api/upload', { method 'post', body formdata }); // parse and return the response return await response json(); }; this code processes files before upload based on import settings applies automatic optimization for supported file types uploads files to the server with appropriate metadata refreshes the directory listing to show new files returns paths to the uploaded files for further use image compression the image compression system optimizes textures for performance // simplified from src/components/assets/imagecompressionpanel tsx import react, { usestate } from 'react'; import { button, select, slider } from '@ir engine/ui'; import { compressimage } from ' / /functions/assetfunctions'; import { filesstate } from ' / /services/filesstate'; / image compression panel component @param props component props @returns image compression panel component / export const imagecompressionpanel react fc<{ file filedata; onclose () => void; }> = ({ file, onclose }) => { // compression settings const \[format, setformat] = usestate('ktx2'); const \[quality, setquality] = usestate(75); const \[mipmaps, setmipmaps] = usestate(true); const \[loading, setloading] = usestate(false); // handle compression const handlecompress = async () => { setloading(true); try { // get the file content const filecontent = await fetch(file url) then(res => res blob()); // compress the image const compressedimage = await compressimage(filecontent, { format, quality, mipmaps, originalname file name }); // upload the compressed image await handleuploadfiles(\[compressedimage], file directory); // close the panel onclose(); } catch (error) { console error('error compressing image ', error); } finally { setloading(false); } }; return ( \<div classname="image compression panel"> \<h3>image compression\</h3> \<div classname="form group"> \<label>format\</label> \<select value={format} onchange={(e) => setformat(e target value)} options={\[ { value 'ktx2', label 'ktx2 (recommended)' }, { value 'webp', label 'webp' }, { value 'jpg', label 'jpg' } ]} /> \</div> \<div classname="form group"> \<label>quality {quality}%\</label> \<slider min={1} max={100} value={quality} onchange={(value) => setquality(value)} /> \</div> \<div classname="form group"> \<label>generate mipmaps\</label> \<input type="checkbox" checked={mipmaps} onchange={(e) => setmipmaps(e target checked)} /> \</div> \<div classname="actions"> \<button onclick={onclose} variant="secondary">cancel\</button> \<button onclick={handlecompress} variant="primary" disabled={loading} \> {loading ? 'compressing ' 'compress'} \</button> \</div> \</div> ); }; this component provides controls for configuring image compression settings supports different output formats optimized for real time rendering allows quality adjustment to balance file size and visual fidelity includes options for generating mipmaps for better rendering at different distances processes and uploads the compressed image to replace or supplement the original model optimization the model optimization system prepares 3d assets for efficient rendering // simplified from src/components/assets/modelcompressionpanel tsx import react, { usestate } from 'react'; import { button, checkbox, numberinput } from '@ir engine/ui'; import { optimizemodel } from ' / /functions/assetfunctions'; import { filesstate } from ' / /services/filesstate'; / model optimization panel component @param props component props @returns model optimization panel component / export const modelcompressionpanel react fc<{ file filedata; onclose () => void; }> = ({ file, onclose }) => { // optimization settings const \[generatelods, setgeneratelods] = usestate(true); const \[lodlevels, setlodlevels] = usestate(3); const \[compresstextures, setcompresstextures] = usestate(true); const \[draco, setdraco] = usestate(true); const \[loading, setloading] = usestate(false); // handle optimization const handleoptimize = async () => { setloading(true); try { // get the file content const filecontent = await fetch(file url) then(res => res blob()); // optimize the model const optimizedmodel = await optimizemodel(filecontent, { generatelods, lodlevels, compresstextures, draco, originalname file name }); // upload the optimized model await handleuploadfiles(\[optimizedmodel], file directory); // close the panel onclose(); } catch (error) { console error('error optimizing model ', error); } finally { setloading(false); } }; return ( \<div classname="model optimization panel"> \<h3>model optimization\</h3> \<div classname="form group"> \<checkbox label="generate lods (levels of detail)" checked={generatelods} onchange={(e) => setgeneratelods(e target checked)} /> \</div> {generatelods && ( \<div classname="form group"> \<label>lod levels\</label> \<numberinput min={1} max={5} value={lodlevels} onchange={(value) => setlodlevels(value)} /> \</div> )} \<div classname="form group"> \<checkbox label="compress textures" checked={compresstextures} onchange={(e) => setcompresstextures(e target checked)} /> \</div> \<div classname="form group"> \<checkbox label="apply draco compression" checked={draco} onchange={(e) => setdraco(e target checked)} /> \</div> \<div classname="actions"> \<button onclick={onclose} variant="secondary">cancel\</button> \<button onclick={handleoptimize} variant="primary" disabled={loading} \> {loading ? 'optimizing ' 'optimize'} \</button> \</div> \</div> ); }; this component provides controls for configuring model optimization settings supports generation of level of detail (lod) variants for performance includes options for texture compression within the model offers mesh compression using algorithms like draco processes and uploads the optimized model to the project asset workflow the complete asset workflow follows this sequence sequencediagram participant user participant assetspanel participant filebrowser participant assetfunctions participant server participant optimizer participant previewpanel user >>assetspanel drag files or click import assetspanel >>assetfunctions handleuploadfiles() assetfunctions >>optimizer apply auto optimization (if enabled) optimizer >>assetfunctions return processed files assetfunctions >>server upload files server >>assetfunctions confirm upload assetfunctions >>filebrowser refresh directory filebrowser >>user display new files user >>filebrowser select file filebrowser >>previewpanel show file preview previewpanel >>user display appropriate preview user >>filebrowser right click file, select "optimize" filebrowser >>assetspanel show optimization panel user >>assetspanel configure optimization settings user >>assetspanel click "optimize" assetspanel >>optimizer process file with settings optimizer >>assetspanel return optimized file assetspanel >>server upload optimized file server >>assetspanel confirm upload assetspanel >>filebrowser refresh directory filebrowser >>user display optimized file this diagram illustrates the user imports files through the assets panel files are processed according to import settings uploaded files appear in the file browser the user can select files to view previews the user can apply additional optimization as needed optimized files are uploaded and displayed in the browser integration with other components the asset handling system integrates with several other components of the world editor scene management assets are used in scene creation and manipulation // example of scene integration import { filesstate } from ' /services/filesstate'; import { editorstate } from ' /services/editorservices'; / adds a model to the scene @param modelid id of the model asset @returns promise resolving to the created entity / export const addmodeltoscene = async (modelid string) promise\<entity> => { // get the model file const modelfile = filesstate getfilebyid(modelid); if (!modelfile) { throw new error(`model with id ${modelid} not found`); } // create an entity for the model const entity = editorstate createentity(); // add a model component to the entity editorstate addcomponent(entity, 'model', { url modelfile url, type modelfile contenttype }); // position the entity at the center of the scene editorstate addcomponent(entity, 'transform', { position { x 0, y 0, z 0 }, rotation { x 0, y 0, z 0 }, scale { x 1, y 1, z 1 } }); return entity; }; this integration retrieves asset information from the file system creates entities in the scene based on assets configures components with asset properties positions assets in the 3d environment connects the asset library to the scene graph material system assets are used in material creation and configuration // example of material system integration import { filesstate } from ' /services/filesstate'; import { materialsstate } from ' /services/materialsstate'; / creates a material from a texture @param textureid id of the texture asset @returns promise resolving to the created material id / export const creatematerialfromtexture = async (textureid string) promise\<string> => { // get the texture file const texturefile = filesstate getfilebyid(textureid); if (!texturefile) { throw new error(`texture with id ${textureid} not found`); } // create a material with the texture const materialid = await materialsstate creatematerial({ name `${texturefile name split(' ')\[0]} material`, type 'standard', properties { map texturefile url, color '#ffffff', roughness 0 5, metalness 0 0 } }); return materialid; }; this integration uses texture assets to create materials configures material properties based on asset characteristics manages relationships between assets and materials provides a workflow for material creation from assets enables reuse of assets across multiple materials visual scripting assets are referenced in visual scripting nodes // example of visual scripting integration import { filesstate } from ' /services/filesstate'; import { visualscriptstate } from ' /services/visualscriptstate'; / creates a play sound node in the visual script @param soundid id of the sound asset @param graphid id of the visual script graph @returns promise resolving to the created node id / export const createplaysoundnode = async ( soundid string, graphid string ) promise\<string> => { // get the sound file const soundfile = filesstate getfilebyid(soundid); if (!soundfile) { throw new error(`sound with id ${soundid} not found`); } // create a play sound node in the graph const nodeid = await visualscriptstate createnode(graphid, { type 'playsound', position { x 100, y 100 }, data { soundurl soundfile url, volume 1 0, loop false } }); return nodeid; }; this integration references audio assets in sound playback nodes configures node properties based on asset metadata maintains connections between assets and script nodes enables dynamic behavior based on project assets provides a visual representation of asset usage in scripts benefits of the asset pipeline the asset handling and pipeline component provides several key advantages organization provides a structured system for managing project resources optimization reduces file sizes and improves runtime performance previewing enables quick visualization of assets without external tools workflow streamlines the process of importing and using assets consistency ensures assets follow project standards and formats performance prepares assets for efficient rendering and memory usage collaboration facilitates team workflows with clear asset organization these benefits create a more efficient and manageable development process for 3d world creation next steps with an understanding of asset handling and pipeline, the next chapter explores how scenes are managed and how gltf models are integrated into the world editor next scene operations and gltf management docid\ h8xem1tx0z6jvy4diooj5