Mediasoup WebRTC communication backbone

overview the mediasoup webrtc communication backbone is the core real time communication system of the ir engine's multiplayer infrastructure it enables the efficient exchange of audio, video, and custom game data between players in a multiplayer environment by leveraging webrtc technology and the mediasoup selective forwarding unit (sfu) architecture, the system provides low latency, scalable communication channels that are essential for creating responsive multiplayer experiences this chapter explores the implementation, components, and workflow of the webrtc communication backbone within the ir engine core concepts real time communication requirements multiplayer games require several types of real time communication audio transmission enabling players to speak to each other video transmission sharing webcam feeds when applicable game data exchange transmitting player actions, positions, and game state updates low latency ensuring all communication happens with minimal delay scalability supporting multiple concurrent players efficiently these requirements create significant technical challenges, particularly when dealing with many simultaneous connections webrtc technology webrtc (web real time communication) provides the foundation for peer to peer communication browser native technology built into modern web browsers direct communication enables peer to peer connections media capabilities handles audio and video streaming data channels supports custom data exchange nat traversal includes mechanisms to establish connections across networks webrtc provides the core protocols and apis needed for real time communication selective forwarding unit architecture while webrtc supports peer to peer connections, direct connections between all players would be inefficient at scale the mediasoup sfu architecture addresses this centralized routing each player connects to the server rather than to every other player efficient distribution the server forwards media and data only to players who need it reduced bandwidth players send their streams once, regardless of the number of recipients improved scalability supports more concurrent players than direct peer to peer server side control enables additional features like recording and stream manipulation this architecture significantly reduces the number of connections required, making large multiplayer sessions feasible implementation core mediasoup components the mediasoup implementation uses several key components workers and routers workers and routers form the foundation of the mediasoup infrastructure // simplified from src/socketwebrtcserverfunctions ts export async function startwebrtc() { // create mediasoup workers (separate processes for media handling) const workers = \[]; for (let i = 0; i < numworkers; i++) { const worker = await createworker({ loglevel 'warn', rtcminport 40000, rtcmaxport 49999 }); workers push(worker); } // create a router in each worker const routers = \[]; for (const worker of workers) { const router = await worker createrouter({ mediacodecs supportedmediacodecs }); routers push(router); } logger info('mediasoup workers and routers created'); return { routers, workers }; } this code creates multiple mediasoup worker processes for load distribution configures each worker with appropriate network ports creates a router within each worker to handle media routing returns the created workers and routers for use by the system transports transports establish the communication channels between clients and the server // simplified from src/webrtcfunctions ts export async function handlewebrtctransportcreate(action) { const { networkid, direction } = action; // get the appropriate router const router = getrouterfornetwork(networkid); // create transport options with ice candidates and dtls parameters const options = { listenips config mediasoup webrtctransport listenips, enableudp true, enabletcp true, preferudp true, initialavailableoutgoingbitrate 1000000 }; // create the webrtc transport const transport = await router createwebrtctransport(options); // store the transport in state const transportstate = getmutablestate(mediasouptransportstate); transportstate transports\[networkid]\[direction] merge({ id transport id, iceparameters transport iceparameters, icecandidates transport icecandidates, dtlsparameters transport dtlsparameters }); // set up transport event handlers transport on('dtlsstatechange', (dtlsstate) => { if (dtlsstate === 'closed') { transport close(); } }); // store the transport object for later use transportobjects\[transport id] = transport; return transport; } this function creates a webrtc transport for a specific network and direction configures the transport with appropriate network parameters stores the transport details in the application state sets up event handlers for transport lifecycle events returns the created transport for further operations producers producers represent outgoing media or data streams // simplified from src/webrtcfunctions ts export async function handlerequestproducer(action) { const { networkid, transportid, kind, rtpparameters, appdata } = action; // get the transport object const transport = transportobjects\[transportid]; if (!transport) { throw new error(`transport ${transportid} not found`); } // create the producer const producer = await transport produce({ kind, rtpparameters, appdata }); // store the producer in state const producerstate = getmutablestate(mediasoupmediaproducerstate); producerstate producers\[networkid] merge({ \[producer id] { id producer id, kind, transportid, appdata } }); // set up producer event handlers producer on('transportclose', () => { producer close(); // remove from state }); // store the producer object for later use producerobjects\[producer id] = producer; return producer; } this function creates a producer for a specific transport and media kind configures the producer with the provided rtp parameters stores the producer details in the application state sets up event handlers for producer lifecycle events returns the created producer for further operations consumers consumers represent incoming media or data streams // simplified from src/webrtcfunctions ts export async function handlerequestconsumer(action) { const { networkid, consumernetworkid, producerid, transportid, appdata } = action; // get the transport and producer objects const transport = transportobjects\[transportid]; const producer = producerobjects\[producerid]; if (!transport || !producer) { throw new error('transport or producer not found'); } // check if the consumer can consume the producer if (!router canconsume({ producerid producer id, rtpcapabilities action rtpcapabilities })) { throw new error('cannot consume this producer'); } // create the consumer const consumer = await transport consume({ producerid producer id, rtpcapabilities action rtpcapabilities, paused true, appdata }); // store the consumer in state const consumerstate = getmutablestate(mediasoupmediaconsumerstate); consumerstate consumers\[networkid] merge({ \[consumer id] { id consumer id, producerid, transportid, appdata } }); // set up consumer event handlers consumer on('transportclose', () => { consumer close(); // remove from state }); // store the consumer object for later use consumerobjects\[consumer id] = consumer; return consumer; } this function creates a consumer for a specific transport and producer checks if the consumer can consume the producer's media configures the consumer with the provided rtp capabilities stores the consumer details in the application state sets up event handlers for consumer lifecycle events returns the created consumer for further operations data channels in addition to media (audio/video), the system supports custom game data exchange // simplified from src/mediasoupserversystem tsx const requestdataproduceractionqueue = defineactionqueue(mediasoupdataproduceractions requestdataproducer matches); const requestdataconsumeractionqueue = defineactionqueue(mediasoupdataconsumeractions requestdataconsumer matches); // handler for data producer requests const handlerequestdataproducer = async (action) => { const { networkid, transportid, label, protocol, appdata } = action; // get the transport object const transport = transportobjects\[transportid]; // create the data producer const dataproducer = await transport producedata({ label, protocol, appdata }); // store the data producer in state const dataproducerstate = getmutablestate(mediasoupdataproducerstate); dataproducerstate dataproducers\[networkid] merge({ \[dataproducer id] { id dataproducer id, label, protocol, appdata } }); return dataproducer; }; // handler for data consumer requests const handlerequestdataconsumer = async (action) => { const { networkid, dataproducerid, transportid, appdata } = action; // get the transport and data producer objects const transport = transportobjects\[transportid]; const dataproducer = dataproducerobjects\[dataproducerid]; // create the data consumer const dataconsumer = await transport consumedata({ dataproducerid, appdata }); // store the data consumer in state const dataconsumerstate = getmutablestate(mediasoupdataconsumerstate); dataconsumerstate dataconsumers\[networkid] merge({ \[dataconsumer id] { id dataconsumer id, dataproducerid, label dataproducer label, protocol dataproducer protocol, appdata } }); return dataconsumer; }; this code defines action queues for data producer and consumer requests implements handlers for creating data producers and consumers configures data channels with appropriate labels and protocols stores data channel details in the application state enables the exchange of custom game data between clients communication workflow the complete webrtc communication workflow follows this sequence sequencediagram participant aliceclient as alice's game client participant server as ir engine server (mediasoup) participant bobclient as bob's game client aliceclient >>server request webrtc transport (send direction) server >>aliceclient transport details (ice candidates, dtls params) aliceclient >>server connect transport (dtls parameters) aliceclient >>server request audio producer server >>aliceclient producer created successfully aliceclient >>server start sending audio stream bobclient >>server request webrtc transport (receive direction) server >>bobclient transport details (ice candidates, dtls params) bobclient >>server connect transport (dtls parameters) bobclient >>server request consumer for alice's audio server >>bobclient consumer details (rtp parameters) server >>bobclient forward alice's audio stream note over aliceclient,bobclient similar flow for video and data channels this diagram illustrates clients establish webrtc transports with the server producers are created for outgoing streams (audio, video, data) consumers are created for incoming streams the server forwards media and data between clients this process enables real time communication between players integration with other components the webrtc communication backbone integrates with several other components of the multiplayer infrastructure instance lifecycle management the webrtc system is initialized during instance startup // simplified from src/instanceservermodule ts import { mediasoupserversystem } from ' /mediasoupserversystem'; export const instanceservermodule = { async initialize() { // initialize mediasoup await startwebrtc(); // register the mediasoup server system engine instance systemregistry add(mediasoupserversystem); // other initialization steps } }; this integration ensures webrtc is available when the instance starts registers the mediasoupserversystem to handle webrtc operations makes real time communication available to connected clients hyperflux state management webrtc components are managed through hyperflux state // simplified from src/transports/mediasoup/mediasouptransportstate ts import { definestate } from '@ir engine/hyperflux'; export const mediasouptransportstate = definestate({ name 'ee common mediasouptransportstate', initial { transports {} as record\<networkid, { send { id '', iceparameters {}, icecandidates \[], dtlsparameters {} }, recv { id '', iceparameters {}, icecandidates \[], dtlsparameters {} } }> } }); this integration stores webrtc transport details in hyperflux state makes transport information available throughout the application enables reactive updates when transport state changes similar state structures exist for producers and consumers user connection and authorization webrtc connections are established after user authentication // conceptual integration with user connection function handleuserconnected(user, socket) { // after user is authenticated socket on('webrtc transport create', async (data, callback) => { try { // create webrtc transport for the authenticated user const transport = await handlewebrtctransportcreate({ networkid user networkid, direction data direction }); // return transport details to the client callback({ id transport id, iceparameters transport iceparameters, icecandidates transport icecandidates, dtlsparameters transport dtlsparameters }); } catch (error) { callback({ error error message }); } }); // other webrtc related event handlers } this integration ensures only authenticated users can establish webrtc connections associates webrtc components with specific user sessions provides secure communication channels for authorized users benefits of mediasoup webrtc the mediasoup webrtc communication backbone provides several key advantages scalability efficiently handles many concurrent users through the sfu architecture low latency minimizes communication delay for responsive multiplayer experiences flexibility supports various media types and custom game data bandwidth efficiency optimizes network usage compared to peer to peer approaches server control enables additional features like recording and stream manipulation cross platform compatibility works across different browsers and devices nat traversal handles complex network configurations through ice mechanisms these benefits make the webrtc communication backbone an essential foundation for creating engaging multiplayer experiences in the ir engine next steps with an understanding of how real time communication is handled, the next chapter explores how game instances are created, managed, and terminated next instance lifecycle management docid\ cmtbojnfby3agr9ng2mbz