Periodic task scheduler

overview the periodic task scheduler is a mechanism within the ir engine's background processing system that enables tasks to run automatically at regular intervals rather than being a standalone component, it's an architectural pattern implemented within each task that requires periodic execution by using javascript's built in timing functions, particularly setinterval , the system ensures that critical background operations like data collection, monitoring, and maintenance happen consistently without manual intervention this chapter explores the implementation, functionality, and benefits of periodic task scheduling within the ir engine purpose and functionality periodic task scheduling serves several essential purposes automation eliminates the need for manual task triggering consistency ensures tasks run at regular, predictable intervals reliability reduces the risk of tasks being forgotten or missed resource management controls when resource intensive operations occur operational autonomy allows the system to maintain itself with minimal intervention many background operations require regular execution collecting analytics data (every 30 minutes) monitoring system events (every few minutes) performing database maintenance (daily) generating reports (hourly or daily) cleaning up temporary resources (periodically) the scheduler ensures these operations happen automatically and reliably implementation core mechanism setinterval the primary tool for periodic scheduling in the ir engine is javascript's built in setinterval function // basic usage of setinterval function periodictask() { console log("executing task at ", new date() toisostring()); // task logic goes here } // execute periodictask every 5 minutes (300,000 milliseconds) const intervalid = setinterval(periodictask, 300000); this function takes a callback function as its first parameter takes a time interval in milliseconds as its second parameter executes the callback repeatedly at the specified interval returns an identifier that can be used to cancel the interval if needed implementation in analytics collector the analytics data collector uses setinterval to schedule its execution // simplified from src/collect analytics ts import config from '@ir engine/server core/src/appconfig'; import multilogger from '@ir engine/server core/src/serverlogger'; const logger = multilogger child({ component 'taskserver\ collect analytics' }); // function that performs the actual data collection async function collectandstoredata(app) { logger info('starting analytics collection cycle'); // analytics collection logic logger info('analytics collection cycle complete'); } // main export function that sets up periodic execution export default (app) => { // get the interval from configuration const default interval seconds = 1800; // 30 minutes const configuredinterval = parseint(config\['task server'] processinterval); const runintervalmilliseconds = (configuredinterval || default interval seconds) 1000; // schedule periodic execution setinterval(async () => { try { await collectandstoredata(app); } catch (error) { logger error('error during analytics collection ', error); } }, runintervalmilliseconds); logger info(`analytics collection scheduled to run every ${runintervalmilliseconds / 1000} seconds`); }; this implementation retrieves the execution interval from configuration sets up error handling to prevent task failures from breaking the schedule logs the schedule for monitoring and debugging uses an async function to handle asynchronous operations implementation in event collector similarly, the kubernetes event collector uses setinterval for its periodic execution // simplified from src/collect events ts import config from '@ir engine/server core/src/appconfig'; import multilogger from '@ir engine/server core/src/serverlogger'; const logger = multilogger child({ component 'taskserver\ collect events' }); // function that collects kubernetes events const collectevents = async (app) => { logger info('starting kubernetes event collection'); // event collection logic logger info('kubernetes event collection complete'); }; // main export function that sets up periodic execution export default (app) => { // get the interval from configuration const default interval seconds = 60; // 1 minute const configuredinterval = parseint(config kubernetes? eventcollectioninterval); const intervalmilliseconds = (configuredinterval || default interval seconds) 1000; // schedule periodic execution setinterval(async () => { try { await collectevents(app); } catch (error) { logger error('error collecting kubernetes events ', error); } }, intervalmilliseconds); // also run once immediately collectevents(app) catch(error => { logger error('error during initial kubernetes event collection ', error); }); logger info(`kubernetes event collection scheduled to run every ${intervalmilliseconds / 1000} seconds`); }; this implementation adds an immediate execution in addition to the periodic schedule, ensuring that event collection begins right away rather than waiting for the first interval to elapse how setinterval works under the hood, setinterval relies on javascript's event loop and timer mechanisms sequencediagram participant code as application code participant timer as javascript timer system participant eventloop as event loop participant task as task function code >>timer setinterval(taskfn, 5000) timer >>timer register recurring timer loop every 5000ms timer >>eventloop queue task execution eventloop >>task execute task function task >>eventloop task complete note over timer wait for next interval end the process works as follows when setinterval is called, the javascript runtime registers a recurring timer when the specified time elapses, the callback is added to the event queue the event loop processes the queue and executes the callback after execution, the timer continues to run, and the process repeats this continues until the interval is explicitly cleared or the process terminates advanced scheduling patterns while setinterval provides basic periodic scheduling, more complex requirements may need additional patterns error handling and retry logic to prevent errors from breaking the schedule setinterval(async () => { try { await performtask(); } catch (error) { logger error('task failed ', error); // optional implement retry logic settimeout(async () => { try { logger info('retrying failed task'); await performtask(); } catch (retryerror) { logger error('retry also failed ', retryerror); } }, 60000); // retry after 1 minute } }, intervalmilliseconds); dynamic interval adjustment to adjust intervals based on system conditions let currentinterval = default interval; // function to adjust the interval function updateinterval() { // check system load or other conditions const systemload = getsystemload(); if (systemload > high threshold) { // reduce frequency during high load currentinterval = reduced frequency interval; } else { // use normal frequency currentinterval = default interval; } } // initial schedule let intervalid = setinterval(async () => { await performtask(); // check if interval needs adjustment const previousinterval = currentinterval; updateinterval(); // if interval changed, reschedule if (previousinterval !== currentinterval) { clearinterval(intervalid); intervalid = setinterval(performtask, currentinterval); logger info(`adjusted task interval to ${currentinterval}ms`); } }, currentinterval); staggered execution to prevent multiple tasks from running simultaneously // schedule tasks with slight offsets settimeout(() => { setinterval(taska, intervalmilliseconds); }, 0); settimeout(() => { setinterval(taskb, intervalmilliseconds); }, 10000); // start 10 seconds after taska settimeout(() => { setinterval(taskc, intervalmilliseconds); }, 20000); // start 20 seconds after taska integration with other components the periodic task scheduler pattern integrates with several other components of the background processing system task server application the task server application initializes each task, which then sets up its own periodic execution // from src/start ts import collectanalytics from ' /collect analytics'; import collectevents from ' /collect events'; export const start = async () promise\<application> => { const app = await createfeatherskoaapp(servermode task); // initialize tasks (which set up their own periodic execution) collectanalytics(app); collectevents(app); // other initialization code return app; }; this integration provides each task with the application context allows tasks to access services and resources ensures tasks start when the task server starts application configuration management tasks retrieve their execution intervals from configuration // from src/collect analytics ts const configuredinterval = parseint(config\['task server'] processinterval); this integration allows intervals to be configured without code changes enables environment specific scheduling supports runtime adjustments to execution frequency service interaction layer periodic tasks often interact with services to perform their work // example of a task interacting with services async function collectandstoredata(app) { // get data from one service const data = await app service('some service') find({ / query / }); // process the data const processeddata = processdata(data); // store results in another service await app service('results service') create(processeddata); } this integration allows tasks to retrieve and store data leverages existing service functionality maintains separation of concerns considerations and best practices when implementing periodic tasks, several considerations should be kept in mind interval selection choosing appropriate intervals involves balancing several factors resource usage shorter intervals increase system load data freshness longer intervals mean less current data business requirements some processes have specific timing needs coordination consider how tasks interact with each other error handling robust error handling is critical for periodic tasks catch exceptions prevent errors from breaking the schedule log errors ensure issues are visible for troubleshooting consider retries implement retry logic for transient failures monitor failures set up alerts for repeated failures resource management periodic tasks should be mindful of resource usage database connections release connections after use memory usage avoid memory leaks in long running processes cpu utilization be aware of task execution time network traffic consider the impact of frequent api calls graceful shutdown tasks should handle shutdown gracefully // example of cleaning up intervals on shutdown export default (app) => { const intervalid = setinterval(periodictask, interval); // clean up on application shutdown app on('close', () => { clearinterval(intervalid); logger info('periodic task scheduler stopped'); }); }; benefits of periodic task scheduling the periodic task scheduler pattern provides several key benefits automation eliminates manual intervention for routine operations reliability ensures critical tasks are executed consistently simplicity uses built in javascript functionality for basic scheduling flexibility allows each task to manage its own execution schedule configurability supports adjustable intervals through configuration maintainability keeps scheduling logic close to the task implementation these benefits make periodic task scheduling a fundamental pattern for background processing in the ir engine next steps with an understanding of how tasks are scheduled to run periodically, the next chapter explores how these tasks interact with other parts of the system to retrieve and store data next service interaction layer docid\ v5bybxvnqt1wgdfxrkjiu