Storage providers

overview the storage providers component is an essential element of the ir engine's server core that manages the storage and retrieval of files such as images, videos, and 3d models it provides a unified interface for file operations while abstracting the underlying storage implementation, whether local filesystem, amazon s3, or google cloud storage by implementing this abstraction layer, the system enables flexibility in storage solutions without requiring changes to application code this chapter explores the implementation, interface, and usage of storage providers within the ir engine core concepts storage abstraction the storage abstraction provides a consistent interface for file operations implementation independence separates file operation logic from storage implementation details unified api provides a standard set of methods for all storage backends interchangeability allows switching between storage solutions with minimal code changes configuration driven uses application settings to determine the active storage provider extensibility supports adding new storage backends without changing application code this abstraction creates a flexible foundation for file management provider implementations the system includes multiple storage provider implementations local storage stores files on the server's filesystem s3 storage stores files in amazon s3 buckets gcs storage stores files in google cloud storage custom providers can be added to support additional storage solutions each implementation handles the specifics of its storage backend while conforming to the common interface file operations storage providers support a standard set of file operations upload store files in the selected storage system download retrieve files from storage url generation create urls for accessing stored files listing enumerate files in a directory or folder deletion remove files from storage existence checking verify if a file exists directory operations create, check, and manage directories these operations cover the essential needs for file management in applications implementation provider interface the storage provider interface defines the contract for all implementations // simplified from src/media/storageprovider/storageprovider interface ts import { passthrough } from 'stream'; / interface for object upload parameters / export interface storageobjectputinterface { // the file's path/name in storage key string; // the file content (buffer or stream) body buffer | passthrough; // the mime type of the file contenttype string; // optional metadata metadata? record\<string, string>; // other optional parameters contentencoding? string; cachecontrol? string; } / interface that all storage providers must implement / export interface storageproviderinterface { / uploads a file to storage @param object upload parameters @param params additional provider specific parameters @returns promise resolving to success status / putobject(object storageobjectputinterface, params? any) promise\<boolean>; / retrieves a file from storage @param key file path/name @returns promise resolving to file content and metadata / getobject(key string) promise<{ body buffer, contenttype string }>; / gets a url for accessing a file @param key file path/name @param internal whether the url is for internal use @returns url string / getcachedurl(key string, internal? boolean) string; / lists files in a folder @param foldername folder path @param recursive whether to include subfolders @returns promise resolving to file information / listfoldercontent(foldername string, recursive? boolean) promise\<any\[]>; / deletes files from storage @param keys array of file paths/names @returns promise resolving to deletion result / deleteresources(keys string\[]) promise\<any>; / checks if a file exists @param filename file name @param directorypath directory path @returns promise resolving to existence status / doesexist(filename string, directorypath string) promise\<boolean>; / checks if a path is a directory @param filename file name @param directorypath directory path @returns promise resolving to directory status / isdirectory(filename string, directorypath string) promise\<boolean>; // additional methods like moveobject, getsignedurl, etc } this interface defines the contract that all storage providers must implement specifies the parameters and return types for each method provides a consistent api for file operations enables interchangeability between different storage implementations local storage implementation the local storage provider stores files on the server's filesystem // simplified from src/media/storageprovider/local storage ts import fs from 'fs'; import path from 'path'; import { storageproviderinterface, storageobjectputinterface } from ' /storageprovider interface'; import config from ' / /appconfig'; / storage provider that uses the local filesystem / export class localstorage implements storageproviderinterface { // base directory for file storage path prefix string; / constructor / constructor() { // set up the base directory from configuration this path prefix = path join(process cwd(), config storage local basedir || 'upload'); // create the directory if it doesn't exist if (!fs existssync(this path prefix)) { fs mkdirsync(this path prefix, { recursive true }); } } / uploads a file to the local filesystem @param data upload parameters @returns promise resolving to success status / async putobject(data storageobjectputinterface) promise\<boolean> { // construct the full file path const filepath = path join(this path prefix, data key); // ensure the directory exists const directorypath = path dirname(filepath); if (!fs existssync(directorypath)) { fs mkdirsync(directorypath, { recursive true }); } // write the file if (buffer isbuffer(data body)) { // handle buffer data fs writefilesync(filepath, data body); } else { // handle stream data return new promise((resolve, reject) => { const writestream = fs createwritestream(filepath); data body pipe(writestream); writestream on('finish', () => resolve(true)); writestream on('error', (err) => reject(err)); }); } return true; } / gets a url for accessing a file @param key file path/name @returns url string / getcachedurl(key string) string { // for local storage, construct a url to the local server const domain = config storage local domain || 'localhost 3030'; const protocol = config storage local protocol || 'http'; return `${protocol} //${domain}/uploads/${key}`; } / lists files in a folder @param foldername folder path @param recursive whether to include subfolders @returns promise resolving to file information / async listfoldercontent(foldername string, recursive = false) promise\<any\[]> { const folderpath = path join(this path prefix, foldername); // check if the folder exists if (!fs existssync(folderpath)) { return \[]; } // read the directory const entries = fs readdirsync(folderpath, { withfiletypes true }); // process each entry const results = \[]; for (const entry of entries) { const entrypath = path join(foldername, entry name); if (entry isdirectory()) { // handle directory results push({ key entrypath + '/', name entry name, type 'folder', size 0 }); // include subdirectory contents if recursive if (recursive) { const subresults = await this listfoldercontent(entrypath, true); results push( subresults); } } else { // handle file const stats = fs statsync(path join(folderpath, entry name)); const extension = path extname(entry name) substring(1); results push({ key entrypath, name path basename(entry name, ' ' + extension), type extension, size stats size, lastmodified stats mtime }); } } return results; } / deletes files from storage @param keys array of file paths/names @returns promise resolving to deletion result / async deleteresources(keys string\[]) promise\<any> { for (const key of keys) { const filepath = path join(this path prefix, key); if (fs existssync(filepath)) { const stats = fs statsync(filepath); if (stats isdirectory()) { // remove directory recursively fs rmdirsync(filepath, { recursive true }); } else { // remove file fs unlinksync(filepath); } } } return { deleted keys }; } // implementations for other interface methods } this implementation uses the node js filesystem api to store and manage files constructs file paths based on the configured base directory handles both buffer and stream data for file uploads generates urls that point to a local file server provides directory listing with file metadata implements file and directory deletion s3 storage implementation the s3 storage provider stores files in amazon s3 buckets // simplified from src/media/storageprovider/s3 storage ts import { s3client, putobjectcommand, getobjectcommand, listobjectsv2command, deleteobjectscommand } from '@aws sdk/client s3'; import { storageproviderinterface, storageobjectputinterface } from ' /storageprovider interface'; import config from ' / /appconfig'; / storage provider that uses amazon s3 / export class s3storage implements storageproviderinterface { // s3 client instance private s3client s3client; // s3 bucket name private bucket string; // cdn domain for urls private cdndomain string; / constructor / constructor() { // create s3 client with configuration this s3client = new s3client({ region config storage s3 region, credentials { accesskeyid config storage s3 accesskeyid, secretaccesskey config storage s3 secretaccesskey } }); // set bucket and cdn domain this bucket = config storage s3 bucket; this cdndomain = config storage s3 cdndomain || `${this bucket} s3 amazonaws com`; } / uploads a file to s3 @param data upload parameters @returns promise resolving to success status / async putobject(data storageobjectputinterface) promise\<boolean> { // create command for s3 upload const command = new putobjectcommand({ bucket this bucket, key data key, body data body, contenttype data contenttype, metadata data metadata, contentencoding data contentencoding, cachecontrol data cachecontrol }); // execute the command await this s3client send(command); return true; } / gets a url for accessing a file @param key file path/name @returns url string / getcachedurl(key string) string { // construct url using cdn domain return `https //${this cdndomain}/${key}`; } / lists files in a folder @param foldername folder path @param recursive whether to include subfolders @returns promise resolving to file information / async listfoldercontent(foldername string, recursive = false) promise\<any\[]> { // ensure folder name ends with a slash const prefix = foldername endswith('/') ? foldername foldername + '/'; // create command for listing objects const command = new listobjectsv2command({ bucket this bucket, prefix prefix, delimiter recursive ? undefined '/' }); // execute the command const response = await this s3client send(command); // process the results const results = \[]; // process common prefixes (folders) if (response commonprefixes) { for (const prefix of response commonprefixes) { if (prefix prefix) { results push({ key prefix prefix, name prefix prefix split('/') slice( 2)\[0], type 'folder', size 0 }); } } } // process objects (files) if (response contents) { for (const object of response contents) { if (object key && object key !== prefix) { const name = object key split('/') pop() || ''; const extension = name includes(' ') ? name split(' ') pop() || '' ''; results push({ key object key, name name replace(` ${extension}`, ''), type extension, size object size || 0, lastmodified object lastmodified }); } } } return results; } / deletes files from storage @param keys array of file paths/names @returns promise resolving to deletion result / async deleteresources(keys string\[]) promise\<any> { // create command for deleting objects const command = new deleteobjectscommand({ bucket this bucket, delete { objects keys map(key => ({ key key })) } }); // execute the command const response = await this s3client send(command); return { deleted response deleted? map(obj => obj key) || \[] }; } // implementations for other interface methods } this implementation uses the aws sdk to interact with amazon s3 creates an s3 client configured with credentials from application settings maps interface methods to s3 specific commands handles s3 bucket operations for file management generates urls that point to s3 or a cdn processes s3 specific responses into a consistent format provider factory the provider factory creates and manages storage provider instances // simplified from src/media/storageprovider/storageprovider ts import config from ' / /appconfig'; import { storageproviderinterface } from ' /storageprovider interface'; import localstorage from ' /local storage'; import s3storage from ' /s3 storage'; import gcsstorage from ' /gcs storage'; // cache of provider instances const providers record\<string, storageproviderinterface> = {}; // map of provider types to implementations const storageimplementations record\<string, any> = { local localstorage, s3 s3storage, gcs gcsstorage }; / gets a storage provider instance @param providername provider name (default 'default') @returns storage provider instance / export const getstorageprovider = (providername = 'default') storageproviderinterface => { // create the default provider if it doesn't exist if (!providers\[providername]) { if (providername === 'default') { createdefaultstorageprovider(); } else { throw new error(`storage provider '${providername}' not found`); } } return providers\[providername]; }; / creates the default storage provider @returns storage provider instance / export const createdefaultstorageprovider = () storageproviderinterface => { // get provider type from configuration const providertype = config storage provider || 'local'; // get the provider implementation class const providerclass = storageimplementations\[providertype] || localstorage; // create an instance const instance = new providerclass(); // store the instance providers\['default'] = instance; return instance; }; this factory maintains a cache of provider instances maps provider types to implementation classes creates the default provider based on configuration provides a consistent way to access the active provider supports multiple provider instances if needed usage examples storage providers are used throughout the application for file operations uploading files files are uploaded using the putobject method // example of file upload import { getstorageprovider } from ' /storageprovider/storageprovider'; / uploads a file to storage @param filebuffer file content @param filename file name @param contenttype mime type @param directory directory path @returns promise resolving to the file key / async function uploadfile( filebuffer buffer, filename string, contenttype string, directory string ) promise\<string> { // get the storage provider const storage = getstorageprovider(); // construct the file key const key = `${directory}/${filename}`; // upload the file await storage putobject({ key key, body filebuffer, contenttype contenttype }); console log(`file ${key} uploaded successfully`); return key; } this function gets the active storage provider constructs a key (path) for the file uploads the file using the provider's putobject method returns the file key for future reference getting file urls urls for accessing files are generated using the getcachedurl method // example of url generation import { getstorageprovider } from ' /storageprovider/storageprovider'; / gets a url for a file @param key file key @returns url string / function getfileurl(key string) string { // get the storage provider const storage = getstorageprovider(); // get the url const url = storage getcachedurl(key); console log(`url for ${key} ${url}`); return url; } this function gets the active storage provider generates a url for the file using the provider's getcachedurl method returns the url for use in the application listing files files in a directory are listed using the listfoldercontent method // example of directory listing import { getstorageprovider } from ' /storageprovider/storageprovider'; / lists files in a directory @param directory directory path @param recursive whether to include subdirectories @returns promise resolving to file information / async function listfiles(directory string, recursive = false) promise\<any\[]> { // get the storage provider const storage = getstorageprovider(); // list the directory contents const files = await storage listfoldercontent(directory, recursive); console log(`found ${files length} files in ${directory}`); return files; } this function gets the active storage provider lists the contents of a directory using the provider's listfoldercontent method returns the file information for use in the application deleting files files are deleted using the deleteresources method // example of file deletion import { getstorageprovider } from ' /storageprovider/storageprovider'; / deletes files from storage @param keys array of file keys @returns promise resolving to deletion result / async function deletefiles(keys string\[]) promise\<any> { // get the storage provider const storage = getstorageprovider(); // delete the files const result = await storage deleteresources(keys); console log(`deleted ${result deleted length} files`); return result; } this function gets the active storage provider deletes the files using the provider's deleteresources method returns the deletion result for confirmation integration with other components the storage providers integrate with several other components of the server core application configuration the storage provider system uses configuration values // example of configuration integration import config from ' / /appconfig'; // in createdefaultstorageprovider const providertype = config storage provider || 'local'; // in localstorage constructor this path prefix = path join(process cwd(), config storage local basedir || 'upload'); // in s3storage constructor this s3client = new s3client({ region config storage s3 region, credentials { accesskeyid config storage s3 accesskeyid, secretaccesskey config storage s3 secretaccesskey } }); this integration uses configuration values to determine the active provider applies provider specific settings from configuration enables changing storage solutions through configuration supports environment specific storage settings services services use storage providers for file operations // example of service integration import { getstorageprovider } from ' /storageprovider/storageprovider'; import { application } from ' / /declarations'; / service for managing file uploads / export class fileuploadservice { app application; constructor(options any, app application) { this app = app; } / handles file upload @param id resource id @param data upload data @returns promise resolving to upload result / async create(id null, data any) promise\<any> { // get the storage provider const storage = getstorageprovider(); // construct the file key const key = `uploads/${data directory}/${data filename}`; // upload the file await storage putobject({ key key, body data file, contenttype data contenttype }); // create a database record for the file const record = await this app service('static resource') create({ name data filename, key key, contenttype data contenttype, url storage getcachedurl(key) }); return record; } } this integration uses the storage provider for file uploads creates database records for uploaded files generates urls for accessing files provides a consistent api for file operations benefits of storage providers the storage providers component provides several key advantages abstraction separates file operation logic from storage implementation details flexibility enables switching between storage solutions with minimal code changes standardization provides a consistent api for file operations scalability supports cloud storage solutions for handling large volumes of files maintainability isolates storage specific code in dedicated implementations testability allows mocking storage operations for testing extensibility supports adding new storage backends as needed these benefits make storage providers an essential component for managing files in the ir engine's server core next steps with an understanding of how the application manages files, the next chapter explores how to add custom logic to service operations next hooks (feathersjs) docid\ fqdwzowgt6cserey5ezup