Streams, Uploads and Background Work
Some work is too big to hold in memory or too slow to make a user wait. Read and send big files as streams, accept uploads with size and type limits, and move slow work out of the request so the response comes back at once.
Updated
Streams
- Streams: Big Files a Piece at a Time (Journeyman): A stream moves a file or a response a piece at a time instead of loading it whole, so pipeline from node:stream/promises can copy a huge file, readline can read a big CSV line by line, and an Express route can send a CSV export without holding it in memory.
Uploads
- File Uploads with Limits (Freelancer): A form with a file arrives as multipart/form-data, which multer reads in Express, saving the file under a name the server makes and refusing a file that's too big (413) or of the wrong type (400) before it fills the disk.
Background Work
- Work That Runs After the Response (Freelancer): When a request starts slow work the caller doesn't need to wait for, the server answers 202 Accepted with a Location to check later, hands the work to a job queue that runs it after the response, and moves heavy calculations to worker_threads, while job services such as BullMQ keep jobs safe across restarts.
Exercise: Photo Contest Uploads
- Exercise: Photo Contest Uploads (Freelancer): Finish Snapshot Saturday's API, accepting one photo per entry with multer's size and type limits, answering 202 while a small job queue makes each thumbnail after the response, and streaming every entry as a CSV download with pipeline, with Supertest and node --test checking each part.
Up Next: Professional Node
- Up Next: Professional Node (Journeyman): See what your APIs can do after Module 5, from live updates over WebSockets to streamed files, uploads with limits and work that runs after the response, meet the Live Quiz Night project where you'll show it, and get a first look at Module 6, where your API is logged, checked on every push and put online.