Upload Engine

What is the Upload Engine?

The Upload Engine is the core component that handles file uploads in Uploadista. It manages the complete upload lifecycle: creating uploads, receiving file data in chunks, tracking progress, and storing files in your configured storage backend.

You don’t interact with the Upload Engine directly - instead, you configure the Uploadista Server which uses the Upload Engine internally. However, understanding how it works helps you:

Debug upload issues
Optimize for large files
Understand why resumable uploads are reliable
Know what happens when a flow processes uploaded files

Upload Lifecycle

Every file upload goes through these stages:

┌─────────────────────────────────────────────────────────────────────┐
│                         UPLOAD LIFECYCLE                            │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│   1. CREATE           2. UPLOAD CHUNKS        3. COMPLETE           │
│   ────────            ────────────────        ────────────          │
│                                                                     │
│   ┌─────────┐         ┌─────────┐             ┌─────────┐          │
│   │ Client  │         │ Client  │             │ Upload  │          │
│   │ Request │         │ Sends   │             │ Ready   │          │
│   │ Upload  │    ──►  │ Data    │    ──►      │ for Use │          │
│   └─────────┘         └─────────┘             └─────────┘          │
│        │                   │                       │                │
│        ▼                   ▼                       ▼                │
│   ┌─────────┐         ┌─────────┐             ┌─────────┐          │
│   │ KV Store│         │  Data   │             │  Flow   │          │
│   │Metadata │         │  Store  │             │ Trigger │          │
│   └─────────┘         └─────────┘             └─────────┘          │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Stage 1: Create Upload

When an upload starts, the engine:

Generates a unique upload ID
Stores metadata in the KV Store (file name, size, type)
Returns the upload ID to the client

At this point, no file data has been transferred yet.

Stage 2: Upload Chunks

The client sends file data, either:

All at once - For small files (< 10MB)
In chunks - For larger files (recommended)

The engine:

Receives each chunk
Writes it to the Data Store
Updates the offset in the KV Store
Emits progress events via Event System

Stage 3: Complete

When all bytes are received:

The upload is marked as complete
A completion event is emitted
If the upload was initiated by a Flow, the flow execution resumes to continue processing

Resumable Uploads

One of the Upload Engine’s key features is resumable uploads. If a connection drops mid-upload, the client can resume from where it left off.

How It Works

Initial upload attempt:
────────────────────────────────────────────────►
[████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░]
              ▲
              Connection lost at 25%

Resume upload:
              ────────────────────────────────────►
[████████████████████████████████████████████████]
              ▲                                  ▲
              Resume from offset 25%             Complete!

Offset tracking - The KV Store records how many bytes have been received
Client queries offset - Before resuming, the client asks “where did we leave off?”
Resume from offset - The client sends only the remaining data
No re-upload needed - Already-received data is preserved

Why This Matters

Scenario	Without Resumable	With Resumable
100MB file, connection drops at 90%	Re-upload all 100MB	Upload remaining 10MB
Mobile user on flaky network	Frequent failures	Graceful recovery
Large video upload	Frustrating experience	Reliable upload

Two Ways to Use the Upload Engine

The Upload Engine can be used in two different ways:

Standalone Uploads

For simple file storage without processing, use the Upload Engine directly:

User uploads file
        │
        ▼
┌───────────────┐
│ Upload Engine │ ──► Stores file in Data Store
└───────────────┘
        │
        ▼
   File ready for use

This is ideal when you just need to store files without any transformations.

Uploads Within Flows

When you need to process files (resize, optimize, transform), the Flows Engine orchestrates the upload:

User triggers flow
        │
        ▼
┌───────────────┐
│ Flows Engine  │ ──► Creates a job
└───────────────┘
        │
        ▼
┌───────────────┐
│  Input Node   │ ──► Initiates upload via Upload Engine
└───────────────┘
        │
        ▼
┌───────────────┐
│ Upload Engine │ ──► Stores file in Data Store
└───────────────┘     (flow paused during upload)
        │
        ▼
┌───────────────┐
│ Flows Engine  │ ──► Resumes after upload completes
└───────────────┘
        │
        ▼
   Processing nodes
   (resize, optimize...)
        │
        ▼
   Output nodes save
   processed files

When a flow executes:

The flow creates a job and starts executing
Input nodes use the Upload Engine to receive files
The flow pauses while waiting for uploads to complete
Once uploads finish, the flow resumes processing
Processing nodes transform the files
Output nodes save results to storage

HTTP API Reference

The Upload Engine exposes these HTTP endpoints (mounted by the Uploadista Server):

Method	Endpoint	Description
`POST`	`/uploadista/api/upload`	Create a new upload
`HEAD`	`/uploadista/api/upload/:id`	Get upload offset (for resume)
`PATCH`	`/uploadista/api/upload/:id`	Upload file data/chunk
`GET`	`/uploadista/api/upload/:id`	Get upload status and metadata
`DELETE`	`/uploadista/api/upload/:id`	Cancel/delete an upload

Create Upload

POST /uploadista/api/upload
Content-Type: application/json

{
  "fileName": "photo.jpg",
  "size": 5242880,
  "type": "image/jpeg"
}

Response:

{
  "id": "upload_abc123",
  "offset": 0,
  "metadata": {
    "fileName": "photo.jpg",
    "size": 5242880,
    "type": "image/jpeg"
  }
}

Upload Chunk

PATCH /uploadista/api/upload/upload_abc123
Content-Type: application/offset+octet-stream
Upload-Offset: 0

[binary data]