Designing a video upload and transcoding pipeline

User video breaks the assumptions that work for images. Files are hundreds of megabytes, uploads take minutes on mobile networks, phones record in formats browsers will not play, and transcoding takes long enough that you cannot do it inside a request. A video feature is really a small distributed system: upload, store, transcode, package, thumbnail, deliver, and keep the user informed in between.

This post walks through those stages, what each one involves if you build it yourself, and how they map onto Dreep, including the part Dreep does not do yet: there are no webhooks, so you poll.

The stages

Upload

Never stream large video through your own API server. Use a presigned upload: your backend asks for a single-use upload URL, the client PUTs the bytes directly, and your backend confirms. Dreep accepts MP4, MOV and WebM up to 500 MB per file. The direct browser uploads post covers the pattern and the security checks in detail.

Transcode

iPhones record HEVC in a MOV container by default, and many Android devices produce variable frame rate video. For broad playback you want H.264 video with AAC audio in MP4, sized for the screen it will play on. Transcoding is CPU-heavy: a few minutes of 1080p can take longer to encode than to watch, depending on settings and hardware.

Package

A single MP4 works for short clips. For anything longer, HLS splits the video into short segments listed in an .m3u8 manifest, so the player fetches only what it needs and can start playing quickly. Safari plays HLS natively; other browsers use hls.js.

Thumbnails and previews

A poster frame for the player, a smaller thumbnail for lists and often a short silent preview for hover states. Each is another derived output from the same original.

What building it yourself involves

• An upload path with presigned URLs, size limits and a record of pending uploads.
• A job queue and FFmpeg workers sized for peak load, with retries and timeouts for files that never finish.
• Encoding settings per output: codec, bitrate or CRF, keyframe interval for HLS, audio handling.
• Storage for originals, renditions and segments, plus lifecycle rules for each.
• Job state in your database and a way to tell your app, and your user, when it is done.
• Handling of corrupt or unusual inputs, which you will meet in week one.

None of this is exotic, but it is weeks of work to get right and ongoing work to keep running. The rest of this post shows how much of it collapses when outputs are URLs.

Outputs as URLs

In Dreep, every derived output is a delivery URL on the original asset with query parameters: format=hls for HLS, format=mp4 with width and height for a resized MP4, trimStart and trimEnd for clips, fps for frame rate, and an image format plus time for a thumbnail. Requesting mp3, wav or aac extracts the audio track.

// Server side: presign a large video (up to 500 MB)
const upload = await dreep.presignUpload({
  filename: "lesson-04.mov",
  contentType: "video/quicktime",
  sizeBytes: 312_000_000,
  folder: "courses/intro",
});
// ...the client PUTs to upload.uploadUrl, then:
const video = await dreep.confirmUpload({ id: upload.id });

// Every output is just a URL on the same asset
const hls = dreep.url(video, { format: "hls", height: 720 });
const mp4 = dreep.url(video, { format: "mp4", width: 1280, height: 720, fit: "contain" });
const preview = dreep.url(video, { format: "gif", trimStart: 0, trimEnd: 3, fps: 12, width: 480 });

// Thumbnails use the time parameter (seconds), added to the URL directly
const poster = new URL(dreep.url(video, { format: "webp", width: 1280 }));
poster.searchParams.set("time", "5");

The 202-then-poll model

Light jobs such as remuxing a MOV into an MP4 container are done inline and return 200 with the file. Heavy jobs, meaning re-encodes, HLS packaging and GIF conversion, are queued. The first request returns 202 Accepted:

• A browser (Accept: text/html) gets a processing page that refreshes itself.
• An API client (Accept: application/json) gets { "error": "Asset is processing" }.
• When the job finishes, the same URL returns 200 and is served from cache from then on.

So the job id is the URL. To know when an output is ready, you request it again. In practice the cleanest place to do that is your backend, right after confirm, so outputs are warm before anyone presses play:

// Ask for an output and wait until it is ready.
// 202 means "queued or processing": request the same URL again later.
const waitForOutput = async (url, { attempts = 40, delayMs = 3000 } = {}) => {
  for (let attempt = 1; attempt <= attempts; attempt += 1) {
    const response = await fetch(url, {
      method: "GET",
      headers: { Accept: "application/json" },
    });

    if (response.status === 200) {
      await response.body?.cancel(); // we only wanted to know it is ready
      return url;
    }
    if (response.status !== 202) {
      throw new Error("Unexpected status " + response.status);
    }

    await new Promise((resolve) => setTimeout(resolve, Math.min(delayMs * attempt, 30_000)));
  }
  throw new Error("Still processing after " + attempts + " attempts");
};

// After confirm, warm the outputs so the first viewer does not wait
await Promise.all([waitForOutput(hls), waitForOutput(poster.toString())]);
await db.videos.update({ id: lessonId, status: "ready", hlsUrl: hls, posterUrl: poster.toString() });

The same loop in Go:

// waitForOutput requests url until it returns 200, backing off between 202s.
func waitForOutput(ctx context.Context, url string) error {
    delay := 3 * time.Second
    for attempt := 0; attempt < 40; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
        if err != nil {
            return err
        }
        req.Header.Set("Accept", "application/json")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return err
        }
        resp.Body.Close()

        switch resp.StatusCode {
        case http.StatusOK:
            return nil
        case http.StatusAccepted:
            // still processing
        default:
            return fmt.Errorf("unexpected status %d", resp.StatusCode)
        }

        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(delay):
        }
        delay = min(delay*2, 30*time.Second)
    }
    return errors.New("still processing")
}

// hls := client.URL(video.ID, &dreep.Transform{Format: dreep.FormatHLS, Height: 720})
// err := waitForOutput(ctx, hls)

Run this in a background job rather than the request that handled confirm, store a processing status on your own record, and flip it to ready when the loop returns. Your UI reads your database, not Dreep.

Playing HLS

<video id="player" controls playsinline poster="POSTER_URL"></video>
<script src="https://cdn.jsdelivr.net/npm/hls.js@1"></script>
<script>
  const video = document.getElementById("player");
  const src = "HLS_URL";

  if (video.canPlayType("application/vnd.apple.mpegurl")) {
    video.src = src; // Safari plays HLS natively
  } else if (Hls.isSupported()) {
    const hls = new Hls();
    hls.loadSource(src);
    hls.attachMedia(video);
  }
</script>

Each HLS URL is a single rendition at the size you ask for, not an adaptive ladder of several bitrates. For most product video, a 720p rendition for mobile and a 1080p one for desktop, chosen by your page, is enough. If you need true adaptive bitrate switching across many renditions, a dedicated video platform is a better fit.

Telling the user what is happening

Between upload and ready there is a gap, and the interface has to say so. Show the video as processing in your own UI, using the status on your record, rather than embedding a player that will hit a 202. If someone does open the HLS URL in a browser early, Dreep's processing page refreshes itself until the output is ready, which is a reasonable fallback but not something to design around. For private courses, put the videos in a signed folder; polling and playback then use signed URLs with an expiry long enough for both.

When you need webhooks

Dreep does not send webhooks or job callbacks at the time of writing. Polling an output URL is the only way to learn that a job has finished. That is fine when:

• Uploads arrive at human pace, tens or hundreds a day, and a background job per upload is cheap.
• A short delay between upload and availability is acceptable, as it is for course content, listings and profile video.
• You already run a job queue for other work.

It is not a good fit when:

• You process thousands of videos an hour and want the provider to push completion events rather than holding a poller per job.
• Downstream systems, such as moderation or publishing pipelines, must be triggered reliably by a completion event.
• You need a delivery guarantee and an audit trail for job completion.

In those cases, build the pipeline with your own queue and FFmpeg workers, or choose a provider with callbacks.

Costs and limits

Video transformations are a separate monthly allowance: 20 on Free, 1,000 on Starter ($2.99 a month) and 4,000 on Pro ($5.99 a month). One-time packs add 10,000 for $29. Assume each distinct output you generate counts once, so a pipeline that makes an HLS rendition, an MP4 fallback and a poster for every upload uses about three per video.

Worked example: 300 lesson uploads a month, each with one 720p HLS rendition, one MP4 fallback and one poster, is roughly 900 video transformations, which fits Starter. Doubling to 1080p and 720p HLS for each pushes it to about 1,200 and onto Pro. Storage is separate: 300 lessons at 200 MB is 60 GB of originals, which needs Pro plus a storage add-on, or your own bucket on a paid plan.

Full parameter reference in the video transformations docs, and an overview on the video transcoding API page.