Area 08 · Backend · Concept 17 of 20

File Uploads to S3

Store user files in object storage, usually through presigned URLs.

11–20 · Production levelPhase 0 + 5 · Weeks 1–2, 18–20≈ 1 hour

The idea

User files (PDFs for RAG, images, audio) belong in object storage like S3, Google Cloud Storage or Cloudflare R2, not in your database or on server disks. Objects are cheap, durable and served at scale.

With presigned URLs, your API creates a short-lived signed URL; the browser uploads the file directly to storage, never passing through your servers. Then the client tells your API the upload finished (or a storage event notifies you), and you enqueue ingestion.

Validate type and size, scan for malware if needed, store files under tenant-scoped keys, keep buckets private, and serve downloads through presigned GET URLs too.

Why it matters

Document upload is the entry point of nearly every RAG product.

Picture it

Words to know

  • Object storageStorage of files (objects) addressed by key in buckets.
  • Presigned URLA temporary URL that grants one specific action on one object.

Try it without code

Create a bucket

  1. Create a private bucket in any cloud.
  2. Upload a file in the console.
  3. Generate a presigned URL and open it in a private browser window.

Use it in code

Issue a presigned upload URL with size and type limits.

python
# pip install boto3
import uuid, boto3
from fastapi import HTTPException

s3 = boto3.client("s3", region_name="eu-west-1")
BUCKET, MAX_MB, ALLOWED = "rag-uploads", 25, {"application/pdf", "text/markdown"}

def create_upload(tenant_id: str, filename: str, content_type: str, size_bytes: int):
    if content_type not in ALLOWED or size_bytes > MAX_MB * 1024 * 1024:
        raise HTTPException(400, f"Only PDF or Markdown up to {MAX_MB} MB")
    key = f"{tenant_id}/{uuid.uuid4()}/{filename}"
    post = s3.generate_presigned_post(BUCKET, key, ExpiresIn=300,
        Conditions=[["content-length-range", 1, MAX_MB * 1024 * 1024],
                    {"Content-Type": content_type}],
        Fields={"Content-Type": content_type})
    return {"key": key, "upload": post}
PRACTICE · TICK IT AFTER THIS

Switch your RAG app to presigned uploads and trigger ingestion when the upload completes.

Common mistakes

  • Public buckets.
  • Trusting the client-declared content type without checking the file.

Go deeper