Modal: Run GPU Python Functions as Serverless Jobs in 30 Seconds

What Is Modal?

Modal is a cloud platform that makes GPU-backed Python functions as easy to deploy as regular functions. You add a decorator, push your code, and Modal handles provisioning, scaling, and teardown. Cold starts are under 5 seconds for cached images - dramatically faster than Lambda or Cloud Run for GPU workloads.

The Core Pattern

import modal

app = modal.App("llm-inference")

# Define a container image with your dependencies
image = modal.Image.debian_slim().pip_install(
    "torch", "transformers", "accelerate"
)

# This function runs on an A100 in the cloud
@app.function(gpu="A100", image=image, timeout=300)
def generate(prompt: str) -> str:
    from transformers import pipeline
    pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.2")
    return pipe(prompt, max_new_tokens=200)[0]["generated_text"]

@app.local_entrypoint()
def main():
    result = generate.remote("Explain quantum entanglement simply:")
    print(result)

Run with: modal run inference.py

Persistent Volumes for Model Weights

Downloading a 7B model on every cold start would take minutes. Modal Volumes solve this:

volume = modal.Volume.from_name("model-weights", create_if_missing=True)

@app.function(gpu="A100", volumes={"/models": volume})
def download_model():
    from huggingface_hub import snapshot_download
    snapshot_download("mistralai/Mistral-7B-Instruct-v0.2", local_dir="/models/mistral-7b")
    volume.commit()

Run the download once. Subsequent function calls mount the volume and skip the download entirely.

Web Endpoints

Turn any function into an HTTP API with @modal.web_endpoint:

@app.function(gpu="T4", image=image)
@modal.web_endpoint(method="POST")
def inference_api(item: dict) -> dict:
    result = generate_text(item["prompt"])
    return {"output": result}

Modal gives you a stable HTTPS URL. Scales to zero when idle, scales up automatically under load.

Scheduled Jobs

@app.function(schedule=modal.Cron("0 8 * * *"))
def daily_report():
    # runs every day at 8 AM UTC
    generate_and_send_report()

Modal vs AWS Lambda

Lambda tops out at 15-minute execution time, has no native GPU support, and cold starts on large packages (PyTorch) can take 30+ seconds. Modal was built specifically for ML workloads: GPU support is first-class, cold starts with cached images are under 5 seconds, and execution time limits are much more generous (1 hour by default). For anything involving torch or model inference, Modal is significantly less painful.

Pricing

Modal charges per second of GPU time: T4 at $0.000164/second, A10G at $0.000306/second, A100 (40GB) at $0.000900/second. No charges when functions aren't running. The free tier includes $30/month of compute.

Resources

• Modal
• Official docs
• Example gallery