Modal: Run GPU Python Functions as Serverless Jobs in 30 Seconds

Updated
8 min read
Modal: Run GPU Python Functions as Serverless Jobs in 30 Seconds
TL;DR

Modal lets you decorate any Python function to run on cloud GPUs with sub-5-second cold starts, persistent model caching, and OpenAI-compatible web endpoints.

What Is Modal?

Modal is a cloud platform that makes GPU-backed Python functions as easy to deploy as regular functions. You add a decorator, push your code, and Modal handles provisioning, scaling, and teardown. Cold starts are under 5 seconds for cached images - dramatically faster than Lambda or Cloud Run for GPU workloads.

The Core Pattern

python
import modal

app = modal.App("llm-inference")

# Define a container image with your dependencies
image = modal.Image.debian_slim().pip_install(
    "torch", "transformers", "accelerate"
)

# This function runs on an A100 in the cloud
@app.function(gpu="A100", image=image, timeout=300)
def generate(prompt: str) -> str:
    from transformers import pipeline
    pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.2")
    return pipe(prompt, max_new_tokens=200)[0]["generated_text"]

@app.local_entrypoint()
def main():
    result = generate.remote("Explain quantum entanglement simply:")
    print(result)

Run with: modal run inference.py

Persistent Volumes for Model Weights

Downloading a 7B model on every cold start would take minutes. Modal Volumes solve this:

python
volume = modal.Volume.from_name("model-weights", create_if_missing=True)

@app.function(gpu="A100", volumes={"/models": volume})
def download_model():
    from huggingface_hub import snapshot_download
    snapshot_download("mistralai/Mistral-7B-Instruct-v0.2", local_dir="/models/mistral-7b")
    volume.commit()

Run the download once. Subsequent function calls mount the volume and skip the download entirely.

Web Endpoints

Turn any function into an HTTP API with @modal.web_endpoint:

python
@app.function(gpu="T4", image=image)
@modal.web_endpoint(method="POST")
def inference_api(item: dict) -> dict:
    result = generate_text(item["prompt"])
    return {"output": result}

Modal gives you a stable HTTPS URL. Scales to zero when idle, scales up automatically under load.

Scheduled Jobs

python
@app.function(schedule=modal.Cron("0 8 * * *"))
def daily_report():
    # runs every day at 8 AM UTC
    generate_and_send_report()

Lambda tops out at 15-minute execution time, has no native GPU support, and cold starts on large packages (PyTorch) can take 30+ seconds. Modal was built specifically for ML workloads: GPU support is first-class, cold starts with cached images are under 5 seconds, and execution time limits are much more generous (1 hour by default). For anything involving torch or model inference, Modal is significantly less painful.

Pricing

Modal charges per second of GPU time: T4 at $0.000164/second, A10G at $0.000306/second, A100 (40GB) at $0.000900/second. No charges when functions aren't running. The free tier includes $30/month of compute.

Resources

#modal#serverless-gpu#python#deployment#inference

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.