OpenAI API and Batch Processing Guide for Developers

Complete OpenAI API reference for 2026 - model catalog, chat completions, function calling, structured outputs with JSON schema, embeddings, rate limits, and cost management.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

May 18, 2026
6 min read
OpenAI API and Batch Processing Guide for Developers

The OpenAI API enables engineering teams to integrate state-of-the-art language models, reasoning engines, and multimodal processing directly into production applications. By pairing standard real-time chat completions with structured outputs and asynchronous batch processing, developers can build reliable data pipelines while cutting inference costs by 50 percent.

This architectural guide covers authentication setup, Chat Completions, schema-enforced Structured Outputs, asynchronous Batch API orchestration, client-side rate limit throttling, and enterprise cost optimization strategies.

OpenAI API Architectural Foundations

Interacting with OpenAI models requires a clean request lifecycle spanning client initialization, model selection, prompt construction, and response parsing. The platform exposes REST endpoints wrapped by production-grade SDKs for Python and TypeScript environments.

Authentication and Project Isolation

Modern API configurations use scoped Project API keys rather than legacy organization-wide credentials. Project keys restrict access to dedicated compute budgets, prevent lateral permission escalation, and enable granular usage tracking across microservices.

Set your project API key as an environment variable rather than hardcoding credentials inside source code:

bash
export OPENAI_API_KEY="sk-proj-your-isolated-project-key"
export OPENAI_ORG_ID="org-your-organization-id"
export OPENAI_PROJECT_ID="proj-your-project-id"

Install the official software development kits:

bash
# Python official SDK
pip install openai pydantic

# Node.js and TypeScript official SDK
npm install openai zod

The official OpenAI Python SDK and OpenAI Node.js SDK automatically resolve these environment variables, manage HTTP keep-alive connection pooling, and handle default socket timeout configurations.


Core Chat Completions and Streaming

The standard interface for text generation and reasoning is the Chat Completions endpoint. Requests accept an array of message objects detailing system instructions, developer context, user inputs, and assistant history.

Synchronous Chat Completion in Python

Here is how to initialize the Python client and execute a standard completion request:

python
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),
    organization=os.environ.get("OPENAI_ORG_ID"),
    project=os.environ.get("OPENAI_PROJECT_ID"),
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "system",
            "content": "You are a senior systems architect. Provide direct, concise analysis."
        },
        {
            "role": "user",
            "content": "Explain the architectural trade-offs between monolithic databases and distributed event logs."
        }
    ],
    temperature=0.2,
    max_tokens=1200,
)

print(response.choices[0].message.content)
print(f"Prompt Tokens: {response.usage.prompt_tokens}")
print(f"Completion Tokens: {response.usage.completion_tokens}")
print(f"Total Tokens: {response.usage.total_tokens}")

Server-Sent Events Streaming in Node.js

For user-facing interfaces where time-to-first-token directly impacts perceived responsiveness, streaming chunks via Server-Sent Events (SSE) avoids multi-second blocking delays:

typescript
import OpenAI from "openai";

const openai = new OpenAI();

async function streamResponse(prompt: string): Promise<void> {
  const stream = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [
      { role: "system", content: "You are an automated code audit assistant." },
      { role: "user", content: prompt }
    ],
    stream: true,
  });

  for await (const chunk of stream) {
    const delta = chunk.choices[0]?.delta?.content || "";
    process.stdout.write(delta);
  }
  process.stdout.write("\n");
}

streamResponse("Analyze memory leak patterns in asynchronous Node.js microservices.");

Structured Outputs with Schema Enforcement

Traditional JSON mode guarantees that output text parses as syntactically valid JSON, but it frequently suffers from missing required fields, hallucinated object keys, and invalid datatypes.

OpenAI Structured Outputs solves this limitation by compiling your supplied schema into a grammar constraint. The sampler evaluates token probabilities strictly within the allowed grammar, guaranteeing 100 percent adherence to your schema definition as detailed in the official Structured Outputs Documentation.

Python Implementation with Pydantic

Using Pydantic Models provides native static typing alongside runtime schema extraction:

python
from typing import List, Optional
from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI()

class VulnerabilityReport(BaseModel):
    cve_id: str = Field(description="Standard CVE identifier if known, or internal tag")
    severity: str = Field(description="Critical, High, Medium, or Low")
    affected_component: str = Field(description="Package or module identifier")
    remediation_steps: List[str] = Field(description="Ordered list of action items")
    cvss_score: float = Field(description="Numeric score between 0.0 and 10.0")

class SecurityAuditResult(BaseModel):
    scan_timestamp: str
    repository_name: str
    vulnerabilities: List[VulnerabilityReport]
    audit_passed: bool

completion = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {
            "role": "system",
            "content": "Extract security vulnerability details from the raw audit output."
        },
        {
            "role": "user",
            "content": "Dependency scan found express-jwt 0.1.2 with algorithmic confusion vulnerability. CVSS 8.5. Remediation upgrade to 8.5.3 and enforce secret rotation."
        }
    ],
    response_format=SecurityAuditResult,
)

parsed_audit = completion.choices[0].message.parsed
print(f"Audit Passed: {parsed_audit.audit_passed}")
for vuln in parsed_audit.vulnerabilities:
    print(f"Found {vuln.severity} issue in {vuln.affected_component}: CVSS {vuln.cvss_score}")

Node.js Implementation with Zod

In TypeScript environments, integrate Zod Validation using the helper library:

typescript
import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";

const openai = new OpenAI();

const DatabaseMigrationSchema = z.object({
  table_name: z.string(),
  operation: z.enum(["CREATE", "ALTER", "DROP"]),
  columns_added: z.array(z.string()),
  estimated_lock_risk: z.enum(["LOW", "MEDIUM", "HIGH"]),
  rollback_query: z.string(),
});

async function extractMigrationPlan(rawNotes: string) {
  const completion = await openai.beta.chat.completions.parse({
    model: "gpt-4o",
    messages: [
      { role: "system", content: "Structure the database migration request into executable metadata." },
      { role: "user", content: rawNotes },
    ],
    response_format: zodResponseFormat(DatabaseMigrationSchema, "migration_plan"),
  });

  return completion.choices[0].message.parsed;
}

Function Calling and Tool Execution

Beyond data extraction, language models act as central reasoning coordinators by selecting and executing external functions. This architecture forms the backbone of modern autonomous workflows and agent systems.

  1. Tool Definition: You provide standard JSON Schema signatures describing available functions.
  2. Model Evaluation: The model inspects user intent and returns a structured call object containing function names and parsed arguments.
  3. Local Execution: Your backend executes the corresponding native function.
  4. Final Synthesis: You pass tool execution output back to the model for conversational synthesis.

To explore building production agent loops, review our in-depth guides on AI Agents Production State and Reliability and AutoGen Multi-Agent Framework. Official API tool specifications are maintained in the Function Calling Documentation.


OpenAI Batch API for 50 Percent Cost Reduction

When workloads do not require sub-second interactive responses, submitting requests through the synchronous Chat Completions endpoint wastes compute budget. The OpenAI Batch API provides a 50 percent discount on both input tokens and output tokens with a guaranteed 24-hour turnaround window.

Supported batch targets include:

  • /v1/chat/completions for bulk classification, summarization, entity extraction, and evaluation datasets.
  • /v1/embeddings for bulk document vector indexing across massive enterprise knowledge bases.

For detailed operational parameters, consult the official OpenAI Batch API Guide.

Batch Processing Workflow Stages

The batch lifecycle consists of four programmatic stages:

  1. File Preparation: Constructing a local JSONL file containing formatted request objects.
  2. File Upload: Uploading the JSONL file to OpenAI storage with the purpose flag set to batch.
  3. Job Creation: Initializing the batch process referencing the uploaded file ID and target endpoint.
  4. Status Polling and Download: Tracking job status and downloading output files upon completion.
snippet
[ Raw Records ] 
       │
       ▼
[ Prepare JSONL File ] ──► [ POST /v1/files (purpose='batch') ]
                                       │
                                       ▼
                       [ POST /v1/batches (endpoint='/v1/chat/completions') ]
                                       │
                                       ▼
                       [ Asynchronous 24-Hour Queue ]
                                       │
                                       ▼
                       [ Completed Batch Job - output_file_id ]
                                       │
                                       ▼
                       [ Download & Ingest Results ]

Complete Batch Processing Implementation in Python

Below is an end-to-end Python pipeline that formats records, uploads the file, creates a batch job, polls for execution progress, and streams the processed outputs:

python
import json
import time
from openai import OpenAI

client = OpenAI()

# Stage One - Construct the JSONL file
records = [
    {"record_id": "cust_101", "text": "Customer inquiring about upgrading from starter to enterprise plan with SSO requirements."},
    {"record_id": "cust_102", "text": "Billing inquiry regarding duplicate credit card charge during renewal cycle."},
    {"record_id": "cust_103", "text": "Feature request for exporting telemetry logs directly to external S3 buckets."}
]

batch_file_path = "support_tickets_batch.jsonl"

with open(batch_file_path, "w", encoding="utf-8") as f:
    for rec in records:
        batch_line = {
            "custom_id": rec["record_id"],
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": "gpt-4o-mini",
                "temperature": 0.1,
                "messages": [
                    {"role": "system", "content": "Classify the ticket into category, urgency (High/Medium/Low), and sentiment."},
                    {"role": "user", "content": rec["text"]}
                ]
            }
        }
        f.write(json.dumps(batch_line) + "\n")

# Stage Two - Upload the JSONL file
print("Uploading batch input file...")
with open(batch_file_path, "rb") as file_stream:
    uploaded_file = client.files.create(
        file=file_stream,
        purpose="batch"
    )

print(f"File uploaded successfully. File ID: {uploaded_file.id}")

# Stage Three - Create the batch job
print("Submitting batch execution job...")
batch_job = client.batches.create(
    input_file_id=uploaded_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"environment": "production", "pipeline": "support_classification"}
)

print(f"Batch created successfully. Batch ID: {batch_job.id}")
print(f"Initial Status: {batch_job.status}")

# Stage Four - Poll for completion
while batch_job.status in ["validating", "in_progress", "finalizing"]:
    print(f"Current status: {batch_job.status}. Waiting 30 seconds...")
    time.sleep(30)
    batch_job = client.batches.retrieve(batch_job.id)

print(f"Batch finished with status: {batch_job.status}")

if batch_job.status == "completed":
    print(f"Total requests: {batch_job.request_counts.total}")
    print(f"Completed requests: {batch_job.request_counts.completed}")
    print(f"Failed requests: {batch_job.request_counts.failed}")

    # Stage Five - Retrieve output content
    output_file_id = batch_job.output_file_id
    file_response = client.files.content(output_file_id)
    
    output_lines = file_response.text.strip().split("\n")
    for line in output_lines:
        result = json.loads(line)
        custom_id = result["custom_id"]
        response_content = result["response"]["body"]["choices"][0]["message"]["content"]
        print(f"\nResult for {custom_id}:\n{response_content}")
elif batch_job.status == "failed":
    print("Batch failed. Inspecting errors:")
    for error in batch_job.errors.data:
        print(f"Error {error.code}: {error.message}")

Batch API Architectural Advantages

MetricReal-Time Chat CompletionsBatch API
Pricing Discount0 percent (Standard Pricing)50 percent flat discount
Rate Limit QuotaShared Tier RPM / TPMSeparate, dedicated Batch Quota
Turnaround LatencySub-second (Interactive)Asynchronous (Within 24 hours)
Throughput CapacitySubject to burst throttlingUp to tens of thousands of requests per file
Input Token CostFull catalog rate50 percent of catalog rate
Output Token CostFull catalog rate50 percent of catalog rate

Rate Limits and Production Resiliency

Enterprise applications encountering heavy traffic must manage platform rate limits effectively. OpenAI throttles accounts across two primary dimensions:

  1. Requests Per Minute (RPM): Total HTTP transactions executed per 60-second window.
  2. Tokens Per Minute (TPM): Combined prompt and completion tokens processed per 60-second window.

Consult the Rate Limits Architecture Guide for complete tier thresholds.

Exponential Backoff with Jitter

When handling transient 429 Too Many Requests or 503 Service Unavailable responses, simple static sleep timers cause thundering herd contention. Implement exponential backoff randomized with full jitter:

python
import random
import time
from openai import OpenAI, RateLimitError, APIConnectionError

client = OpenAI()

def execute_with_jitter_retry(messages, model="gpt-4o", max_retries=5, base_delay=1.0, max_delay=32.0):
    retries = 0
    while True:
        try:
            return client.chat.completions.create(
                model=model,
                messages=messages,
                temperature=0.2
            )
        except (RateLimitError, APIConnectionError) as error:
            retries += 1
            if retries > max_retries:
                raise RuntimeError(f"Exceeded maximum retries ({max_retries}). Last error: {error}")
            
            # Calculate exponential backoff with full jitter
            delay = min(max_delay, base_delay * (2 ** (retries - 1)))
            jittered_delay = random.uniform(0, delay)
            
            print(f"Rate limited or transient failure. Retrying in {jittered_delay:.2f}s (Attempt {retries}/{max_retries})...")
            time.sleep(jittered_delay)

Client-Side Token Bucket Throttling

To prevent burst saturation before making network calls, position a token bucket rate limiter in front of your dispatch loop:

typescript
export class TokenBucketLimiter {
  private capacity: number;
  private tokens: number;
  private refillRate: number; // tokens per millisecond
  private lastRefill: number;

  constructor(capacity: number, refillRatePerSecond: number) {
    this.capacity = capacity;
    this.tokens = capacity;
    this.refillRate = refillRatePerSecond / 1000;
    this.lastRefill = Date.now();
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = now - this.lastRefill;
    this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.refillRate);
    this.lastRefill = now;
  }

  async acquire(tokensNeeded: number = 1): Promise<void> {
    while (true) {
      this.refill();
      if (this.tokens >= tokensNeeded) {
        this.tokens -= tokensNeeded;
        return;
      }
      const missingTokens = tokensNeeded - this.tokens;
      const waitTimeMs = Math.ceil(missingTokens / this.refillRate);
      await new Promise((resolve) => setTimeout(resolve, waitTimeMs));
    }
  }
}

Cost Optimization and Prompt Caching

Managing OpenAI inference budgets across large-scale deployments requires a layered cost optimization strategy.

Automatic Prompt Caching

Modern models automatically cache prompt prefixes longer than 1,024 tokens. Cached input tokens benefit from a 50 percent discount and execute with lower time-to-first-token latency.

To maximize prompt cache hit ratios:

  • Keep System Prompts Identical: Place unchanging system instructions and tool definitions at the absolute beginning of the request array.
  • Append Dynamic Context at the End: Place user-specific variables, query inputs, and dynamic timestamps at the tail of the message sequence so the preceding prefix remains byte-identical.
  • Inspect Usage Objects: Monitor response.usage.prompt_tokens_details.cached_tokens to measure cache efficiency across workloads.

Model Routing Strategies

Never default all application tasks to flagship models. Route workloads dynamically based on semantic complexity:

  • Flagship Reasoning (gpt-4o): Multi-step code synthesis, nuanced legal document extraction, architectural decisions, and complex structured output schemas.
  • High-Throughput Lightweight (gpt-4o-mini): Ticket categorization, entity tagging, simple sentiment parsing, and initial RAG relevance filtering.
  • Dedicated Embeddings: Vector indexing with text-embedding-3-small and text-embedding-3-large. Review our architectural comparison of Open Source Embedding Models and BGE-M3 Multilingual Embeddings for vector search benchmarks.

Production Deployment Checklist

Before rolling out OpenAI integrations to production clusters, verify that your service satisfies the following operational criteria:

  1. Scoped Project Credentials: Ensure API keys belong to isolated project containers with strict spend alert notifications configured.
  2. Schema-Enforced Outputs: Use Structured Outputs via Pydantic or Zod for every programmatic data extraction task.
  3. Offline Batch Offloading: Route non-urgent bulk classifications and backfills to the Batch API to immediately secure 50 percent compute savings.
  4. Adaptive Backoff with Jitter: Implement client-side exponential backoff with full jitter to gracefully navigate 429 rate limits.
  5. Prompt Prefix Stability: Anchor common system instructions at the front of messages to exploit automatic 50 percent prompt caching discounts.
  6. Telemetry and Tracing: Log prompt token counts, completion token counts, latency distributions, and cached token metrics to observability platforms.

Frequently Asked Questions

How does the OpenAI Batch API reduce inference costs by 50 percent?

The OpenAI Batch API processes requests asynchronously within a 24-hour turnaround window using separate non-real-time compute queues. Because workloads run during cluster downtime, OpenAI offers a flat 50 percent discount on both input prompt tokens and output completion tokens across supported models.

What is the difference between Structured Outputs and JSON mode?

JSON mode guarantees valid JSON syntax but does not enforce that the output conforms to a specific schema or contains required fields. Structured Outputs uses constrained sampling to strictly guarantee 100 percent mathematical adherence to your supplied JSON Schema or Pydantic model without missing keys or hallucinated data structures.

How can developers manage Tier rate limits and handle HTTP 429 errors?

Manage rate limits by implementing client-side token bucket or leaky bucket algorithms, reading response headers like x-ratelimit-remaining-tokens, using exponential backoff with full jitter on retries, and routing bulk offline jobs to the Batch API which has independent quota allocations.

What file format is required for submitting Batch API jobs?

The Batch API requires JSONL (JSON Lines) files where each line is an independent JSON object containing a unique custom_id, an HTTP method (POST), the target endpoint (/v1/chat/completions or /v1/embeddings), and the request body.

Can OpenAI Batch API jobs be cancelled while running?

Yes. Developers can issue a cancel call against an active batch ID. The job transitions into cancelling state, stops queuing new lines, and returns output files containing any records that completed processing before the cancellation was initiated.

#openai#gpt-4o#api#structured-outputs#embeddings

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Visionary technologist, software engineer, and machine learning specialist. Founder and CEO of Pristren, directing engineering teams that ship production-grade AI/ML pipelines, mission-critical full-stack applications, and developer tooling. Creator of Zlyqor, the unified team workspace platform. Author of 540+ technical guides and benchmark research reports on large language models, agentic workflows, Model Context Protocol (MCP), and modern web stacks.

PristrenZlyqor