GPT-4o Language Model

• GPT-4o is a multilingual, multimodal generative model featuring a unified, decoder-only transformer that processes text, vision, and audio through cross-modal attention.
• It employs end-to-end autoregressive training across diverse data with real-time streaming capabilities, offering improved performance and robust ethical filtering.
• Evaluations reveal strong performance in language, reasoning, vision, and audio tasks while highlighting challenges in ambiguous input handling and domain-specific limitations.

GPT-4o is a multilingual, multimodal generative pre-trained transformer (LLM) released by OpenAI in May 2024. Distinguished by cross-modal capabilities, end-to-end training across text, vision, and audio, and reduced latency, GPT-4o represents a substantive advance in unified multimodal AI architectures. It is implemented as a single, autoregressive transformer, supporting real-time streaming interaction and offering both improved benchmark performance and enhanced safety/ethical filtering compared to prior OpenAI models. Evaluation across language, reasoning, vision, speech, and integrated multimodal tasks identifies both remarkable strengths and persistent domain-specific limitations.

1. Model Architecture and Training Paradigm

GPT-4o utilizes a unified, decoder-only transformer architecture. All modalities—text, audio (waveform or speech), images, and video—are encoded into a single shared embedding space, with modality-agnostic weights and a common sequence processing stack for cross-modal attention. At each transformer layer $\ell$, self-attention is applied over interleaved tokens (text, image patches, audio frames, video), with cross-modal fusion implemented by learned projection matrices allowing attention to flow between modality-specific queries, keys, and values. The primary training objective is standard cross-entropy over sequences of discretized tokens (text, image, audio):

$\mathcal{L} = - \sum_{i=1}^N \log p(x_i | x_{<i})$

Training is end-to-end: all modalities are treated identically as autoregressive targets, with no modality-specific “head.” Pretraining data encompasses a broad mixture of web-scraped text, structured code/math, images, audio, and video, filtered with extensive safety and privacy preprocessing (OpenAI Moderation API, personal-data scrubbing, image fingerprint opt-outs, targeted CSAM/hate/CBRN filters). Post-training, model alignment uses reinforcement learning from human feedback (RLHF), synthetic safety traces, and product-level classifiers to promote safe and grounded outputs.

Estimated model parameter counts exceed one trillion, surpassing GPT-3 (175B) and prior GPT-4 variants, though precise configuration is unpublished. Notable architectural changes include refined attention mechanisms for ambiguous/conflicted inputs, cross-modal embedding heads, and infrastructure for real-time inference and streaming outputs (OpenAI et al., 2024, Shahriar et al., 2024).

2. Multimodal Capabilities and Performance Across Domains

Text and Language

On English text, GPT-4o matches GPT-4 Turbo; code generation accuracy (pass@1/10/100) is statistically equivalent. Notable non-English gains are observed: ARC-Easy (Hausa) accuracy rises from ~6% (GPT-3.5) to 75.4% (GPT-4o), reducing cross-lingual disparities by >30 percentage points (OpenAI et al., 2024). In USMLE, CFA, SAT, and Bar Exam standardized language evaluations, accuracy ranges from 75% to 90%, often slightly below GPT-4 for domain-specialized questions but ahead of GPT-3.5.

Reasoning and Few-shot Learning

Deductive, inductive, and abductive tasks (bAbI, EntailmentBank, CLUTRR, aNLI) confirm GPT-4o leads prior models (e.g., bAbI-15: 30/30 correct) and exhibits robust few-shot learning in structured settings. However, ambiguity sensitivity is observed: identical queries sometimes elicit inconsistent answers, and follow-up clarification requests may arise on ill-posed tasks (Shahriar et al., 2024).

Vision

The vision encoder matches GPT-4V in object recognition, OCR, and figure interpretation. On MM-Vet, GPT-4o attains 83.9% across recognition, spatial, math, and OCR tasks (vs 67.7% for GPT-4V). Classification scores on general images are typically high (e.g., fruit: F$_1$ = 0.98); in domain-specialized tasks (medical, agricultural), performance falls without additional tuning (glaucoma F$_1$ = 0.69, cancer F$_1$ = 0) (Shahriar et al., 2024). BLEU-4 for open-ended captioning remains low (0.031), indicating continued generation challenges.

Speech and Audio

GPT-4o processes speech-to-speech and audio-text tasks with sub-320 ms response latency, outperforming Whisper-LLaMA and relevant LALMs in ASR (10–30% relative WER reduction), spoken command, intent, and semantic/paralinguistic understanding across >10 languages (Lin et al., 14 Feb 2025). In reasoning-heavy audio tasks (MMAU), test accuracy reaches 60.5%, exceeding Gemini 1.5 Pro’s 53%. Hallucination resistance on multi-modal benchmarks (CMM HR = 83.8%) is substantially improved over baselines (34–59%) (Lin et al., 14 Feb 2025). Persistent refusal behavior is observed for potentially sensitive audio tasks (e.g., speaker identification, age/gender classification, deepfake detection), driven by strict post-training safeguards.

Integrated Multimodal Reasoning

Visual question answering (VQA), OCR+reasoning (e.g., math signage), and spatial tasks are leading strengths, with integrated pipelines yielding both accurate perception and explanation. Peak zero/few-shot VQA accuracy is ~36%, and MM-Vet sub-task scores are consistently highest among contemporaries (Shahriar et al., 2024).

3. Safety, Alignment, and Ethical Filtering

GPT-4o incorporates multi-layered safety mechanisms at both pre- and post-training stages. Input data is filtered for prohibited or sensitive content prior to training. After pretraining, outputs are further filtered or refused via RLHF-based alignment strategies and specialized classifiers that target:

• Unauthorized voice generation and voiceprint cloning (audio refusal: precision ≥ 0.96, recall = 1.0).
• Speaker identification/trait attribution (consistent refusal on speaker ID, age, and gender tasks).
• Ungrounded inferences and sensitive trait attributions (post-training safe response accuracy increased from 0.60 to 0.84).
• Refusal/compliance accuracy reached 0.98 (“should-refuse”) and 0.83 (“should-comply”) on held-out voice safety data (OpenAI et al., 2024, Lin et al., 14 Feb 2025).

External evaluations (e.g., METR, Apollo Research) identify low autonomy, minimal “scheming” capability, moderate theory-of-mind, and negligible dangerous capability advancement compared to earlier GPT-4 (OpenAI et al., 2024).

4. Domain-Specific Evaluations and Limitations

Educational Assessment

In blind, proof-based undergraduate algorithm exams at ETH Zürich, GPT-4o under zero-shot prompts produced LaTeX-formatted proofs but consistently failed to reach passing criteria:

Exam	Passing	GPT-4o Score (% total)	GPT-4o Student Quantile	o1-preview Score	o1-preview Student Quantile
First	50%	46%	17.7%	62%	36.0%
Second	60%	56%	6.5%	92%	58.1%

GPT-4o answers were characterized by unjustified claims (7/8 exercises), misleading arguments (5/8), and mathematical errors (6/8), often failing to fully derive key steps or introducing logically incoherent or spurious assertions. In contrast, the o1-preview model exceeded passing and student median levels, indicating rapid, but uneven, progress in this domain (Ding et al., 19 May 2025).

Speech and Audio Domain Pitfalls

Refusal rates are variable across tasks and evaluation protocols. For speaker verification: refusal is 0–10% on some datasets (“SUPERB SV”) but 80–100% on others, depending on prompt framing and input quality, indicating guardrail sensitivity. Instrument/genre classification, audio duration prediction, and singing synthesis/assessment all exhibit refusals or reduced accuracy, reflecting a conservative bias from post-training rubrics (Lin et al., 14 Feb 2025).

Ambiguous or Adversarial Inputs

Ambiguity in textual and multimodal prompts elicits inconsistent answers or requests for clarification; session-level reproducibility is impacted. Image classification generalizes poorly to visually similar, underrepresented domain classes (F$_1$ < 0.35 for crop disease subtypes, total failure on “cancer” class), demonstrating the limits of broad pretraining without targeted fine-tuning (Shahriar et al., 2024).

5. Deployment, Cost, and Open-Source Replication Attempts

GPT-4o delivers real-time, streaming inference—especially notable in speech-to-speech use—under 320 ms average latency on Azure-backed GPU clusters. Throughput is staged for streaming high-availability deployments. Pricing is 50% below GPT-4 Turbo per token at API release (OpenAI et al., 2024). Public accessibility spans native ChatGPT voice mode, programmatic API, and SDK endpoints for full multimodal tasks.

Open-source attempts to approximate GPT-4o's core architecture include Mini-Omni2, which stitches pretrained CLIP and Whisper encoders with a compact LLM backbone (Qwen2-0.5B) and command-based full-duplex interaction. Mini-Omni2 achieves competitive ASR WER and robust modality alignment, but is limited by model/data scale, basic output control, and lacks advanced audio style modeling or semantic interruption detection (Xie et al., 2024).

6. Comparative Assessments, Strengths, and Known Gaps

GPT-4o establishes new state-of-the-art results in integrated language, vision-language, and multimodal reasoning, with notable performance in multilingual benchmarks and reasoning-heavy domains. It matches or exceeds proprietary and open-source predecessors on MM-Vet, MMAU, and several standardized language exams. Key strengths include real-time cross-modal inference, robust few-shot learning, and leading safety architecture.

Documented limitations include inconsistent handling of ambiguous inputs, refusal overshoot on benign audio tasks, lack of specialized domain generalization (especially for medical/agricultural vision), moderate speech understanding biases, and low coherence in open-ended generation (captioning, VQA free-form). Authors across multiple studies urge the development of richer, human-in-the-loop benchmarks, refined few-shot and prompt-calibration strategies, and domain-specific fine-tuning pipelines to close these gaps (Shahriar et al., 2024, Lin et al., 14 Feb 2025, Ding et al., 19 May 2025).

7. Future Directions and Research Frontiers

Recommendations for future work converge on several axes:

• Intensifying multimodal evaluation protocols to disentangle refusal vs. genuine error, with expanded datasets for underrepresented sub-tasks and longitudinal, real-time benchmarks.
• Public releases of architecture and audio encoder details for transparency and replicability.
• Finer-grained refusal calibration (e.g., via R-Tuning) to decrease false-positive refusals on benign tasks while preserving safety.
• Extended post-training pipelines incorporating diverse feedback and human judgment, particularly for ambiguous or adversarial prompts.
• Scaling open-source replications, integrating full-duplex dialogue and nuanced audio style, and promoting standardized leaderboards aligning with proprietary benchmarks (Xie et al., 2024, Lin et al., 14 Feb 2025, Shahriar et al., 2024).

GPT-4o thus represents a reference point for unified, instruction-based multimodal AI, combining efficient language reasoning, advanced perceptual capabilities, and sophisticated safety frameworks, while presenting unresolved research challenges in calibration, domain adaptation, and robust, reproducible evaluation.