Walrus

Overview

We present Walrus, a unified 8B speech-and-audio foundation model that performs speech recognition, speech translation, speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio understanding.

Our central contribution is not architectural. We build Walrus on a standard decoder-only backbone initialized from Gemma-4 and keep the architecture deliberately simple, so that every gain can be attributed to the data. The core of our method is a three-stage data curriculum that progresses from a broad multilingual foundation to increasingly complex multi-talker, code-switching, and domain-specific data, paired with task- and dataset-aware weighting.

Trained on over 600M samples spanning more than 20 tasks and 100 languages, Walrus leads on multilingual ASR, code-switching ASR, speech-to-text/speech translation, and audio understanding, surpassing strong open and commercial systems.

Architecture

Walrus keeps the backbone deliberately simple: a decoder-only Transformer initialized from Gemma-4-E4B. Speech is treated as a sequence of discrete tokens (8192 dedicated speech tokens), and task selection is controlled entirely through natural-language prompts. For generation, Walrus emits low-bitrate discrete speech tokens (325 bps) that a lightweight token-to-waveform decoder renders into high-fidelity audio.

Figure 1. The Walrus architecture.

Main Results

Performance across all evaluated benchmarks. For each dataset we report the best prior system among comparable 4-10B models (Gemma-4, Phi-4-MM, Audio-Flamingo-3, Qwen2.5-Omni), the Commercial-Ready Walrus (Walrus-CR) and the Research Walrus (Walrus-R). Metrics: WER/CER (↓) for ASR, cpWER/cpCER (↓) for MT-ASR, BLEU/ASR-BLEU (↑) for AST/S2ST, accuracy (↑) for IF and SER, CIDEr/CLAP-Sim (↑) for captioning, R@1/5/10 (↑) for retrieval. Best results per dataset in green/bold.

Benchmark Overview

Walrus against the best previous system (among AF3, Gemma-4, Qwen2.5-Omni-7B, Phi4-MM) across recognition, translation, and understanding benchmarks. Scores are normalized so higher is better. Both Walrus variants advance the frontier on most axes, and the Research variant extends it further, most notably on code-switching, internal-domain ASR, and audio understanding.

Figure 2. Walrus vs. the best prior system across task families.

Data Curriculum

The data curriculum is the core contribution of Walrus. Recognition dominates the mix (45%), followed by translation (27%) and understanding (21%), while generation is deliberately kept small (7%). This distribution reflects our data-centric design: the core recognition skills are heavily represented to protect them from forgetting, while token-heavy generative tasks are downweighted.

(a) Overall task distribution across the four families.

(b) Detailed breakdown within recognition, translation, and understanding.

Audio Samples

Below are qualitative examples of Walrus across some of the supported tasks.

Citation

If you find Walrus useful in your research, please cite:

@article{sonyspeechwalrus,
  title  = {Walrus Technical Report},
  author = {Koudounas, Alkis and Take, Osamu and Futami, Hayato and Jodelet, Quentin and Tsunoo, Emiru},
  year   = {2026},
  institution = {Sony Group Corporation}
}