· SPEECH
The Voice Came Loose. The state of speech AI: cloning, codecs and super-resolution
01
Three seconds
In January 2023 Microsoft’s VALL-E cloned a voice from a three-second sample. It treated speech like text: tokens from a neural audio codec, predicted one after another. Codecs like SoundStream and EnCodec made that possible, and the framing took over the field.
By 2026 zero-shot cloning is a product feature: a few seconds of audio carries timbre, accent and rhythm, often across languages.
02
Beyond tokens
Newer systems drop discrete tokens. VoxCPM2, released this year, works in a continuous latent space. It encodes at 16 kHz and decodes at 48 kHz, so super-resolution comes free. It was trained on over two million hours in 30 languages, with open weights.
03
Editing, not recording
The bigger shift is control. Research codecs try to separate timbre from prosody: keep the speaker, change the delivery. SpeechX folds noise removal, speaker extraction and word-level editing into one model.
Each edit is a fresh generation, not a lossless one, but the ear accepts it as continuous. Audio is now as editable as text. Fix a fluffed line after the fact and nobody hears the seam.
04
Real time
Native speech-to-speech models like Kyutai’s Moshi answer in around 200 milliseconds and listen while they talk. Most products still chain recognition, a language model and synthesis, trading speed for control. That is now a design choice, not a research wall.
05
Aura
In 1935 Walter Benjamin wrote that mechanical reproduction strips a work of art of its aura, its presence in one time and place. Recording did that to the voice. Generation finishes the job. A voice no longer implies a throat.
I find that unsettling and beautiful. The voice is an instrument anyone can play, and who is speaking becomes a question of context, not sound.