I had the chance to lead an expedition team in TinyAya Expedition of Cohere, here’s my experience on building a Turkish - Hindi speech to speech translation model, from scratch (?).

The Premise#

I was pretty interested in Speech to Speech models and was thinking of working on some more modeling in my spare time, while looking around for what to build, I saw TinyAya Expedition; an expedition where teams are trying to build something out of TinyAya models. The model doesn’t support multimodality but supports 70+ languages, so figured it’d be a fun project to try to make it S2S.

Thankfully when I proposed the project there were some people that are interested in it, so we met, tried to come up with a plan, and then started working. Since our members were either Hindi or Turkish, we picked those two for our translation task.

The question was if a text-only multilingual model would learn speech-to-speech translation without explicit and massive audio pre-training beforehand. We picked translation because we wanted it to have a purpose, and was curious if TinyAya’s language support would help it learn the task.

Our plan was pretty simple though, adapt Moshi’s architecture, swap the backbone with TinyAya, generate data and then train!

Applying the plan though… wasn’t so simple.

1: The Codec Dead End#

Our first major decision was whether to fine-tune Mimi (Moshi’s audio codec) on Hindi/Turkish speech or use it off-the-shelf.

So we spent some time on benchmarking several codecs, tried finetuning Mimi (Moshi’s codec), DualCodec, and while there were several more codecs on our list, we weren’t able to finish benchmarking all of them. Mainly because we were busy with our daily work, the expedition was a side project, after all.

Though at the end, we couldn’t see any marginal improvement on our target languages when we tried finetuning Mimi and DualCodec. While i’m sure our training setup wasn’t optimized for increasing their performance on the languages we picked, the diminishing results from the training indicated we’re already good to go with the base Mimi:

Decision: Use base Mimi unmodified. This was the right call — it also meant Moshiko’s pretrained depth decoder weights worked without adaptation. So… less parameters to optimize.

What we’d do differently: Skip the codec fine-tuning entirely. The quality we were getting from the base codecs were good-enough already and we neither had training compute nor time to work on the project exhaustively. We should have recognized this from the start and saved the time.

2: Data Generation — Cheap but Messy#

No TR<->HI parallel speech corpus exists, so we built one. This part actually worked well — eventually.

TTS Model Selection#

We evaluated four TTS models: XTTSv2, Chatterbox, Fish Speech, and OmniVoice.

Our setup was simple, collect Turkish or Hindi text from online datasets or generate the text -> translate using Command from Cohere if we don’t have the Turkish<->Hindi pair -> generate speech -> check the generated speech.

The best decision was using Vast.ai, apparently it’s possible to manage bunch of instances through an api key, so with limited scopes, it’s also possible to come up with a simple MCP that manages these instances and get you the cheapest instances possible, I have a small mcp server for it, so we used that one: vast-mcp

I guess the second best decision here was using OmniVoice voice design, it supports structured text descriptions like [gender]female[pitch]moderate[accent]Indian so it was possible to expand our dataset to variety of speakers. We had tried simple voice cloning but it was fairly slow compared to designing the voice. But it had its setbacks too, some text descriptions we’ve used throughout the data generation was producing erroneous samples pretty frequently.

At the end, we designed 14 voice profiles and deployed 42 Vast.ai GPU instances, cost ~$50 for 911 hours and 1.3M clips of data.

Quality Control#

Of course since we’re generating synthetic data, we had to come up with a quality control mechanism, we built sound-quality-check, a simple round-trip ASR validation pipeline: generate audio → transcribe back with Whisper → compare against original text. 86% pass rate. The 14% failures concentrated in extreme voice designs (male_old_deep had the highest rejection rate) and very long sentences.

3: Architecture — Getting the Details Wrong#

The architecture itself (Moshi-style three-stream with TinyAya backbone) was sound. The implementation had bugs, because debugging all those stuff after my 12+ hr shifts were a bit tiring, honestly. Here are some of the bugs we found out and fixed along the way:

The Off-by-One That Looked Fine#

The composite model’s depth decoder position mapping was initially wrong: position 0 (text prediction) was matched against CB0 targets. Total training loss was 0.007 — looked perfect. But per-codebook accuracy revealed every single codebook was misaligned by one position. Aggregate metrics hid a fundamental bug.

The Silent Collator Drop#

The batch collator silently dropped user_audio_codes and model_audio_codes during batching. Training fell back to single-stream mode with model_audio_embed receiving zero gradients. We completed an entire test run before discovering this. But that’s why we have test runs, right?

4: Multi-GPU Training — FSDP Hell#

We went through 8 distinct FSDP checkpoint failures before save/load/resume worked especially in TPUs:

1. model_audio_embed not included in the checkpoint
2. FSDP shards saved instead of full tensors (needed summon_full_params)
3. save_pretrained calling allgather on rank 0 only (all ranks must participate)
4. Optimizer state not gathered properly (needed full_optim_state_dict)
5. Resume loading model weights after FSDP wrapping (must load before)
6. Audio demo generation causing NCCL timeout (rank 0 doing inference while rank 1 waits)
7. HuggingFace Hub push causing rank desync
8. Disk filling up during training (no checkpoint pruning)

Each of these caused a failed training run. Some lost hours of compute; one filled a 2TB disk.

The core lesson: With FSDP, any operation that touches model parameters must involve all ranks. Logging, saving, pushing, generating samples — anything that only runs on rank 0 will deadlock the other ranks. We eventually wrote a validate_full_pipeline.sh that tests train → save → load → forward → resume → save → compare. This should have existed before the first real training run.

5: TPU Training — A Different Kind of Pain#

The TPU adaptation (for Google TRC-granted v6e chips) introduced a new class of issues:

• 4-hour backward compile: scan_layers TypeError on torch_xla 2.9 forced a manual loop fallback, which meant XLA couldn’t fuse the backward pass. First backward step: 4 hours 25 minutes. After the fix: ~25 minutes.
• IP quota exhaustion: v6e-64 (8-host slice) needs exactly 8 external IPs, hitting GCP’s 8-IP regional limit. We pivoted to single-host v6e-8 to avoid multi-host complexity entirely.
• Attention mask NaN: v6e bf16 turned -inf attention mask values into NaN. Fix: clamp masks to >= -1e4.
• No TPU memory reporting: torch_xla memory info returns None. We had no visibility into HBM usage until Mayank added per-chip telemetry.

What we’d do differently: Start with single-host TPU from the beginning. Multi-host XLA introduces an entirely separate class of problems (FSDPv2 sharding, IP quotas, multi-host rendezvous). Single-host v6e-8 was sufficient and avoided all of this.

Why TPUs though?#

Duh, we didn’t have any GPU compute. I was going to train that model regardless of the compute type, we found TPUs so we supported TPUs.

6: The Text Stream Revelation#

Honestly not much of a revelation, available text stream would carry over some information regarding to the speech tokens, Moshi’s paper calls this the “inner monologue” — a parallel text stream that anchors the model’s hidden states in language meaning.

But one of our buggy runs helped us to validate it, we didn’t have text alignments in the training run, so the model fell back to audio-only loss and yeah, that didn’t help at all.

This was later confirmed at scale: Mayank’s v0.3 sweep on the full 1.24M corpus showed text accuracy reaching 77% (ppl ~3.25) after just 1 epoch, with audio still improving. The model nearly solved text prediction while audio was still catching up — exactly what you’d expect if TinyAya’s multilingual text knowledge was transferring.

The Autoregressive Evaluation Bug#

During pipeline validation for the production run, an off-by-one was found in eval_checkpoint.py: the AR loop read hidden[t] (which predicts position t+1 under the training convention) but wrote the token at position t. Every generated frame was shifted one position AND conditioned on a placeholder token.

This means every AR/ASR-BLEU number previously reported understated actual quality. The bug lived only in the evaluation harness — the model and training pipeline were never affected.

Lesson: For the love of god, please, please validate AI agent’s results.

What We Got Right#

Despite the failures, the core hypothesis held:

1. Moshi’s architecture is modular. The depth decoder transferred across backbones without retraining. This means teams can focus compute on the temporal backbone and reuse pretrained acoustic modeling.

2. Data generation scales cheaply. $50 for 911 hours using OmniVoice voice design mode. The bottleneck is compute for training, not data.

3. Text-only backbones can learn audio. TinyAya went from zero audio experience to 27% CB0 accuracy on the full corpus — 544x above random chance — in one epoch.

4. The text stream provides real semantic grounding. 77% text accuracy after 1 epoch confirms that TinyAya’s multilingual knowledge transfers to the audio modality.

What Remains Open#

• Translation quality hasn’t emerged yet. The model learns language identity (Hindi output for TR→HI, Turkish for HI→TR) but sentence-level translation hasn’t been verified at scale. The 3-epoch production run should answer this.
• Uniform vs Whisper alignments. We generated uniform alignments (words spread evenly across duration) as a quick fix. Whisper-based word-level timestamps would be more accurate for natural-sounding TTS output. The current results show uniform alignments work, but we don’t know how much better Whisper-based would be.
• The D+E combination. The sweep winner (arm D: LoRA on all 36 layers) and runner-up (arm E: stronger regularization) were never tested together. This is the obvious next hyperparameter probe.

The Real Lesson#

The biggest time sink wasn’t any single bug — it was the compounding effect of silent failures. The collator dropping tensors, the text padding drowning real signal, the alignment data missing from the upload, the AR evaluation off-by-one. Each one looked fine from the outside. Loss was decreasing. No errors in the logs. Everything appeared to work.

Almost all of us were using AI agents to write the pieces of the code for us. While coding ourselves, our intuition about the subject helps us tremendously while finding out bugs and errors along the pipeline, your agent doesn’t have that. It doesn’t know what to validate, what to check, and how to check. The suggestions it comes up within its loop doesn’t make sense at times.

I think this kind of supports the idea of AI being just a tool, while it helps tremendously, it doesn’t have the unconscious intuition you have about the subject, so without validating the steps, or without installing validation steps along the way, it loses it somewhere in the loop. Without knowing the correct validation steps, or little sanity checks, I’d lose it too, though.