alperiox

Draft Example

Fri, 01 Jul 2022 00:00:00 GMT

This Article is a Draft

This article is currently in a draft state and is not published. Therefore, it will not be visible to the general audience. The content is still a work in progress and may require further editing and review.

When the article is ready for publication, you can update the "draft" field to "false" in the Frontmatter:

---
title: Draft Example
published: 2024-01-11T04:40:26.381Z
tags: [Markdown, Blogging, Demo]
category: Examples
draft: false
---

Training YOLOv9 for Detecting Firearms

Thu, 14 Mar 2024 00:00:00 GMT

I try to build some simple projects when new YOLO versions come out. This time, I tried to build a firearm detector.

As always, the most difficult aspect of training YOLO models is preparing the dataset (at least for me). So this time too, I had too much work to do to prepare the dataset. While checking out datasets in Roboflow Universe, I knew that I needed a way to merge different datasets to increase the generalization power of the model. Unfortunately, there wasn't such a tool in the supervision module so I had to prepare one for myself. [^1] I started by defining some requirements:

I needed to be able to,

merge different datasets
group similar labels together
filter the labels that I don't want to include

and finally,

datasets must have a similar structure (I picked COCO by default)

But that wasn't enough. Since I had picked COCO as my default dataset structure, I needed to transform the merged dataset to YOLO's supported format to set up the training phase. So, I needed to use COCO's outdated Python API.

I have found a rather updated version of it, so forked it and updated the API to make it compatible with my current workflow. (my fork)

Most of the work was done in the dataset preparation, but I was able to implement the functionality to merge datasets while also merging the labels (or filtering the unwanted ones). Check the messy colab notebook here if you're interested. I plan to wrap it up into a simple script but in the future, please.

Training the model couldn't be easier, really. I used Piotr Skalski's YOLOv9 fork and simply trained the model for 10 epochs using 16 samples per batch and augmented the training samples using mozaic augmentation [^2]

The training results weren't as bad as I thought!

False negatives are critical in this project, as incorrect negative classification might have serious consequences. Thus, while the model had decent precision, having rather lower recall meant that there are impactful amount of false negatives. We can see that from the following confusion matrix as well

As a result, while the model does a decent job for a model trained in an hour or two, it's not ready to use in real-life applications where having false negatives would impact peoples lives.

TODO: model onnx'e cevrildi, calisiyoda. ama detection pipeline'in duzenlenmesi gerekiyor, ona bakip devam etmek gerek

But hey, we can still build a gradio app to showcase the model! It's live in huggingface

[^1]: there's a merge function in the supervision module, but it's just doesn't cover my use case where I want to merge multiple datasets while also filtering the respective classes. [^2]: Mosaic data augmentation simply combines some training samples into one in certain ratios, it was first introduced in YOLOv4 and it lets model to learn how to identify smaller objects.

Simple Guides for Fuwari

Mon, 01 Apr 2024 00:00:00 GMT

Cover image source: Source

This blog template is built with Astro. For the things that are not mentioned in this guide, you may find the answers in the Astro Docs.

Front-matter of Posts

---
title: My First Blog Post
published: 2023-09-09
description: This is the first post of my new Astro blog.
image: ./cover.jpg
tags: [Foo, Bar]
category: Front-end
draft: false
---

Attribute	Description
`title`	The title of the post.
`published`	The date the post was published.
`description`	A short description of the post. Displayed on index page.
`image`	The cover image path of the post.<br/>1. Start with `http://` or `https://`: Use web image<br/>2. Start with `/`: For image in `public` dir<br/>3. With none of the prefixes: Relative to the markdown file
`tags`	The tags of the post.
`category`	The category of the post.
`draft`	If this post is still a draft, which won't be displayed.

Where to Place the Post Files

Your post files should be placed in src/content/posts/ directory. You can also create sub-directories to better organize your posts and assets.

src/content/posts/
├── post-1.md
└── post-2/
    ├── cover.png
    └── index.md

Markdown Extended Features

Wed, 01 May 2024 00:00:00 GMT

GitHub repository cards

You can add dynamic cards that link to GitHub repositories, on page load, the repository information is pulled from the GitHub API.

::github{repo="Fabrizz/MMM-OnSpotify"}

Create a GitHub repository card with the code ::github{repo="<owner>/<repo>"}.

::github{repo="saicaca/fuwari"}

Admonitions

Following types of admonitions are supported: note tip important warning caution

:::note Highlights information that users should take into account, even when skimming. :::

:::tip Optional information to help a user be more successful. :::

:::important Crucial information necessary for users to succeed. :::

:::warning Critical content demanding immediate user attention due to potential risks. :::

:::caution Negative potential consequences of an action. :::

:::note
Highlights information that users should take into account, even when skimming.
:::

:::tip
Optional information to help a user be more successful.
:::

The title of the admonition can be customized.

:::note[MY CUSTOM TITLE] This is a note with a custom title. :::

:::note[MY CUSTOM TITLE]
This is a note with a custom title.
:::

[!TIP] The GitHub syntax is also supported.

> [!NOTE]
> The GitHub syntax is also supported.

> [!TIP]
> The GitHub syntax is also supported.

Markdown Example

Sun, 01 Oct 2023 00:00:00 GMT

An h1 header

Paragraphs are separated by a blank line.

2nd paragraph. Italic, bold, and monospace. Itemized lists look like:

this one
that one
the other one

Note that --- not considering the asterisk --- the actual text content starts at 4-columns in.

Block quotes are written like so.

They can span multiple paragraphs, if you like.

Use 3 dashes for an em-dash. Use 2 dashes for ranges (ex., "it's all in chapters 12--14"). Three dots ... will be converted to an ellipsis. Unicode is supported. ☺

An h2 header

Here's a numbered list:

first item
second item
third item

Note again how the actual text starts at 4 columns in (4 characters from the left side). Here's a code sample:

# Let me re-iterate ...
for i in 1 .. 10 { do-something(i) }

As you probably guessed, indented 4 spaces. By the way, instead of indenting the block, you can use delimited blocks, if you like:

define foobar() {
    print "Welcome to flavor country!";
}

(which makes copying & pasting easier). You can optionally mark the delimited block for Pandoc to syntax highlight it:

import time
# Quick, count to ten!
for i in range(10):
    # (but not *too* quick)
    time.sleep(0.5)
    print i

An h3 header

Now a nested list:

First, get these ingredients:
- carrots
- celery
- lentils
Boil some water.

Dump everything in the pot and follow this algorithm:

 find wooden spoon
 uncover pot
 stir
 cover pot
 balance wooden spoon precariously on pot handle
 wait 10 minutes
 goto first step (or shut off burner when done)

Do not bump wooden spoon or it will fall.

Notice again how text always lines up on 4-space indents (including that last line which continues item 3 above).

Here's a link to a website, to a local doc, and to a section heading in the current doc. Here's a footnote [^1].

[^1]: Footnote text goes here.

Tables can look like this:

size material color

9 leather brown 10 hemp canvas natural 11 glass transparent

Table: Shoes, their sizes, and what they're made of

(The above is the caption for the table.) Pandoc also supports multi-line tables:

keyword text

red Sunsets, apples, and other red or reddish things.

green Leaves, grass, frogs and other things it's not easy being.

A horizontal rule follows.

Here's a definition list:

apples : Good for making applesauce. oranges : Citrus! tomatoes : There's no "e" in tomatoe.

Again, text is indented 4 spaces. (Put a blank line between each term/definition pair to spread things out more.)

Here's a "line block":

| Line one | Line too | Line tree

and images can be specified like so:

Inline math equations go in like so: $\omega = d\phi / dt$. Display math should get its own line and be put in in double-dollarsigns:

$$I = \int \rho R^{2} dV$$

And note that you can backslash-escape any punctuation characters which you wish to be displayed literally, ex.: `foo`, *bar*, etc.

RAG for Q&A with PDFs

Fri, 07 Jun 2024 00:00:00 GMT

In order to improve my Upwork profile a little, I grabbed the most successful freelancers in AI and checked their projects. One of them was building a retrieval system for question answering with PDFs, so here we are.

I always lean on cloud providers when it comes to LLMs. This time too, I used Gemini with GCP. The project was rather simple, I needed to construct vector database, fetch documents and use them in the prompt that will generate the real answer. I also needed a graphical interface so I had to choose between Gradio or Streamlit.

My toolset was pretty simple, Langchain for utilizing the LLM and setting up the retrieval system and Streamlit for the user interface. I defined several variables to store the session data in Streamlit, specifically for storing the path of currently selected PDF, the list of uploaded documents, currently loaded vector store, and for the retrieval system.

I simply created a new Chroma vector database when a document is uploaded for the first time, the documents were splitted to chunks of 1000 characters and the embeddings were calculated using VertexAIs embedding model, textembedding-gecko. The database was also stored in the filesystem with a random uuid. Loading the corresponding database when it's selected in the user interface was simple, the important thing was to set up the LLM chain successfully so we could retrieve the data from the database. And thankfully it was also easy thanks to Langchain.

And in the end, here's the finished version of it:

The project is not deployed on anywhere, but is available on github.

TinyAya Expedition: S2S Translation from a Text Backbone?

Fri, 10 Jul 2026 00:00:00 GMT

I had the chance to lead an expedition team in TinyAya Expedition of Cohere, here's my experience on building a Turkish - Hindi speech to speech translation model, from scratch (?).

The Premise

I was pretty interested in Speech to Speech models and was thinking of working on some more modeling in my spare time, while looking around for what to build, I saw TinyAya Expedition; an expedition where teams are trying to build something out of TinyAya models. The model doesn't support multimodality but supports 70+ languages, so figured it'd be a fun project to try to make it S2S.

Thankfully when I proposed the project there were some people that are interested in it, so we met, tried to come up with a plan, and then started working. Since our members were either Hindi or Turkish, we picked those two for our translation task.

The question was if a text-only multilingual model would learn speech-to-speech translation without explicit and massive audio pre-training beforehand. We picked translation because we wanted it to have a purpose, and was curious if TinyAya's language support would help it learn the task.

Our plan was pretty simple though, adapt Moshi's architecture, swap the backbone with TinyAya, generate data and then train!

Applying the plan though... wasn't so simple.

1: The Codec Dead End

Our first major decision was whether to fine-tune Mimi (Moshi's audio codec) on Hindi/Turkish speech or use it off-the-shelf.

So we spent some time on benchmarking several codecs, tried finetuning Mimi (Moshi's codec), DualCodec, and while there were several more codecs on our list, we weren't able to finish benchmarking all of them. Mainly because we were busy with our daily work, the expedition was a side project, after all.

Though at the end, we couldn't see any marginal improvement on our target languages when we tried finetuning Mimi and DualCodec. While i'm sure our training setup wasn't optimized for increasing their performance on the languages we picked, the diminishing results from the training indicated we're already good to go with the base Mimi:

Codec	OOD STOI	OOD PESQ-wb	OOD DNSMOS	Verdict
Base Mimi	0.921	3.31	2.87	Best generalization
Fine-tuned Mimi	0.825	2.27	2.76	Overfitted, degraded
DualCodec	0.881	2.40	2.95	Best perceptual, worst signal

Decision: Use base Mimi unmodified. This was the right call — it also meant Moshiko's pretrained depth decoder weights worked without adaptation. So... less parameters to optimize.

What we'd do differently: Skip the codec fine-tuning entirely. The quality we were getting from the base codecs were good-enough already and we neither had training compute nor time to work on the project exhaustively. We should have recognized this from the start and saved the time.

2: Data Generation — Cheap but Messy

No TR<->HI parallel speech corpus exists, so we built one. This part actually worked well — eventually.

TTS Model Selection

We evaluated four TTS models: XTTSv2, Chatterbox, Fish Speech, and OmniVoice.

Our setup was simple, collect Turkish or Hindi text from online datasets or generate the text -> translate using Command from Cohere if we don't have the Turkish<->Hindi pair -> generate speech -> check the generated speech.

The best decision was using Vast.ai, apparently it's possible to manage bunch of instances through an api key, so with limited scopes, it's also possible to come up with a simple MCP that manages these instances and get you the cheapest instances possible, I have a small mcp server for it, so we used that one: vast-mcp

I guess the second best decision here was using OmniVoice voice design, it supports structured text descriptions like [gender]female[pitch]moderate[accent]Indian so it was possible to expand our dataset to variety of speakers. We had tried simple voice cloning but it was fairly slow compared to designing the voice. But it had its setbacks too, some text descriptions we've used throughout the data generation was producing erroneous samples pretty frequently.

At the end, we designed 14 voice profiles and deployed 42 Vast.ai GPU instances, cost ~$50 for 911 hours and 1.3M clips of data.

Quality Control

Of course since we're generating synthetic data, we had to come up with a quality control mechanism, we built sound-quality-check, a simple round-trip ASR validation pipeline: generate audio → transcribe back with Whisper → compare against original text. 86% pass rate. The 14% failures concentrated in extreme voice designs (male_old_deep had the highest rejection rate) and very long sentences.

3: Architecture — Getting the Details Wrong

The architecture itself (Moshi-style three-stream with TinyAya backbone) was sound. The implementation had bugs, because debugging all those stuff after my 12+ hr shifts were a bit tiring, honestly. Here are some of the bugs we found out and fixed along the way:

The Off-by-One That Looked Fine

The composite model's depth decoder position mapping was initially wrong: position 0 (text prediction) was matched against CB0 targets. Total training loss was 0.007 — looked perfect. But per-codebook accuracy revealed every single codebook was misaligned by one position. Aggregate metrics hid a fundamental bug.

The Silent Collator Drop

The batch collator silently dropped user_audio_codes and model_audio_codes during batching. Training fell back to single-stream mode with model_audio_embed receiving zero gradients. We completed an entire test run before discovering this. But that's why we have test runs, right?

4: Multi-GPU Training — FSDP Hell

We went through 8 distinct FSDP checkpoint failures before save/load/resume worked especially in TPUs:

model_audio_embed not included in the checkpoint
FSDP shards saved instead of full tensors (needed summon_full_params)
save_pretrained calling allgather on rank 0 only (all ranks must participate)
Optimizer state not gathered properly (needed full_optim_state_dict)
Resume loading model weights after FSDP wrapping (must load before)
Audio demo generation causing NCCL timeout (rank 0 doing inference while rank 1 waits)
HuggingFace Hub push causing rank desync
Disk filling up during training (no checkpoint pruning)

Each of these caused a failed training run. Some lost hours of compute; one filled a 2TB disk.

The core lesson: With FSDP, any operation that touches model parameters must involve all ranks. Logging, saving, pushing, generating samples — anything that only runs on rank 0 will deadlock the other ranks. We eventually wrote a validate_full_pipeline.sh that tests train → save → load → forward → resume → save → compare. This should have existed before the first real training run.

5: TPU Training — A Different Kind of Pain

The TPU adaptation (for Google TRC-granted v6e chips) introduced a new class of issues:

4-hour backward compile: scan_layers TypeError on torch_xla 2.9 forced a manual loop fallback, which meant XLA couldn't fuse the backward pass. First backward step: 4 hours 25 minutes. After the fix: ~25 minutes.
IP quota exhaustion: v6e-64 (8-host slice) needs exactly 8 external IPs, hitting GCP's 8-IP regional limit. We pivoted to single-host v6e-8 to avoid multi-host complexity entirely.
Attention mask NaN: v6e bf16 turned -inf attention mask values into NaN. Fix: clamp masks to >= -1e4.
No TPU memory reporting: torch_xla memory info returns None. We had no visibility into HBM usage until Mayank added per-chip telemetry.

What we'd do differently: Start with single-host TPU from the beginning. Multi-host XLA introduces an entirely separate class of problems (FSDPv2 sharding, IP quotas, multi-host rendezvous). Single-host v6e-8 was sufficient and avoided all of this.

Why TPUs though?

Duh, we didn't have any GPU compute. I was going to train that model regardless of the compute type, we found TPUs so we supported TPUs.

6: The Text Stream Revelation

Honestly not much of a revelation, available text stream would carry over some information regarding to the speech tokens, Moshi's paper calls this the "inner monologue" — a parallel text stream that anchors the model's hidden states in language meaning.

But one of our buggy runs helped us to validate it, we didn't have text alignments in the training run, so the model fell back to audio-only loss and yeah, that didn't help at all.

Run	Text Loss	Audio Loss	Text Acc
Broken (no alignments)	12.51 (random)	6.1 (plateaued)	0%
Working (with alignments)	7.2	6.03	learning

This was later confirmed at scale: Mayank's v0.3 sweep on the full 1.24M corpus showed text accuracy reaching 77% (ppl ~3.25) after just 1 epoch, with audio still improving. The model nearly solved text prediction while audio was still catching up — exactly what you'd expect if TinyAya's multilingual text knowledge was transferring.

The Autoregressive Evaluation Bug

During pipeline validation for the production run, an off-by-one was found in eval_checkpoint.py: the AR loop read hidden[t] (which predicts position t+1 under the training convention) but wrote the token at position t. Every generated frame was shifted one position AND conditioned on a placeholder token.

This means every AR/ASR-BLEU number previously reported understated actual quality. The bug lived only in the evaluation harness — the model and training pipeline were never affected.

Lesson: For the love of god, please, please validate AI agent's results.

What We Got Right

Despite the failures, the core hypothesis held:

Moshi's architecture is modular. The depth decoder transferred across backbones without retraining. This means teams can focus compute on the temporal backbone and reuse pretrained acoustic modeling.
Data generation scales cheaply. $50 for 911 hours using OmniVoice voice design mode. The bottleneck is compute for training, not data.
Text-only backbones can learn audio. TinyAya went from zero audio experience to 27% CB0 accuracy on the full corpus — 544x above random chance — in one epoch.
The text stream provides real semantic grounding. 77% text accuracy after 1 epoch confirms that TinyAya's multilingual knowledge transfers to the audio modality.

What Remains Open

Translation quality hasn't emerged yet. The model learns language identity (Hindi output for TR→HI, Turkish for HI→TR) but sentence-level translation hasn't been verified at scale. The 3-epoch production run should answer this.
Uniform vs Whisper alignments. We generated uniform alignments (words spread evenly across duration) as a quick fix. Whisper-based word-level timestamps would be more accurate for natural-sounding TTS output. The current results show uniform alignments work, but we don't know how much better Whisper-based would be.
The D+E combination. The sweep winner (arm D: LoRA on all 36 layers) and runner-up (arm E: stronger regularization) were never tested together. This is the obvious next hyperparameter probe.

The Real Lesson

The biggest time sink wasn't any single bug — it was the compounding effect of silent failures. The collator dropping tensors, the text padding drowning real signal, the alignment data missing from the upload, the AR evaluation off-by-one. Each one looked fine from the outside. Loss was decreasing. No errors in the logs. Everything appeared to work.

Almost all of us were using AI agents to write the pieces of the code for us. While coding ourselves, our intuition about the subject helps us tremendously while finding out bugs and errors along the pipeline, your agent doesn't have that. It doesn't know what to validate, what to check, and how to check. The suggestions it comes up within its loop doesn't make sense at times.

I think this kind of supports the idea of AI being just a tool, while it helps tremendously, it doesn't have the unconscious intuition you have about the subject, so without validating the steps, or without installing validation steps along the way, it loses it somewhere in the loop. Without knowing the correct validation steps, or little sanity checks, I'd lose it too, though.

I built GPT from scratch (a very, very small version of it)

Sun, 21 Jul 2024 00:00:00 GMT

I was following Andrej Karpathy's wonderful series about deep learning on YouTube when I saw the videos about implementing GPT and GPT-2. Just how cool is that?

To be honest, I was pretty excited to get there since I always wanted to dive deeper into such concepts. I'll try to explain how this stuff works internally in this piece, hopefully with some visualizations as well.

Include Video in the Posts

Tue, 01 Aug 2023 00:00:00 GMT

Just copy the embed code from YouTube or other platforms, and paste it in the markdown file.

---
title: Include Video in the Post
published: 2023-10-19
// ...
---

<iframe width="100%" height="468" src="https://www.youtube.com/embed/5gIf0_xpFPI?si=N1WTorLKL0uwLsU_" title="YouTube video player" frameborder="0" allowfullscreen></iframe>

YouTube

Bilibili

Personal Protective Equipment detection with YOLOv10

Wed, 29 May 2024 00:00:00 GMT

Introduction

Colab Notebook for training

Paper

As always, I built a simple model as the new YOLO version came out. This time, it was personal protective equipment detection.

The YOLOv10's modeling approach is just so cool! We usually get multiple assignments per instance in the given object, resulting in a one-to-many mapping. While this approach results in more supervisory signals, it also introduces inefficiencies in inference. Because we would need to apply an algorithm like NMS to filter out redundant predictions.

So... YOLOv10 included a one-to-one mapping head to the model along with a one-to-many head. The authors introduce a consistent matching metric to quantitatively assess the level of concordance between predictions and instances, this metric also includes two hyperparameters. Then they derived a metric from the Wasserstein-1 metric including the consistent matching metric to determine the supervision gap between two heads.

And in the end, it was possible to optimize the one-to-one head in the direction of the one-to-many head's optimization. Because the derived distance converges to 0 as the one-to-one head's prediction goes up in the ranks of one-to-many head's outputs. If they also choose the consistent matching metric's hyperparameters well, then mathematically the best output from one-to-many is the best for one-to-one as well. This means that they can train the model using richer signals from one-to-many head and optimize the one-to-one head as well. This approach removes the need for NMS in the inference phase, reducing the potential performance drawbacks.

I wanted to highlight a few things before moving on to my project. Because damn, training YOLO models are easy. Like really, really easy. Often, the most cumbersome part of the project is the data preparation phase, which is also much easier thanks to Roboflow Universe. So all you need to do is pick a dataset, download it in YOLO format, and run the training script.

Well, I just... did that too. Picked a dataset for PPE detection and ran the training script. I usually convert the model to ONNX before trying to deploy it, it was the same this time too. I prepared a simple demo as a Huggingface Space. The training was pretty successful, as I trained the model with default hyperparameters for just 20 epochs.

Results

It's easy to see that predicting a bounding box when the person doesn't have a glove, goggles, hard hat, mask or safety vest introduces a difficulty in predicting other labels. I think that's because there would be multiple similar bounding boxes with different labels.

I can imagine why Masks are often confused with NO-Hardhat boxes, probably because of the data labeling part. When I checked the initial dataset, I could see there are similar images where Mask or NO-Hardhat labels both fit well. Like:

Mask	NO-Hardhat

Like, NO-Hardhat could be easily determined as Mask by a human like me. So I don't put it on the algorithm.

In the end, the model trained well but I think it could use some more training steps. I'm happy with the results though.