Spaces:
Running
Running
Commit History
-
Fix jiwer crash on post-normalisation empty refs; register SLR106/105 datasets
jefffffff9 +1: Claude Sonnet 4.6compute_metrics: pre-apply lowercasing + punctuation removal before filtering empty references — a label like "?" passes r.strip() but becomes "" after RemovePunctuation, causing jiwer to raise ValueError mid-evaluation. Pass already-normalised strings directly to jiwer.cer / jiwer.wer so no second transform is applied. web_harvester: add commented-in registry entries for OpenSLR SLR106 (10k clean Guinea Pular utterances, 49 speakers, CC BY-SA 4.0) and SLR105 (Guinea radio corpus, Pular-tagged validation split). Both are uncommented once the audio is uploaded to HF — best available Guinea Pular ASR data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>6682858
-
Add Adlam/Pular Fula integration: transliterator + 3 new datasets + normalisation pipeline
jefffffff9 +1: Claude Sonnet 4.6- src/data/adlam.py: full Adlam↔Latin transliterator for Pular (Guinea Fula) — bidirectional char map (U+1E900–U+1E95F), digraph-aware latin_to_adlam(), contains_adlam(), normalize_pular() (Adlam→Latin + NFC + lowercase) - src/data/web_harvester.py: add 3 Fula training sources to HF_DATASET_REGISTRY: Pullo-Africa-Protagonist/Fula-pular (9,761 audio rows, primary), guizme/adlam_fulfulde (51 Adlam-script rows, auto-converted to Latin) - notebooks/kaggle_master_trainer.ipynb: Cell 11 gets inline _normalize_pular (Adlam→Latin); Cell 12 applies it for lang=='ful' in prepare_dataset - app.py: import normalize_pular; apply in _do_asr and _convo_pipeline for Fula; update Self-Teaching tab help text to list all 3 Fula datasets Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>ced078c
-
Fix: add trust_remote_code=True for datasets 3.x compatibility
Broulaye Doumbia71bb3bc
-
Phase 3: Voice-to-Voice S2S pipeline — F5-TTS, LLM brain, CER metric
jefffffff9 +1: Claude Sonnet 4.6app.py: - Add LLM_MODEL_ID env var (default: Qwen/Qwen2.5-72B-Instruct) - Import GemmaClient + bam_normalize at module level - Add _voice_ref_path / _voice_ref_text / _llm_client state - Add set_voice_reference(): converts MP3->24kHz WAV, auto-transcribes - Add _convo_pipeline(): ASR -> bam_normalize -> LLM (phonetic Bambara system prompt) -> F5-TTS (voice ref) with MMS-TTS fallback - handle_ask() accepts convo_mode=bool, routes to _convo_pipeline or the original sensor pipeline accordingly - Tab 1 UI: Conversation Mode toggle, Voice Reference upload accordion, stop_recording auto-submit for true back-to-back loop src/tts/f5_tts.py (new): - Lazy-loaded F5TTS wrapper; synthesize() with ref_wav + ref_text - to_wav_24k() resampler (F5-TTS needs 24 kHz input) - Graceful fallback (returns None) when f5-tts not installed src/data/bam_normalize.py (new): - _bam_norm(): ou->u, dj->j, gn->ny, ch->c, oo->o-open, ee->e-open - Used at inference (app.py) and training (notebook Cell 11) req…8952fff
-
Fix Self-Teaching tab: float sliders, deduplication, Kaggle API fallback
jefffffff9 +1: Claude Sonnet 4.6- Cast slider values to int (Gradio returns float; broke Wikipedia aplimit and Kaggle ds.take() in notebook Cell 9) - Deduplicate dataset_sources.jsonl on each HF import (avoid loading same dataset multiple times per training run) - Kaggle trigger: try Python API (KaggleApiExtended) first to avoid binary PATH issues; subprocess + shutil.which as fallback - Fix Fula dataset registry: fleurs uses legacy script loader incompatible with datasets>=3.0; switch to google/WaxalNLP ful_asr - Fix UI description: HF import writes to dataset_sources.jsonl (streamed by Kaggle), not to corrections.jsonl Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>24b1617 -
Fix Wikipedia and HF harvest buttons
jefffffff9 +1: Claude Sonnet 4.6Wikipedia fix: - Add User-Agent header to all requests (Wikipedia API returns 403 without it) - Extract shared _upload_jsonl() helper to reduce code duplication HF harvest fix: - Remove audio re-upload approach (500 HTTP requests -> Gradio timeout) - Instead write dataset references to dataset_sources.jsonl on the Hub - One small file upload (~instant) instead of gigabytes of audio - Kaggle notebook Cell 4 now reads dataset_sources.jsonl dynamically: any dataset registered in Space Self-Teaching tab is automatically picked up at training time with no notebook edits needed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>a5737ac -
Add mass training + internet self-teaching features
jefffffff9 +1: Claude Sonnet 4.6New Tab 5 — Bulk Upload: - Upload ZIP of audio files + CSV (filename,transcription) in one operation - Batch-inserts all samples into corrections.jsonl + uploads audio to Hub New Tab 6 — Self-Teaching: - Wikipedia harvest: bm/ff Wikipedia API → sentence text → vocabulary.jsonl (868 Bambara articles, 17k+ Fula articles) - HF dataset import: RobotsMali/jeli-asr (33k Bambara) + google/fleurs ff_sn (Fula) → corrections.jsonl, one click - Auto-training trigger: fires Kaggle kernel via REST API when corrections.jsonl reaches AUTO_TRAIN_THRESHOLD (default 50) entries Requires KAGGLE_USERNAME + KAGGLE_KEY Space secrets New src/data/web_harvester.py: - harvest_wikipedia_text(lang, max_articles) — MediaWiki API, no deps - harvest_hf_audio(lang, token) — generator yielding (wav_bytes, text, path) - _numpy_to_wav_bytes() — stdlib-only WAV encoder Auto-trigger also fires after each Tab 2 correction save. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>eddcaea
-
Switch training dataset from google/waxal to google/fleurs
jefffffff9 +1: Claude Sonnet 4.6google/waxal was removed from HuggingFace Hub. google/fleurs is the maintained replacement with identical column schema (audio, transcription). Subset mapping: bam → bam_ML, ful → ff_SN. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>d2183cd
-
Initial commit: Sahel-Agri Voice AI
jefffffff976db545