Giving Our Mother Tongue a Digital Future: The MTLTag & VITS Pipeline

Taiwanese is defined by the Tone Sandhi Circle. Unlike non-tonal languages such as English and Japanese, where intonation is the primary carrier of affect, Taiwanese requires ”Tonal-Safe Emotional Mapping.”

MTL is a phonetically consistent, multi-syllable orthography based on POJ. MTL is sandhi-aware, meaning the written form of a multi-syllable word reflects the tone of the front syllable after Tone Sandhi has applied. This overcame the homonym ambiguity in character-based writing system.

Historically, this system functioned as a structural checksum; our research suggests that native speakers utilize a form of ”Perfect Pitch” to decode these melodic headers. When universal AI models fail to respect this melodic continuity, intelligibility drops significantly.

By treating Taiwanese as a formal signal-processing system rather than a collection of statistical probabilities, we enable a ”Sovereign AI” that outperforms universal models in both accuracy and cultural resonance.

MTL-TTS model utilized new modeling technologies:

• MTLTag — The AI’s Linguistic Map. We utilize XLM-RoBERTa for predicting the words should keep the lexical tone or not.

  Computers don't inherently understand the musicality of Taiwanese. MTLTag is our custom labeling system that acts as a "script supervisor" for the AI:

  Precision Tone Sandhi: Taiwanese is famous for words changing tones based on context. MTLTag pre-calculates these shifts so the AI doesn't have to guess. Prosody Guidance: It marks where a sentence should breathe, pause, or rise in pitch, moving away from robotic, word-by-word synthesis.

• MTL-Converter: A critical role in MTL-TTS's pipeline process for converting the input sentence to surface tone based tokens upon the predicted tags in MTLTag.

• VITS Acoustic Synthesis — The "Soul" of the Voice.
  Rhythmic Intelligence: VITS understands the "flow" of a sentence. It learns the natural speed and weight of human speech.
  High Sensitivity to Tones 1-8: Our current model is so precise that if we make a single labeling mistake in the database, the AI will "call us out" by sounding off. This proves the model has truly mastered the 8-tone system of Taiwanese.

  Example:
  Is Sony's mobile-phone easy to use?
  Note: The sentence includes a brand name 'Sony' which was converted to MTL tokens with two syllables (i.e.flat tone and high tone - 'y'). We are tagging them as <en>Sony</en> for the next phase of training.

  Data-Driven Naturalness: By indexing over 10,000 sentences, we are training the AI to handle the natural pauses and transitions found in real conversation.

  Why This Project Matters for the Next Generation

  1. From "Canned" to "Conversational": We aren't just making a translator; we are recreating the soul of the language.
  2. Tech-Forward Preservation: We are using RoBERTa and XLM (Cross-lingual Language Models) to help the AI "understand" Taiwanese at a semantic level, not just a phonetic one.
  3. Building a Bridge: This project connects the deep linguistic knowledge of the 1st generation with the engineering skills of the 2nd generation.

  Get Involved: We Need Your Tech Skills!

  We have the data, we have the pipeline, and we have a model that is already "impressing the neighbors." But there is more to build.

  Are you a Python developer? Help us refine our Class-based converters for KIP/POJ. Are you into ML/NLP? Help us optimize our RoBERTa/XLM tagging and loss functions (`loss_mel`, `loss_kl`). Are you a Heritage Speaker? Help us review and "slice" audio to ensure the highest quality training data. Or better yet, help to design a model that can review synthesized audio file for correctness.

  "I can't do this alone, but together we can fix it." Let’s ensure the language of our parents and grandparents has a seat at the table in the future of AI.

Synthesized Audio Samples

Note: This model was trained from a pool of two females and one male with 9000+ sentences for 36 hours with 160,000 steps of iteration(reading and listening 4 sentences at a time). For VITs, it is said that 350,000 steps are required to achieve a best model. We are sorting out and balancing the next training dataset. by changing the speed and pitch of one female and one male who are the minority of this dataset. By balancing multiple speakers training, we have better control. As a result, we are able to choose which person's voice for audio synthesis.