Voice Core for AI Agents

Explore the Virtuals Protocol Voice Core for AI agent speech, including speech-to-text, text-to-speech, VITS voice synthesis, and audio data preprocessing.

The Voice Core gives each VIRTUAL agent a distinct, personality-aligned voice. AI voice model training creates realistic, consistent speech for each agent and role.

AI agent voice modules

Speech-to-text (STT): The STT module trains on diverse voice data. It accurately transcribes accents, dialects, and speech patterns across user scenarios.

Text-to-speech (TTS): The TTS module uses Variational Inference for Text-to-Speech (VITS) training. VITS produces high-quality, natural-sounding speech and supports voice synthesis customized to each AI agent’s personality.

Audio data preprocessing occurs before voice model training.

Audio data preprocessing for voice models

1. Audio format consistency: WAV files at 22050 Hz in mono create consistent training inputs. Consistent input data helps machine learning voice models perform reliably.

2. Sampling-rate normalization: A 22050 Hz sampling rate captures human speech frequencies while keeping file sizes manageable. It captures frequencies up to 11025 Hz under the Nyquist theorem.

3. Mono audio channels: Converting stereo or multi-channel audio to mono gives the voice model one training channel and simplifies learning.

Sample Code

import os
from pydub import AudioSegment

upload_dir = 'upload_dir'
output_dir = 'out'

# Ensure the output directory exists
os.makedirs(output_dir, exist_ok=True)

extensions = ['wav', 'mp3', 'ogg']

# Process all files in the upload directory
for filename in os.listdir(upload_dir):
    if any(filename.lower().endswith(ext) for ext in extensions):
        # Construct file paths
        file_path = os.path.join(upload_dir, filename)
        output_path = os.path.join(output_dir, os.path.splitext(filename)[0] + '.wav')

        # Load the audio file
        audio = AudioSegment.from_file(file_path)

        # Convert to WAV, 22050 Hz, mono
        audio = audio.set_frame_rate(22050).set_channels(1)

        # Export the processed audio
        audio.export(output_path, format='wav')

Learn more about contributing to Voice Core.