Voice Core for AI Agents
Explore the Virtuals Protocol Voice Core for AI agent speech, including speech-to-text, text-to-speech, VITS voice synthesis, and audio data preprocessing.
Last updated
Explore the Virtuals Protocol Voice Core for AI agent speech, including speech-to-text, text-to-speech, VITS voice synthesis, and audio data preprocessing.
The Voice Core gives each VIRTUAL agent a distinct, personality-aligned voice. AI voice model training creates realistic, consistent speech for each agent and role.
Speech-to-text (STT): The STT module trains on diverse voice data. It accurately transcribes accents, dialects, and speech patterns across user scenarios.
Text-to-speech (TTS): The TTS module uses Variational Inference for Text-to-Speech (VITS) training. VITS produces high-quality, natural-sounding speech and supports voice synthesis customized to each AI agent’s personality.
Audio data preprocessing occurs before voice model training.
Audio format consistency: WAV files at 22050 Hz in mono create consistent training inputs. Consistent input data helps machine learning voice models perform reliably.
Sampling-rate normalization: A 22050 Hz sampling rate captures human speech frequencies while keeping file sizes manageable. It captures frequencies up to 11025 Hz under the Nyquist theorem.
Mono audio channels: Converting stereo or multi-channel audio to mono gives the voice model one training channel and simplifies learning.
import os
from pydub import AudioSegment
upload_dir = 'upload_dir'
output_dir = 'out'
# Ensure the output directory exists
os.makedirs(output_dir, exist_ok=True)
extensions = ['wav', 'mp3', 'ogg']
# Process all files in the upload directory
for filename in os.listdir(upload_dir):
if any(filename.lower().endswith(ext) for ext in extensions):
# Construct file paths
file_path = os.path.join(upload_dir, filename)
output_path = os.path.join(output_dir, os.path.splitext(filename)[0] + '.wav')
# Load the audio file
audio = AudioSegment.from_file(file_path)
# Convert to WAV, 22050 Hz, mono
audio = audio.set_frame_rate(22050).set_channels(1)
# Export the processed audio
audio.export(output_path, format='wav')
Last updated