🐢 Tortoise#

Tortoise is a very expressive TTS system with impressive voice cloning capabilities. It is based on an GPT like autogressive acoustic model that converts input text to discritized acoustic tokens, a diffusion model that converts these tokens to melspectrogram frames and a Univnet vocoder to convert the spectrograms to the final audio signal. The important downside is that Tortoise is very slow compared to the parallel TTS models like VITS.

Big thanks to 👑@manmay-nakhashi who helped us implement Tortoise in 🐸TTS.

Example use:

from TTS.tts.configs.tortoise_config import TortoiseConfig
from TTS.tts.models.tortoise import Tortoise

config = TortoiseConfig()
model = Tortoise.init_from_config(config)
model.load_checkpoint(config, checkpoint_dir="paths/to/models_dir/", eval=True)

# with random speaker
output_dict = model.synthesize(text, config, speaker_id="random", extra_voice_dirs=None, **kwargs)

# cloning a speaker
output_dict = model.synthesize(text, config, speaker_id="speaker_n", extra_voice_dirs="path/to/speaker_n/", **kwargs)

Using 🐸TTS API:

from TTS.api import TTS
tts = TTS("tts_models/en/multi-dataset/tortoise-v2")

# cloning `lj` voice from `TTS/tts/utils/assets/tortoise/voices/lj`
# with custom inference settings overriding defaults.
tts.tts_to_file(text="Hello, my name is Manmay , how are you?",
                file_path="output.wav",
                voice_dir="path/to/tortoise/voices/dir/",
                speaker="lj",
                num_autoregressive_samples=1,
                diffusion_iterations=10)

# Using presets with the same voice
tts.tts_to_file(text="Hello, my name is Manmay , how are you?",
                file_path="output.wav",
                voice_dir="path/to/tortoise/voices/dir/",
                speaker="lj",
                preset="ultra_fast")

# Random voice generation
tts.tts_to_file(text="Hello, my name is Manmay , how are you?",
                file_path="output.wav")

Using 🐸TTS Command line:

# cloning the `lj` voice
tts --model_name  tts_models/en/multi-dataset/tortoise-v2 \
--text "This is an example." \
--out_path "output.wav" \
--voice_dir path/to/tortoise/voices/dir/ \
--speaker_idx "lj" \
--progress_bar True

# Random voice generation
tts --model_name  tts_models/en/multi-dataset/tortoise-v2 \
--text "This is an example." \
--out_path "output.wav" \
--progress_bar True

Important resources & papers#

Original Repo: https://github.com/neonbjb/tortoise-tts
Faster implementation: https://github.com/152334H/tortoise-tts-fast
Univnet: https://arxiv.org/abs/2106.07889
Latent Diffusion:https://arxiv.org/abs/2112.10752
DALL-E: https://arxiv.org/abs/2102.12092

TortoiseConfig#

class TTS.tts.configs.tortoise_config.TortoiseConfig(output_path='output', logger_uri=None, run_name='run', project_name=None, run_description='🐸Coqui trainer run.', print_step=25, plot_step=100, model_param_stats=False, wandb_entity=None, dashboard_logger='tensorboard', save_on_interrupt=True, log_model_step=None, save_step=10000, save_n_checkpoints=5, save_checkpoints=True, save_all_best=False, save_best_after=10000, target_loss=None, print_eval=False, test_delay_epochs=0, run_eval=True, run_eval_steps=None, distributed_backend='nccl', distributed_url='tcp://localhost:54321', mixed_precision=False, precision='fp16', epochs=1000, batch_size=32, eval_batch_size=16, grad_clip=0.0, scheduler_after_epoch=True, lr=0.001, optimizer='radam', optimizer_params=None, lr_scheduler=None, lr_scheduler_params=<factory>, use_grad_scaler=False, allow_tf32=False, cudnn_enable=True, cudnn_deterministic=False, cudnn_benchmark=False, training_seed=54321, model='tortoise', num_loader_workers=0, num_eval_loader_workers=0, use_noise_augment=False, audio=<factory>, use_phonemes=False, phonemizer=None, phoneme_language=None, compute_input_seq_cache=False, text_cleaner=None, enable_eos_bos_chars=False, test_sentences_file='', phoneme_cache_path=None, characters=None, add_blank=False, batch_group_size=0, loss_masking=None, min_audio_len=1, max_audio_len=inf, min_text_len=1, max_text_len=inf, compute_f0=False, compute_energy=False, compute_linear_spec=False, precompute_num_workers=0, start_by_longest=False, shuffle=False, drop_last=False, datasets=<factory>, test_sentences=<factory>, eval_split_max_size=None, eval_split_size=0.01, use_speaker_weighted_sampler=False, speaker_weighted_sampler_alpha=1.0, use_language_weighted_sampler=False, language_weighted_sampler_alpha=1.0, use_length_weighted_sampler=False, length_weighted_sampler_alpha=1.0, model_args=<factory>, model_dir=None, temperature=0.2, length_penalty=1.0, repetition_penalty=2.0, top_p=0.8, cond_free_k=2.0, diffusion_temperature=1.0, num_autoregressive_samples=16, diffusion_iterations=30, sampler='ddim')[source]#

Defines parameters for Tortoise TTS model.

Parameters:

model (str) – Model name. Do not change unless you know what you are doing.
model_args (TortoiseArgs) – Model architecture arguments. Defaults to TortoiseArgs().
audio (TortoiseAudioConfig) – Audio processing configuration. Defaults to TortoiseAudioConfig().
model_dir (str) – Path to the folder that has all the Tortoise models. Defaults to None.
temperature (float) – Temperature for the autoregressive model inference. Larger values makes predictions more creative sacrificing stability. Defaults to 0.2.
length_penalty (float) – Exponential penalty to the length that is used with beam-based generation. It is applied as an exponent to the sequence length, which in turn is used to divide the score of the sequence. Since the score is the log likelihood of the sequence (i.e. negative), length_penalty > 0.0 promotes longer sequences, while length_penalty < 0.0 encourages shorter sequences.
reperation_penalty (float) – The parameter for repetition penalty. 1.0 means no penalty. Defaults to 2.0.
top_p (float) – If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation. Defaults to 0.8.
cond_free_k (float) – Knob that determines how to balance the conditioning free signal with the conditioning-present signal. [0,inf]. As cond_free_k increases, the output becomes dominated by the conditioning-free signal. Formula is: output=cond_present_output*(cond_free_k+1)-cond_absenct_output*cond_free_k. Defaults to 2.0.
diffusion_temperature (float) – Controls the variance of the noise fed into the diffusion model. [0,1]. Values at 0 are the “mean” prediction of the diffusion network and will sound bland and smeared. Defaults to 1.0.
num_autoregressive_samples (int) – Number of samples taken from the autoregressive model, all of which are filtered using CLVP. As Tortoise is a probabilistic model, more samples means a higher probability of creating something “great”. Defaults to 16.
diffusion_iterations (int) – Number of diffusion steps to perform. [0,4000]. More steps means the network has more chances to iteratively refine the output, which should theoretically mean a higher quality output. Generally a value above 250 is not noticeably better, however. Defaults to 30.
sampler (str) – Diffusion sampler to be used. ddim or dpm++2m. Defaults to ddim.

Note

Check TTS.tts.configs.shared_configs.BaseTTSConfig for the inherited parameters.

Example

>>> from TTS.tts.configs.tortoise_config import TortoiseConfig
>>> config = TortoiseConfig()

TortoiseArgs#

class TTS.tts.models.tortoise.TortoiseArgs(autoregressive_batch_size=1, enable_redaction=False, high_vram=False, kv_cache=True, ar_checkpoint=None, clvp_checkpoint=None, diff_checkpoint=None, num_chars=255, vocoder=VocConf.Univnet, ar_max_mel_tokens=604, ar_max_text_tokens=402, ar_max_conditioning_inputs=2, ar_layers=30, ar_model_dim=1024, ar_heads=16, ar_number_text_tokens=255, ar_start_text_token=255, ar_checkpointing=False, ar_train_solo_embeddings=False, diff_model_channels=1024, diff_num_layers=10, diff_in_channels=100, diff_out_channels=200, diff_in_latent_channels=1024, diff_in_tokens=8193, diff_dropout=0, diff_use_fp16=False, diff_num_heads=16, diff_layer_drop=0, diff_unconditioned_percentage=0, clvp_dim_text=768, clvp_dim_speech=768, clvp_dim_latent=768, clvp_num_text_tokens=256, clvp_text_enc_depth=20, clvp_text_seq_len=350, clvp_text_heads=12, clvp_num_speech_tokens=8192, clvp_speech_enc_depth=20, clvp_speech_heads=12, clvp_speech_seq_len=430, clvp_use_xformers=True, duration_const=102400)[source]#

A dataclass to represent Tortoise model arguments that define the model structure.

Parameters:

autoregressive_batch_size (int) – The size of the auto-regressive batch.
enable_redaction (bool, optional) – Whether to enable redaction. Defaults to True.
high_vram (bool, optional) – Whether to use high VRAM. Defaults to False.
kv_cache (bool, optional) – Whether to use the kv_cache. Defaults to True.
ar_checkpoint (str, optional) – The checkpoint for the autoregressive model. Defaults to None.
clvp_checkpoint (str, optional) – The checkpoint for the ConditionalLatentVariablePerseq model. Defaults to None.
diff_checkpoint (str, optional) – The checkpoint for the DiffTTS model. Defaults to None.
num_chars (int, optional) – The maximum number of characters to generate. Defaults to 255.
vocoder (VocType, optional) – The vocoder to use for synthesis. Defaults to VocConf.Univnet.
model (For ConditionalLatentVariablePerseq) –
ar_max_mel_tokens (int, optional) – The maximum mel tokens for the autoregressive model. Defaults to 604.
ar_max_text_tokens (int, optional) – The maximum text tokens for the autoregressive model. Defaults to 402.
ar_max_conditioning_inputs (int, optional) – The maximum conditioning inputs for the autoregressive model. Defaults to 2.
ar_layers (int, optional) – The number of layers for the autoregressive model. Defaults to 30.
ar_model_dim (int, optional) – The model dimension for the autoregressive model. Defaults to 1024.
ar_heads (int, optional) – The number of heads for the autoregressive model. Defaults to 16.
ar_number_text_tokens (int, optional) – The number of text tokens for the autoregressive model. Defaults to 255.
ar_start_text_token (int, optional) – The start text token for the autoregressive model. Defaults to 255.
ar_checkpointing (bool, optional) – Whether to use checkpointing for the autoregressive model. Defaults to False.
ar_train_solo_embeddings (bool, optional) – Whether to train embeddings for the autoregressive model. Defaults to False.
model –
diff_model_channels (int, optional) – The number of channels for the DiffTTS model. Defaults to 1024.
diff_num_layers (int, optional) – The number of layers for the DiffTTS model. Defaults to 10.
diff_in_channels (int, optional) – The input channels for the DiffTTS model. Defaults to 100.
diff_out_channels (int, optional) – The output channels for the DiffTTS model. Defaults to 200.
diff_in_latent_channels (int, optional) – The input latent channels for the DiffTTS model. Defaults to 1024.
diff_in_tokens (int, optional) – The input tokens for the DiffTTS model. Defaults to 8193.
diff_dropout (int, optional) – The dropout percentage for the DiffTTS model. Defaults to 0.
diff_use_fp16 (bool, optional) – Whether to use fp16 for the DiffTTS model. Defaults to False.
diff_num_heads (int, optional) – The number of heads for the DiffTTS model. Defaults to 16.
diff_layer_drop (int, optional) – The layer dropout percentage for the DiffTTS model. Defaults to 0.
diff_unconditioned_percentage (int, optional) – The percentage of unconditioned inputs for the DiffTTS model. Defaults to 0.
model –
clvp_dim_text (int) – The dimension of the text input for the CLVP module. Defaults to 768.
clvp_dim_speech (int) – The dimension of the speech input for the CLVP module. Defaults to 768.
clvp_dim_latent (int) – The dimension of the latent representation for the CLVP module. Defaults to 768.
clvp_num_text_tokens (int) – The number of text tokens used by the CLVP module. Defaults to 256.
clvp_text_enc_depth (int) – The depth of the text encoder in the CLVP module. Defaults to 20.
clvp_text_seq_len (int) – The maximum sequence length of the text input for the CLVP module. Defaults to 350.
clvp_text_heads (int) – The number of attention heads used by the text encoder in the CLVP module. Defaults to 12.
clvp_num_speech_tokens (int) – The number of speech tokens used by the CLVP module. Defaults to 8192.
clvp_speech_enc_depth (int) – The depth of the speech encoder in the CLVP module. Defaults to 20.
clvp_speech_heads (int) – The number of attention heads used by the speech encoder in the CLVP module. Defaults to 12.
clvp_speech_seq_len (int) – The maximum sequence length of the speech input for the CLVP module. Defaults to 430.
clvp_use_xformers (bool) – A flag indicating whether the model uses transformers in the CLVP module. Defaults to True.
duration_const (int) – A constant value used in the model. Defaults to 102400.

Tortoise Model#

class TTS.tts.models.tortoise.Tortoise(config)[source]#

Tortoise model class.

Currently only supports inference.

Examples

>>> from TTS.tts.configs.tortoise_config import TortoiseConfig
>>> from TTS.tts.models.tortoise import Tortoise
>>> config = TortoiseConfig()
>>> model = Tortoise.inif_from_config(config)
>>> model.load_checkpoint(config, checkpoint_dir="paths/to/models_dir/", eval=True)

get_conditioning_latents(voice_samples, return_mels=False, latent_averaging_mode=0, original_tortoise=False)[source]#: Transforms one or more voice_samples into a tuple (autoregressive_conditioning_latent, diffusion_conditioning_latent). These are expressive learned latents that encode aspects of the provided clips like voice, intonation, and acoustic properties. :param voice_samples: List of arbitrary reference clips, which should be pairs of torch tensors containing arbitrary kHz waveform data. :param latent_averaging_mode: 0/1/2 for following modes:

0 - latents will be generated as in original tortoise, using ~4.27s from each voice sample, averaging latent across all samples 1 - latents will be generated using (almost) entire voice samples, averaged across all the ~4.27s chunks 2 - latents will be generated using (almost) entire voice samples, averaged per voice sample

inference(text, voice_samples=None, conditioning_latents=None, k=1, verbose=True, use_deterministic_seed=None, return_deterministic_state=False, latent_averaging_mode=0, num_autoregressive_samples=16, temperature=0.8, length_penalty=1, repetition_penalty=2.0, top_p=0.8, max_mel_tokens=500, diffusion_iterations=100, cond_free=True, cond_free_k=2, diffusion_temperature=1.0, sampler='ddim', half=True, original_tortoise=False, **hf_generate_kwargs)[source]#

This function produces an audio clip of the given text being spoken with the given reference voice.

Parameters:

text – (str) Text to be spoken.
voice_samples – (List[Tuple[torch.Tensor]]) List of an arbitrary number of reference clips, which should be tuple-pairs of torch tensors containing arbitrary kHz waveform data.
conditioning_latents – (Tuple[autoregressive_conditioning_latent, diffusion_conditioning_latent]) A tuple of (autoregressive_conditioning_latent, diffusion_conditioning_latent), which can be provided in lieu of voice_samples. This is ignored unless voice_samples=None. Conditioning latents can be retrieved via get_conditioning_latents().
k – (int) The number of returned clips. The most likely (as determined by Tortoises’ CLVP model) clips are returned. latent_averaging_mode: (int) 0/1/2 for following modes: 0 - latents will be generated as in original tortoise, using ~4.27s from each voice sample, averaging latent across all samples 1 - latents will be generated using (almost) entire voice samples, averaged across all the ~4.27s chunks 2 - latents will be generated using (almost) entire voice samples, averaged per voice sample
verbose – (bool) Whether or not to print log messages indicating the progress of creating a clip. Default=true.
num_autoregressive_samples – (int) Number of samples taken from the autoregressive model, all of which are filtered using CLVP. As Tortoise is a probabilistic model, more samples means a higher probability of creating something “great”.
temperature – (float) The softmax temperature of the autoregressive model.
length_penalty – (float) A length penalty applied to the autoregressive decoder. Higher settings causes the model to produce more terse outputs.
repetition_penalty – (float) A penalty that prevents the autoregressive decoder from repeating itself during decoding. Can be used to reduce the incidence of long silences or “uhhhhhhs”, etc.
top_p – (float) P value used in nucleus sampling. (0,1]. Lower values mean the decoder produces more “likely” (aka boring) outputs.
max_mel_tokens – (int) Restricts the output length. (0,600] integer. Each unit is 1/20 of a second.
typical_sampling – (bool) Turns typical sampling on or off. This sampling mode is discussed in this paper: https://arxiv.org/abs/2202.00666 I was interested in the premise, but the results were not as good as I was hoping. This is off by default, but could use some tuning.
typical_mass – (float) The typical_mass parameter from the typical_sampling algorithm.
diffusion_iterations – (int) Number of diffusion steps to perform. [0,4000]. More steps means the network has more chances to iteratively refine the output, which should theoretically mean a higher quality output. Generally a value above 250 is not noticeably better, however.
cond_free – (bool) Whether or not to perform conditioning-free diffusion. Conditioning-free diffusion performs two forward passes for each diffusion step: one with the outputs of the autoregressive model and one with no conditioning priors. The output of the two is blended according to the cond_free_k value below. Conditioning-free diffusion is the real deal, and dramatically improves realism.
cond_free_k – (float) Knob that determines how to balance the conditioning free signal with the conditioning-present signal. [0,inf]. As cond_free_k increases, the output becomes dominated by the conditioning-free signal.
diffusion_temperature – (float) Controls the variance of the noise fed into the diffusion model. [0,1]. Values at 0 are the “mean” prediction of the diffusion network and will sound bland and smeared.
hf_generate_kwargs – (**kwargs) The huggingface Transformers generate API is used for the autoregressive transformer. Extra keyword args fed to this function get forwarded directly to that API. Documentation here: https://huggingface.co/docs/transformers/internal/generation_utils

Returns:

Generated audio clip(s) as a torch tensor. Shape 1,S if k=1 else, (k,1,S) where S is the sample length. Sample rate is 24kHz.

inference_with_config(text, config, **kwargs)[source]#: inference with config #TODO describe in detail

load_checkpoint(config, checkpoint_dir, ar_checkpoint_path=None, diff_checkpoint_path=None, clvp_checkpoint_path=None, vocoder_checkpoint_path=None, eval=False, strict=True, **kwargs)[source]#

Load a model checkpoints from a directory. This model is with multiple checkpoint files and it expects to have all the files to be under the given checkpoint_dir with the rigth names. If eval is True, set the model to eval mode.

Parameters:

config (TortoiseConfig) – The model config.
checkpoint_dir (str) – The directory where the checkpoints are stored.
ar_checkpoint_path (str, optional) – The path to the autoregressive checkpoint. Defaults to None.
diff_checkpoint_path (str, optional) – The path to the diffusion checkpoint. Defaults to None.
clvp_checkpoint_path (str, optional) – The path to the CLVP checkpoint. Defaults to None.
vocoder_checkpoint_path (str, optional) – The path to the vocoder checkpoint. Defaults to None.
eval (bool, optional) – Whether to set the model to eval mode. Defaults to False.
strict (bool, optional) – Whether to load the model strictly. Defaults to True.

synthesize(text, config, speaker_id='random', voice_dirs=None, **kwargs)[source]#

Synthesize speech with the given input text.

Parameters:

text (str) – Input text.
config (TortoiseConfig) – Config with inference parameters.
speaker_id (str) – One of the available speaker names. If random, it generates a random speaker.
voice_dirs (List[str]) – List of paths that host reference audio files for speakers. Defaults to None.
**kwargs – Inference settings. See inference().

Returns:

A dictionary of the output values with wav as output waveform, deterministic_seed as seed used at inference, text_input as text token IDs after tokenizer, voice_samples as samples used for cloning, conditioning_latents as latents used at inference.