rhj) d dl mZmZmZ ddlmZ ddlmZ ddlm Z e rd dl
Z
ddl m Z dd l
mZ d
Z G d d e Zy)
)AnyUnionoverload )GenerationConfig)is_torch_available )PipelineN)%MODEL_FOR_TEXT_TO_SPECTROGRAM_MAPPING)SpeechT5HifiGanzmicrosoft/speecht5_hifiganc
e Zd ZdZdZdZdZdZdZdZ e d Z
dddd fd
Z d Z d
Z
ed ed ed
eeef fd Zed ee d ed
eeeef fd Zd eeee f d
eeeef eeeef f f fd Z ddZd Z xZS )TextToAudioPipelinea
Text-to-audio generation pipeline using any `AutoModelForTextToWaveform` or `AutoModelForTextToSpectrogram`. This
pipeline generates an audio file from an input text and optional other conditional inputs.
Unless the model you're using explicitly sets these generation parameters in its configuration files
(`generation_config.json`), the following default values will be used:
- max_new_tokens: 256
Example:
```python
>>> from transformers import pipeline
>>> pipe = pipeline(model="suno/bark-small")
>>> output = pipe("Hey it's HuggingFace on the phone!")
>>> audio = output["audio"]
>>> sampling_rate = output["sampling_rate"]
```
Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial)
You can specify parameters passed to the model by using [`TextToAudioPipeline.__call__.forward_params`] or
[`TextToAudioPipeline.__call__.generate_kwargs`].
Example:
```python
>>> from transformers import pipeline
>>> music_generator = pipeline(task="text-to-audio", model="facebook/musicgen-small", framework="pt")
>>> # diversify the music generation by adding randomness with a high temperature and set a maximum music length
>>> generate_kwargs = {
... "do_sample": True,
... "temperature": 0.7,
... "max_new_tokens": 35,
... }
>>> outputs = music_generator("Techno music with high melodic riffs", generate_kwargs=generate_kwargs)
```
This pipeline can currently be loaded from [`pipeline`] using the following task identifiers: `"text-to-speech"` or
`"text-to-audio"`.
See the list of available models on [huggingface.co/models](https://huggingface.co/models?filter=text-to-speech).
TF )max_new_tokensN)vocoder
sampling_rate no_processorc | t | |i | || _ | j dk( r t d d | _ | j j t j v rE|>T ! TU U :: #H#O#O#Q Q ? //0BCFFtzzGXGXY
L + << #!% !4!4!B!BD % ZZ&&F ,,001DdKJ%
j 0 0 23&F
7" '0BD I
,)6D&
7 %d.?.?GDNN\oDp!% !A!A!O!OD Eq.? % c R t |t r|g}| j j j dk( r?| j
j j dd ddddd}|j | |}| j r | j n | j } ||fi |dd i}|S )
Nbarkmax_input_semantic_lengthr FT
max_length)r6 add_special_tokensreturn_attention_maskreturn_token_type_idspaddingreturn_tensorspt)
isinstancestrr r$
model_typer semantic_configr& r' r tokenizerr+ )r, textr.
new_kwargs preprocessoroutputs r1
preprocessz TextToAudioPipeline.preprocess s dC 6D :: ' '6 1 # 4 4 D D H HIdfi j&+)-).' J
f % F)-):):t~~ dBfBTB
r2 c | j || j }|d }|d }| j j r`| j || j }d|vr| j |d< |j | | j j di ||}n>t | r t d|j | j di ||d }| j | j | }|S )N)r# forward_paramsgenerate_kwargsr zYou're using the `TextToAudioPipeline` with a forward-only model, but `generate_kwargs` is non empty. For forward-only TTA models, please use `forward_params` instead of `generate_kwargs`. For reference, the `generate_kwargs` used here are: r ) _ensure_tensor_on_devicer# r can_generater r' generatelenr keysr )r, model_inputsr. rH rI rE s r1 _forwardz TextToAudioPipeline._forward s ..vdkk.J 01 !23 :: " " $ " ; ;OTXT_T_ ; `O #/97;7M7M 34
! !/ 2(TZZ((J<J>JF?# KKZK_K_KaJbd
TZZA,A.A!DF << #\\&)F
r2 text_inputsrH returnc y NrJ r, rR rH s r1 __call__z TextToAudioPipeline.__call__ s SVr2 c y rU rJ rV s r1 rW z TextToAudioPipeline.__call__ s _br2 c $ t | |fi |S )a
Generates speech/audio from the inputs. See the [`TextToAudioPipeline`] documentation for more information.
Args:
text_inputs (`str` or `list[str]`):
The text(s) to generate.
forward_params (`dict`, *optional*):
Parameters passed to the model generation/forward method. `forward_params` are always passed to the
underlying model.
generate_kwargs (`dict`, *optional*):
The dictionary of ad-hoc parametrization of `generate_config` to be used for the generation call. For a
complete overview of generate, check the [following
guide](https://huggingface.co/docs/transformers/en/main_classes/text_generation). `generate_kwargs` are
only passed to the underlying model if the latter is a generative model.
Return:
A `dict` or a list of `dict`: The dictionaries have two keys:
- **audio** (`np.ndarray` of shape `(nb_channels, audio_length)`) -- The generated audio waveform.
- **sampling_rate** (`int`) -- The sampling rate of the generated audio waveform.
)r rW )r, rR rH r s r1 rW z TextToAudioPipeline.__call__ s 0 w >~>>r2 c t | dd | j |d< t | dd | j |d< | j |d< |r|ni |r|ni d}|i }i }|||fS )Nassistant_modelassistant_tokenizerrA )rH rI )r) r[ rA r\ )r, preprocess_paramsrH rI paramspostprocess_paramss r1 _sanitize_parametersz(TextToAudioPipeline._sanitize_parameters s 4*D 1 =151E1EO - . 4. 5 A+/>>OK (595M5MO 1 2 1?nB2Ar
$ " &*<< r rf rW listr r` rm
__classcell__)r s @r1 r r s 2j O # O ! #O "2" '+$T $PL0 B VCV3V4S>V V
bDIbbdSVX[S[nI]b b? d3i 0? tCH~tDcN 33 4?8 =.r2 r )typingr r r
generationr utilsr baser
ri models.auto.modeling_autor !models.speecht5.modeling_speecht5r r! r rJ r2 r1 r s; ( ' ) & QC1 k( kr2