rh>ddlZddlmZddlmZmZmZddlZddl m Z m Z e rddl Z e rddl ZddlmZmZddlmZddlmZmZmZmZdd lmZmZGd d ed ZGdded ZGddeZdgZy)N)Path)AnyOptionalUnion)is_soundfile_availableis_torch_available) AudioInputmake_list_of_audio) BatchFeature) AudioKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInputc(eZdZUeeeefed<y)CsmAudioKwargsencoded_length_kwargsN)__name__ __module__ __qualname__rdictstrr__annotations__y/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/csm/processing_csm.pyrr%s#DcN33rrF)totalcLeZdZUeed<ddddgdgdgddd d d d d idZy)CsmProcessorKwargs audio_kwargsTleftF)padding padding_sideadd_special_tokens)rrr( rr( rr(r)r(r(r(r-r(r(r(r(r(r(r)r()r(r(r(r(r(r(r(r(r(r(r(r(r(r(r() kernel_sizesstrides dilationsuse_causal_convi])r sampling_ratereturn_tensorspt) text_kwargsr" common_kwargsN)rrrrr _defaultsrrrr!r!)sG  ""' !QHJ#' & # +D1Irr!c eZdZdZddgZdZdZ dfd ZeddZ de d e e e ee e e ffd eefd Z dd ee eeeeeefdee d eedeed eef dZxZS) CsmProcessora  Constructs a Csm processor which wraps [`EncodecFeatureExtractor`] and [`PretrainedTokenizerFast`] into a single processor that inherits both the audio feature extraction and tokenizer functionalities. See the [`~CsmProcessor.__call__`] for more information. The preferred way of passing kwargs is as a dictionary per modality, see usage example below. ```python from transformers import CsmProcessor from datasets import load_dataset ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") audio = ds[0]["audio"]["array"] processor = CsmProcessor.from_pretrained("sesame/csm-1b") processor( text=["<|begin_of_text|>[0]What are you working on?<|end_of_text|><|AUDIO|><|audio_eos|><|begin_of_text|>[1]I'm figuring out my budget.<|end_of_text|>"], audio=audio, text_kwargs = {"padding": False}, audio_kwargs = {"sampling_rate": 16000}, common_kwargs = {"return_tensors": "pt"}, ) # this should error out because EncodecFeatureExtractor expects a 24kHz audio :) ``` Args: feature_extractor ([`EncodecFeatureExtractor`]): The feature extractor is a required input. tokenizer ([`PreTrainedTokenizer`, `PreTrainedTokenizerFast`]): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. feature_extractor tokenizerEncodecFeatureExtractorPreTrainedTokenizerFastct|ds(d|_|j|j|_n"|j|_|j|_t|ds(d|_|j|j|_n"|j|_|j |_t ||||y)N audio_tokenz <|AUDIO|>audio_eos_tokenz <|audio_eos|>) chat_template)hasattrrBconvert_tokens_to_idsaudio_token_idrCaudio_eos_token_idsuper__init__)selfr=r>rD __class__s rrJzCsmProcessor.__init__fs y-0*D "+"A"A$BRBR"SD (44D "+":":D y"34#2D &/&E&EdFZFZ&[D ##,#<#|}|||||St|||D]\}}}|dz |zdz} ||z } | dz} | | z } || z | z|z dz} tj| dz } | |z|z| z }||z }|r| } |} n| } | |z} || z| z}|||dz zz dz |zdz}|S)a| Compute the length of the encoded audio sequence. Args: audio_length (int): The length of the audio sequence. kernel_sizes (list[int]): The kernel sizes for the convolutional layers. strides (list[int]): The strides for the convolutional layers. use_causal_conv (bool): Whether to use causal convolutions. r(r0)zipmathceil) audio_lengthr1r2r3r4 cur_length kernel_sizestridedilationeffective_kernel_size padding_total padding_right padding_leftn_frames ideal_length extra_paddings r_get_encoded_lengthz CsmProcessor._get_encoded_length|s"  7?i6G?Kb -0w -R W )K%01_$@1$D !'&0M)Q.M(=8L"%::]JfTWXXHyy*Q.H#f,{:]JL(:5M, - + - = #l2]BJ$x;?'CCaGFRUVVJ' W*raudio saving_pathkwargsc Fts tdt|}t|tt fr|g}n3t|t tfrtd|Ds tdt|t|k7r td|jtfi|}|d}|d}t||D]b\}}t|tjr,|j!j#j%}t'j(|||dy)Nz/Please install `soundfile` to save audio files.c3HK|]}t|ttfywN) isinstancerr).0ps r z*CsmProcessor.save_audio..s@q`aAPSUY{A[@qs "zAInvalid input path. Please provide a string, or a list of stringsz5The number of audio and saving paths must be the samer"r5)r ImportErrorr rdrrlisttupleall ValueErrorlen _merge_kwargsr!rNtorchTensorcpufloatnumpysfwrite) rKr^r_r` output_kwargsr"r5 audio_valuerfs r save_audiozCsmProcessor.save_audios&'OP P#5) kC; /&-K[4-8S@qep@q=q`a a u:[) )TU U***   %^4 $_5 !%5 4NK+u||4)oo/557==? HHQ ] 3 4rtext output_labelsdepth_decoder_labels_ratioc h |jtfd|jji|}|d}|d}|d} | j dd} | dk7r"t |j jdt|tr|g}n3t|ttfrtd |Ds t d |D cgc]} | j|j} } d } |t|}t!|} t#| d kDr-| t#| k7r| t d t d | d| d||j di}|Dcgc]"}|j$|j&dfi|$}}|j)}g}|D]}g}|j|vr]|j d }|j|z}|j+||j-|jdd}|j|vr]d|vr'|j-d|j d d}d|vr'|j+||}|j|fi|}i}|j/|||j ddgg}}d }| D]} | d k(rJ|j+t1j2d |j+t5j6dgS|j+t1j8|||| zDcgc]<}t|t4j:r|j=j?n|>c}d|j+t5j6|||| zDcgc]}|j&dc}jAd|| z }!|jB|fi|}|j dd|j/|tEd|D}|Dcgc]@}t4jFjHjK|d ||j&dz fdB}}t5jL|d |d<|r|d|jNk(jQ} | j&d }!|dkr-t5jR|!dtU|!d|z z}"| |"}#n| }#t5jV|d|jNk(|d|jXk(z|dd}$d|$|#ddd f|#dddff<|$|d<t[|| Scc} wcc}wcc}wcc}wcc}w)!a Main method to prepare text(s) and audio to be fed as input to the model. This method forwards the `text` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] to encode the text. To prepare the audio, this method forwards the `audio` arguments to EncodecFeatureExtractor's [`~EncodecFeatureExtractor.__call__`]. Please refer to the docstring of the above two methods for more information. Args: audio (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`): The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch tensor. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). output_labels (bool, *optional*, default=False): Whether to return labels for training. Indices will be in `[config.audio_token_id, -100, -101]`. - `config.audio_token_id` indicates an audio frame (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) depth_decoder_labels_ratio (float, *optional*, default=1.0): The ratio of audio frames to keep for the depth decoder labels. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **input_values** -- List of audio values to be fed to a model. Returned when `audio` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **labels** -- List of labels for the audio frames. Returned when `output_labels=True`. tokenizer_init_kwargsr8r"r9r6Nr7z% only supports `return_tensors='pt'`.c3<K|]}t|tywrc)rdr)rets rrgz(CsmProcessor.__call__..s9[QR*Q:L9[szAInvalid input text. Please provide a string, or a list of stringsrz@No audio were provided, but there are audio tokens in the promptz)The number of audio tokens in each text (z7) should be the same as the number of provided audios (z).rz r(return_attention_mask)axis)dim padding_maskc3:K|]}|jdyw)rN)shape)recut_idxss rrgz(CsmProcessor.__call__..OsR(..,Rs)valueinput_values_cutoffs input_ids?iilabels)data tensor_type).rnr!r> init_kwargspoprlrLrrdrrirjrkcountrBr rmsumr]rcopyappendreplaceupdatenpzerosrotensor concatenaterprqrscumsumr=maxnn functionalpadstackrGnonzerorandpermintwhererHr )%rKryr^rzr{r`rvr8r"r9r6rn_audio_in_textn_audior audio_arraynum_audio_tokens_listnum_audio_tokens_list_copy expanded_textsample replace_strnum_audio_tokensexpanded_audio_tokenencodingrconcatenated_audioroffsetel audio_inputsmax_lenraudio_frame_idxsn_audio_frames rand_idxsskip_frames_idxsrs% r__call__zCsmProcessor.__call__s^+**  "&.."<"<  $M2 $^4 %o6 &**+;TB T ! 7 788]^_ _ dC 6DTD%=1c9[VZ9[6[`a a>BC1774#3#34CC  &u-E%jG  ! #33G(G} !cdd ??PQ229">  $0$4$45Lb$Q !lq%]h((():):2)>XBWX% !%*?)C)C)E &M - &&&0'A'E'Ea'H$+/+;+;>N+N(&&';<#^^D,<,>$6+6 H     4d ;792 4 F* &a<&--bhhqk:(// bT0BC&--+0'9I*J$&5?r5<<4P 0VX X"$ )// U6FU\L\=]%^rbhhrl%^_ffkmfng%F# &&24112DU UL   ^T 2 KK %R=QRRG!5$##''1gr@R6R2S[]'^$ $,1;;7KQR+SD' (  $[ 1T5H5H HQQS -33A6N)S0!NN>:;sSSTWqSqAr=st #3I#> #3 [[k"d&9&99d;>OSWSjSj>jk[!F FJF#AqD)+;AqD+AA B#DN>BBGD$%L&_$s "T'T :AT%,T*1AT/rc)NNNN)NFr)rrr__doc__ attributesfeature_extractor_classtokenizer_classrJ staticmethodr]r rrrrirr!rxrrrboolrrr __classcell__)rLs@rr<r<>s!F&{3J7/O  T,$$L 4 43d5d+;&<<= 4+, 4J'+(-69 dCuY(94 ?DQbLccdedC #dC ~ dC %-UO dC +, dCrr<) rOpathlibrtypingrrrrsrutilsrr ro soundfilert audio_utilsr r feature_extraction_utilsr processing_utilsr rrrtokenization_utils_baserrrr!r<__all__rrrrst  ''?94UUC4[4)*kC>kC\  r