rh֊ddlmZddlmZmZddlZddlmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZd dlmZd dlm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ejZe.Z/eedGddeZ0Gdde%Z1Gdde&Z2Gdde#Z3Gdde Z4Gdd e!Z5ed!eGd"d#eZ6eGd$d%e$e6Z7Gd&d'ejpZ9ed(Gd)d*e"eZ:Gd+d,ejpZ;eGd-d.e$Z<ed/Gd0d1e6e,Z=gd2Z>y)3) dataclass)OptionalUnionN)check_model_inputs)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introceZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee ejdfed<dZ ee ejdfed<dZeejed <dZejed <dZee e ejed <dZee ejdfed <dZee ejdfed <dZeejed<y)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r%rtorch FloatTensor__annotations__r&r'tupler(r)r*r+r,r-r.r/v/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/csm/modular_csm.pyr$r$1s'<)-D(5$$ %, $FE  $AEOXeE%*;*;$<=>E=AM8E%"3"3S"89:A:>Ju00#567>6:!2!23:.2%++2OS!8E%8I8I2J,K#LSKO%0A0A30F*G!HOHLhuU->->-C'DEL15M8E--.5r9r$c eZdZy) CsmRMSNormNr0r1r2r8r9r:r<r<dr9r<c eZdZy)CsmRotaryEmbeddingNr=r8r9r:r@r@hr>r9r@c eZdZy)CsmMLPNr=r8r9r:rBrBlr>r9rBc eZdZy) CsmAttentionNr=r8r9r:rDrDpr>r9rDc eZdZy)CsmDecoderLayerNr=r8r9r:rFrFtr>r9rFz[ The bare Csm Model outputting raw hidden-states without any specific head on top. cXeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ e edZfdZxZS)CsmPreTrainedModelconfigmodelTrFr')r(r)ct||t|tr^|j}t |dz D]@}|j j|jd|jjByy)Nrg)meanstd) super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeweightdatanormal_rIinitializer_range)selfmodulerRi __class__s r:rOz CsmPreTrainedModel._init_weightssr f% f. /"00M=1,- [ ""1%--3DKK*>ARAR*RU[UpUpq')yy1L1LfN`N`gl'm$r9 input_idsbackbone_last_hidden_stateattention_mask position_idsr' inputs_embeds use_cachecache_positionkwargsreturnc |5tjjstj dd}|du|duz r t d|r | t }|i||jnd} ||jdn|jd} | |jn |j} tj| | | z| }|tj|dz d} | |jz}|j||z}|ddk(}| ||dddf<n5tjjs|rtjd|j|}t!|j"||||| }|}|j%d}|j'||}|j(d|j"j*D]}||f||||||d | }|j-|}t/||r| Sd S) aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids from `cache_position` and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.rrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)rI input_embedsrxr|r'ry)rxrypast_key_valuer{r|position_embeddings)last_hidden_stater')r4compiler is_compilinglogger warning_once ValueErrorr get_seq_lengthshaperarangeclamprorqwarningrtr rI unsqueeze rotary_emblayersnum_hidden_layersnormr )rXrvrwrxryr'rzr{r|r}past_seen_tokensinputs_seq_lengthr codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr(r decoder_layers r:forwardzCsmDepthDecoderModel.forwards>&  #ENN,G,G,I   M  L  -t"; <Z[ [ 0*nO  !CRC^==?de :G:S 3 3A 6YbYhYhijYk -:-F]))IL\L\F"\\*:B  r9)NNNNNNNN)r0r1r2r r6rlrrr4 LongTensorrr5Tensorrboolrrrr7r rrerfs@r:rhrhs !!n '+BF1537+/59$(59R ##R %-U->->$?R !. R u//0 R "% R   1 12R D>R !!1!12R +,R  u-- .R R r9rhc&eZdZfdZddZxZS)rQct|||_tjt j |jdz |||_y)Nr)rNrlrRrm Parameterr4emptyrT)rXrsrRror[s r:rlzCsmCodebooksHead.__init__s? *ll5;;t/A/AA/E{T^#_` r9c |2|jd}|jtj|}n|dz }|j|}t |jdDcgc]9}t j j|dd|ddf||j;}}tj|d}|Scc}w)Nrrdim) rrTr4rrSrm functionallinearTstack)rXr(r| seq_lengthcodebook_weightr codebook_idxs r:rzCsmCodebooksHead.forwards  !&,,Q/J"kk%,,z*BCO*Q.M"kk-8O!&o&;&;A&> ?  MM q,/A!BOT`DaDcDc d   Mq9   s#>B<Nr0r1r2rlrrerfs@r:rQrQsa r9rQa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) c,eZdZdZdZdZfdZ ddejde e de ejde ejde ejf fd Z e e ddejd e ejde ejd e ejde ee eejfde ejd e ejd e ede ejd eeejfdeedeeeffdZxZS)CsmDepthDecoderForCausalLMNct|||`t|j|j |j |_t||_ yr) rNrllm_headrQrsrRrocodebooks_headrhrJrus r:rlz#CsmDepthDecoderForCausalLM.__init__sE  L.v/A/A6CWCWY_YjYjk)&1 r9rvr'rxrzr|c t ||||||fi|}|dddk(}|s|jd|jd|S)Nr|rrwry)rNprepare_inputs_for_generationpop) rXrvr'rxrzr|r} model_inputsis_first_generation_stepr[s r:rz8CsmDepthDecoderForCausalLM.prepare_inputs_for_generation!sgw<  ~ Y_ $00@#A!#D#I '   9 : (r9rwrylabelsr{logits_to_keepr}r~c |jd|||||||| d| } | d} t| tr | dk(r tdd}nt| d}n| }|j | dd|ddf| | |nd}|j }d}|B|dddfj }|j d|d|jj|d| }t||| j| j| jS) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )rvrwrxryr'rzr{r|rrN.)r&rro shift_labels)r%r&r'r(r)r8) rJrPintslicer contiguous loss_functionrIror r'r(r))rXrvrwrxryr'rzrr{r|rr}outputsr( slice_indicesr&r%rs r:rz"CsmDepthDecoderForCausalLM.forward7s;2$**  'A)%+')      nc *" %a %~ot < *M$$ !]A- .Q_Qk}0Mqu ""$  !#qr'?557L%4%%dt{{7M7M\hlrD&#33!//))   r9NNNN) NNNNNNNNNr)r0r1r2_tied_weights_keys_tp_plan_pp_planrlr4rrrr5rrrrrlistrrrrr7r rrerfs@r:rrsHH2,0595959 ##"%!!1!12    1 12  !!1!12 ,'+BF1537KO59-1$(5934@ ##@ %-U->->$?@ !. @ u//0 @ "%tE4E4E/F(F"GH @   1 12@ ))*@ D>@ !!1!12@ c5<</0@ +,@  u,, -@ @ r9rc$eZdZfdZdZxZS)CsmBackboneModelEmbeddingsct|tj|j|j z|j |_|jdtj|j|j zdy)Naudio_tokens_offsetsF) persistent) rNrlrmrnrRrorsembed_audio_tokensregister_bufferr4rrus r:rlz#CsmBackboneModelEmbeddings.__init__}sn "$,,0D0DvGXGX0X[a[m[m"n  "ELL1E1E$FIZIZ$Zgl  r9cf|j||jz}|jd}|S)Nrr)rrsum)rXrvrs r:rz"CsmBackboneModelEmbeddings.forwards6..y4;T;T/TU #''A'. r9rrfs@r:rr|s  r9rc<eZdZfdZeefdZxZS)CsmBackboneModelcDt||t||_yr)rNrlrrqrus r:rlzCsmBackboneModel.__init__s  6v>r9c "t|di|S)a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) r8)rNr)rX super_kwargsr[s r:rzCsmBackboneModel.forwardsw...r9)r0r1r2rlrrrrerfs@r:rrs$? / /r9rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c eZdZddgZfdZdZdZdZefdZ fdZ dd e e jd e e jd e e jd e e jd e e jf dZ dd e jde ede e jde e j"de e jf fd Zee dd e jd e e jde e jd e e jde e jde eeee j"fde e j"d e e jde ede e jdeee jfdeed eeeffdZxZS)CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightct|||j|_tj|j |jd|_tj|j|j |_ tj||_ tj|j|_t!j"|j$|_|j)yrj)rNrlrormrrrsrrntext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initrus r:rlz$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE r9c.|jjSrrrqrXs r:get_input_embeddingsz0CsmForConditionalGeneration.get_input_embeddingss""///r9c&||j_yrr)rXvalues r:set_input_embeddingsz0CsmForConditionalGeneration.set_input_embeddingss+0(r9c|jjrO|j|jjj |j jjyyr)rItie_codebooks_embeddings_tie_or_clone_weightsrrqrrrJrs r: _tie_weightsz(CsmForConditionalGeneration._tie_weightssL ;; / /  & &##00CC""((55  0r9c|jddrt ||i|\}}nt ||i|}d}t|}t |j j Dcic]\}}|j|r||d|} }}t |jj jddi| | D]}t|j ||zd|vr|fS|Scc}}w)Noutput_loading_infoFdepth_decoder__from_model_config) getrNfrom_pretrainedlenvarsgeneration_configitems startswithrupdatedelattr) clsargsr}rJ loading_infoprefix prefix_lenattrrdepth_decoder_attrsr[s r:rz+CsmForConditionalGeneration.from_pretraineds ::+U 3"''"94"J6"J E<G+TzQCsmForConditionalGeneration._merge_input_ids_with_input_values..s&Orrxx{&Os)rdtypeiTas_tuple)rzr)$rrmrpaddiffr4rmaxrexpandrrno_gradziprSrrencode audio_codes transposeappendrget_audio_codes_maskrIaudio_token_idrrqonesrRlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)rXrvr r rrz audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrZ start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesrbatched_audio_token_idsaudio_codes_maskr!audio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss r:"_merge_input_ids_with_input_valuesz>CsmForConditionalGeneration._merge_input_ids_with_input_valuess*..y9  ##%==#4#45I6#R 01E1JKPPRM)-!*;q$A "O9O&P (,(8(8(?(? @U@UVW@X(Y '4'@'@'J'J1b'Q )00aA BB$'&O=N&O#O */++`qrZ\R]]&&rAq!5EQR 5S+TUr+'$(#3#3#H#HIZ#[ ! \$"[[77N(N: ..;;r'rxrzr|c 0t |d |||||d|}|}|jdk(rn|jd]|j ||jd|jd|jd}|j |d|ddd|S) N)rvr'rxrzr|rrzr r r)rvr r r)rzrrvr8)rNrndimrr=r) rXrvr'rxrzr|r}r merged_inputsr[s r:rz9CsmForConditionalGeneration.prepare_inputs_for_generation;sw< +)')      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM   "/"@MZbLcrvw r9ryr{rr}c |/|jdk(r |j||||} | d}| d}d}|jd|||||| | d| }|d}t| tr t | dn| }|j |dd|ddf}d}d}d}d}||dddddf}|jd|||jjd| }|ddddddfd k(jd }||d d|jjdz f}tjj|d d}|jd}||d|ddz ddf}||}|j d||| d|d| }|j"}||z}t%|||||j&|j(|j*| |j,nd| |j&nd| |j(nd||j* Sd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be infered from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrzr)rvrxryr'rzr{r|r)r&rrorrr r.r )rTr)rvrwr{ return_dictr) r%r/r*r&r'r(r)r+r,r-r.r8)r?r=rrPrrrrrIroallrRrmrrr(rr%r$r'r(r)r&)rXrvr rxr ryr'rzrr{r|rr}r@backbone_outputsbackbone_hidden_statesrbackbone_logitsr%r/r*depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r:rz#CsmForConditionalGeneration.forwardZsf  Y^^q%8 CC<)=vM*/:M"8,FI.4..  )%+')    "2!!48B>SV8W~ot4]k ,,'=aPQ>Q'RS ! $  $Q1WoO.D..&4;;KaKaekM "!Q(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $6D$6$6%1+F# + %  % !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  r9r) NNNNNNNNNNr)r0r1r2rrlrrr classmethodrrrr4rr=rrr5rrrrrrrrrr7r$rrerfs@r:rrs @1 0141-1/37;)- PBELL)PBu||,PB'u||4 PB & PB %,,  PBj,0595959 ##"%!!1!12    1 12  !!1!12 >'+/3157;37KO59-1$(5934[ ##[ u||,[ !. [ 'u||4 [ u//0 [ "%tE4E4E/F(F"GH[   1 12[ ))*[ D>[ !!1!12[ c5<</0[ +,[  u'' ([ [ r9r)rHrrhrr)? dataclassesrtypingrrr4torch.nnrmtransformers.utils.genericr cache_utilsrr generationr masking_utilsr modeling_outputsr r modeling_utilsrprocessing_utilsrutilsrrrrautorllama.modeling_llamarrrrrrrrconfiguration_csmrr generation_csmr! get_loggerr0rr$r<r@rBrDrFrHrhModulerQrrrr__all__r8r9r:ras "" 9.)/O-&KK   @.   H %  )6 )6  )6Z   -  X  >  '  [[  [4\ :'9\ \ ~ryy.c !1?c c L  /z//. P "46HP  P f  r9