rhTdZddlZddlmZmZddlZddlZddlmZddlm Z m Z m Z m Z ddlm ZddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZddl m!Z!m"Z"ddl#m$Z$e"jJe&Z'd(dZ(GddejRZ*GddejRZ+GddeZ,e!GddeZ-e!Gdde-Z.e!dGdde-eZ/e!d Gd!d"e-Z0e!Gd#d$e-Z1e!Gd%d&e-Z2gd'Z3y))zPyTorch MPT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache)GenerationMixin)!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging) MptConfigcRtjd|z dtj|jddd|}dt j t j |z}tjd|dztj|j}|||z z}dtjd|z }|jd|dd}||k7r9tj|ddddddf|ddddddfgdddd|df}||z}|jdS) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedevice?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopess w/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensorr6-s LL_,au{{6 R W WXY[\^_ap qE $))I*> ?? <</!35;;v V \ \ ^D >$88 9D 599Q% %F [[0!Q 7Fy(vaAsl3VAssCK5HIqQRSU_V_U_adRde FNE == c eZdZdZd dedeeffd Z d dejdejdee deejd eejf d Z xZ S) MptAttentionzzMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. config layer_idxct||j|_|j|_|j|_|j|jz|_|jj|_|j4dtj|j|jz z |_|jj|_ |jj|_ tj|jd|jzd|_tj|j|jd|_||_y)Nrr Fbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler&sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projr;)selfr:r; __class__s r5r@zMptAttention.__init__Is !--~~ $00((DLL8 #//==    %!"TYYt/?/?$,,/N%O!OD $00;;**33 IId..D4D4D0D5Q  $"2"2D4D4D5Q "r7 hidden_states position_biaspast_key_valueattention_maskcache_positioncp|jdd\}}|j|}|jr(|j|j |j}|j dd\} } } | j |||j |jjdd} | j |||j |jjdd} | j |||j |jjdd} |%d|i} |j| | |j| \} } tj| | jdd|jz} ||n||jz}|t|jdk7r!t!d t|j| jd}t#d |j%d|z }t#d |j%d|z }|dd|d|df}| |z} |9| j'|tj(| j*j,} t.j0j3| j5dj7| j*}t.j0j9||j:|j< }tj|| }|j?d dddjAjC||d}|jE|}||fS) Nr)minmaxr r rrUz6Expecting position_bias shape to be 3 dimensions, got rptraining)#shaperMrKclampchunkreshaperBrE transposeupdater;r"matmulrGget_seq_lengthlen ValueErrorrXsize masked_fillfinforrWrr softmaxr*todropoutrJr]permute contiguousr%rN)rOrQrRrSrTrU batch_size seq_length mixed_qkv query_states key_states value_states cache_kwargsattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputs r5forwardzMptAttention.forwardYs"/!4!4Ra!8 JIIm, ==!T]]N NI1:1J. j,#++J DLLRVR_R_`jjklnop '' J dmm\ffghjkl #++J DLLRVR_R_`jjklnop  %,n=L'5'<'rBr.r9attnnorm_2rffnrFrI dropout_raterDropoutresid_attn_dropout)rOr:r;rArPs r5r@zMptBlock.__init__s ((  1J1JK   3  1J1JK  &>"..99"$**T->->"?r7rQrRrT layer_past use_cacheoutput_attentionsrUc|j|}|} |j|||||\} } |j| | z}|j|}|} |j || } | | fS)N)rRrTrSrU)rrrrr) rOrQrRrTrrrrUlayernorm_outputr attn_outputsr|rs r5rzMptBlock.forwards ;;}5 &*YY ')%) &/& " l// =H ;;}5!*H5|##r7r)NFFN) rrrrrrr@r"rr boolrrrs@r5rrs@y@Xc]@2'+"'15"$||"$||"$ "$ UO "$  "$ "$!."$r7rceZdZUeed<dZdZdgZdgZfdZ de jfdZ e d eeej ej fd eeej ej ffd ZxZS) MptPreTrainedModelr: transformerTrz lm_head.*.c$t||i|yr)r?r@)rOinputskwargsrPs r5r@zMptPreTrainedModel.__init__s &+F+r7modulect|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|trV|j$|jjj|jjjdyy)zInitialize the weights.g)meanstdNr) isinstancerrLweightdatanormal_r:initializer_ranger>zero_ Embedding padding_idxrfill_)rOrs r5 _init_weightsz MptPreTrainedModel._init_weightss  fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  *{{&   &&( MM   $ $S )+r7rSrcl|ddj\}}||ztfd|DS)zw Converts the cache to the format expected by Mpt, i.e. to tuple(tuple([batch_size * num_heads, ...])) rc3vK|]0}|dj|djf2yw)rrN)ra).0rbatch_size_times_num_headsrErqs r5 z;MptPreTrainedModel._convert_to_mpt_cache..sK 1 %%&@(JW1 %%&@*hW  s69)r^tuple)rSrpr.rrErqs @@@r5_convert_to_mpt_cachez(MptPreTrainedModel._convert_to_mpt_cachesM7EQ6G6J6P6P3 Ix%/)%;" -    r7)rrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_keys_to_ignore_on_load_missingr@rModuler staticmethodrr"rrrrs@r5rrs %&*## '4o#,*BII*" eELL%,,$>?@ uU\\5<</0 1  r7rceZdZdeffd ZdZddZdejfdZ e dde ejde e eeejejfd fefd e ejd e ejd e ed e ede ede ede ejde eejd feffdZxZS)MptModelr:c t|||j|_|j|_t j |j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_d|j _d|_|j'ycc}w)N)r;rF)r?r@rArBr.rr vocab_sizewte ModuleListrangen_layersrblocksrrnorm_fr>gradient_checkpointing post_init)rOr:irPs r5r@zMptModel.__init__s  !--<< 1 143C3CDmmERXRaRaLb$cqXf%B$cd   0 0f6O6OP  &+# %dsC7c|jSrr)rOs r5get_input_embeddingszMptModel.get_input_embeddings$s xxr7ct||||Sr)r6)rOr.r/r0rs r5r6zMptModel.build_mpt_alibi_tensor's%i.RXYYr7new_embeddingsc||_yrrrOrs r5set_input_embeddingszMptModel.set_input_embeddings*s !r7 input_idspast_key_values.rT inputs_embedsrroutput_hidden_states return_dictrUrc ||n|jj}||n|jj}||n|jj}||n|jj}| | t d||j \} } n||j \} } } n t d|jr%|jr|rtjdd}||j|}d}|r.sghgts)last_hidden_staterrQ attentions)r:rrruse_return_dictrgr^rr]logger warning_oncerrr r from_legacy_cacherer"onesrrlr6r.rCrrrrto_legacy_cacherr)rOrrrTrrrrrrUrrprq_return_legacy_cacherQall_self_attentionsall_hidden_statespast_key_values_lengthseq_length_with_pastr1 causal_maskblockoutputss r5rzMptModel.forward-s62C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "%.__ "J  &(5(;(; %J ATU U  & &4==##p"   HHY/M# Z?"&    U  +<<_MO% $5b4"6BDFUE`!?!?!Afg),BB  !"ZZ5I(JS`SgSghN+..}/C/CDN++DNNDKK w $D>w 'tnw d^w !.w  uU\\3&')RR Sw w r7rz The MPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introceZdZdgZdeffd ZdejfdZe dde ejde e e ejejfdfd e ejd e ejd e ejd e e d e e de e de e de ejdee ejeffdZxZS)MptForCausalLMzlm_head.weightr:ct||t||_t j |j |jd|_|jyNFr=) r?r@rrrrLrArlm_headrrOr:rPs r5r@zMptForCausalLM.__init__sI  #F+yy!3!3V5F5FUS  r7rc||_yr)r rs r5set_output_embeddingsz$MptForCausalLM.set_output_embeddingss % r7rr.rTrlabelsrrrrrUrc | | n|jj} |j|||||||| |  } | d} |j| }d}|E|j |j }|j ||fd|jji| }| s|f| ddz}||f|zS|St||| j| j| jS)a\ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)rrTrrrrrrUrrrlosslogitsrrQr) r:rrr rlr loss_functionrrrrQr)rOrrrTrrrrrrrUrtransformer_outputsrQ lm_logitsrrs r5rzMptForCausalLM.forwards@&1%F $D>F 'tnF d^F !.F  uU\\"$EE FF F r7ra The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cXeZdZdeffd Ze ddeejdee e ejejfdfdeejdeejdeejd ee d ee d ee d ee d e e eje ffdZxZS)MptForSequenceClassificationr:ct|||j|_t||_t j |j|jd|_|jyr ) r?r@ num_labelsrrrrLrAscorerr s r5r@z%MptForSequenceClassification.__init__sV   ++#F+YYv1163D3D5Q  r7rr.rTrrrrrrrc r| | n|jj} |j|||||||| } | d} |j| } ||jd} n|jd} |jj | dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj| | j |f}d}|^|jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}nc|||}nY|jj"d k(rt1}|||}n,|jj"d k(rt3}|||}| s|f| ddz}||f|zS|St5||| j6| j8| j: S)6 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrTrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.rY)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r:rrrr^ pad_token_idrgrlrr"r$r#argmaxrrrPr problem_typerrlongrr r-rrrrrQr)rOrrrTrrrrrrrrQrrplast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r5rz$MptForSequenceClassification.forwards<&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- v6))-II,. v6#%(;AB(??F)-)9TGf$ Ev E/ /??-;;*55   r7r)rrrrr@rrr"rrrrrrrrrs@r5rrsy15SW1504)-$(,0/3&*d E,,-d "%ellELL.H(I3(N"OPd !. d   - d & d D>d $D>d 'tnd d^d  uU\\"$DD Ed d r7rcXeZdZdeffd Ze ddeejdee e ejejfdfdeejdeejdeejd ee d ee d ee d ee d e e eje ffdZxZS)MptForTokenClassificationr:ct|||j|_t||_t |dr|j |j }n't |dr|j |j}nd}tj||_ tj|j|j|_ |jy)Nclassifier_dropoutrg?)r?r@rrrhasattrr.rrrrmrLrA classifierr)rOr:r.rPs r5r@z"MptForTokenClassification.__init__s   ++#F+ 6/ 0V5N5N5Z!'!:!:  V- .63H3H3T!'!6!6 !$ zz"45 ))F$6$68I8IJ r7rr.rTrrrrrrrc | | n|jj} |j|||||||| } | d} |j| } |j | } d}|l|j | j }|j\}}t}|| j||z|j|j||z}| s| f| ddz}||f|zS|St|| | j| jS)rNrrr)rrrQr)r:rrrmr0rlrr^rr%rrrQr)rOrrrTrrrrrrdeprecated_argumentsrrQrrrprqr*rs r5rz!MptForTokenClassification.forwards+>&1%B $D>B 'tnB d^B  uU\\"$99 :B B r7r,ceZdZfdZe d deejdeejdeejdeejdeejdee dee d ee d e e e ffd Z xZS) MptForQuestionAnsweringct||t||_t j |j d|_|jy)Nr) r?r@rrrrLrA qa_outputsrr s r5r@z MptForQuestionAnswering.__init__sA  #F+))F$6$6: r7rrTrstart_positions end_positionsrrrrc "||n|jj}|j||||||} | d} |j| } | j dd\} } | j dj } | j dj } d}||t|jdkDr|j d}t|jdkDr|j d}| jd}|jd|}|jd|}t|}|| |}|| |}||zdz }|s| | f| ddz}||f|zS|St|| | | j| j S) rN)rTrrrrrrrYr ) ignore_indexr)r start_logits end_logitsrQr)r:rrr6splitr-rorfrhr_rrrQr)rOrrTrr7r8rrrrsequence_outputrr;r< total_loss ignored_indexr* start_lossend_lossrs r5rzMptForQuestionAnswering.forwards2&1%E 'tnE d^E  u22 3E E r7r4)rrrrr,r4r)4rr&typingrrr"torch.utils.checkpointrtorch.nnrrrr r r cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrutilsrrconfiguration_mptr get_loggerrrr6rr9rrrrrrr,r4__all__rr7r5rQsf " LL$.)I9.,(   H %.F)299F)RRYY*7$)7$t, , , ^X !X X v U 'U  U p  o #5o  o dU 2U U pO 0O O d r7