rh}dZddlmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z mZddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZd dlmZej@e!Z"dZ#dZ$d#dZ%GddejLZ'dZ(GddejLZ)eGddeZ*eGdde*Z+edGdde*eZ,edGd d!e*Z-gd"Z.y)$zPyTorch CTRL model.)OptionalUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)Cache DynamicCache)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) CTRLConfigcPdtjdd|dzz|z z }||zS)Nri')torchpow)posi d_model_size angle_ratess y/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defnr!(s/eiiQ!V 'DEEK  cttj|tjj |j dtj|tjj |j d|}tj |dddddf}tj|dddddf}tj||gd}|S)Ndtyperrrdim) r!rarangeint64to unsqueezesincoscat)positionrr% angle_radssinescosines pos_encodings r positional_encodingr5-s XU[[144U;EEaH \588?II!LJ IIjADqD) *Eii 1add7+,G99eW-26L r"c tj||jdddd}|jd}|t j |z }|6|j d|j d} } ||| | z | d| fdzz }|||z}tj|d} || |z} tj| |} | | fS) Nrrr rr&gr')rmatmulpermuteshapenpsqrtsizesoftmax) qkvmaskattention_mask head_mask matmul_qkdkscaled_attention_logitsndnsattention_weightsoutputs r scaled_dot_product_attentionrL<s Q !Q1 56I B'"''"+5 (--b13J3O3OPR3SB4R" crc(9#:T#AA!"9N"J &=2F- 9 \\+Q /F $ $$r"c@eZdZdfd ZdZdZ ddZxZS)MultiHeadAttentionc|t|||_||_||_t ||jz |_tj|||_ tj|||_ tj|||_ tj|||_ t|_yN)super__init__ num_headsr layer_idxintdepthrLinearWqWkWvdenseset pruned_heads)selfrrSrT __class__s r rRzMultiHeadAttention.__init__Ws "("67 ))L,7))L,7))L,7YY|\: Er"c|j|jz}t|dk(ryt||j||j\}}t |j ||_t |j||_t |j||_t |j|d|_ |jt|z |_||jz|_|jj||_y)Nrrr') rrSlenrr]rrXrYrZr[union)r^headsattention_head_sizeindexs r prune_headszMultiHeadAttention.prune_headsfs"//4>>A u:? 7t~~Obdhduduv u%TWWe4$TWWe4$TWWe4' EqA #e*4/$..@ --33E:r"cx|j|d|j|j}|jgdS)Nr&rrrr )reshaperSrVr9)r^x batch_sizes r split_into_headsz#MultiHeadAttention.split_into_headsws- IIj"dnndjj Ayy&&r"c |jd} |j|}|j|}|j|}|j || }|j || }|j || }|#|j |||j d| i\}}t||||||} | djgd} | d}| j| d|j}|j|} | |fS)Nrcache_positionrhrr&) r:rXrYrZrlupdaterTrLr9rirr[)r^rAr@r?rB layer_pastrCrD use_cacheoutput_attentionsrnrkrKscaled_attentionattnoriginal_size_attentions r forwardzMultiHeadAttention.forward{sWWQZ GGAJ GGAJ GGAJ  ! !!Z 0  ! !!Z 0  ! !!Z 0  !$$Q4>>eps) rQrRrNmulti_head_attentionrffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)r^rrSrraterTr_s r rRzEncoderLayer.__init__sr $6|YZc$d!2<E,,|>,,|> 4(  4( r"c |j|} |j| | | ||||||| } | d} |j| } || z} |j| } |j | }|j |}| |z} | f| ddz}|S)NrprCrDrqrrrnrr)rrrrrr)r^rjrBrprCrDrqrrrnnormed attn_outputs attn_outputout1out2 ffn_outputoutputss r rvzEncoderLayer.forwards#00    !)/)1  #1o mmK0 ;t$XXd^ ]]:. j 'L,,r")g?Nrw)rxryrzrRrvr{r|s@r rrs! ) "r"rc"eZdZUeed<dZdZy)CTRLPreTrainedModelconfig transformerct|tjtfrm|jj j d|jj|j%|jj jyyt|tjrz|jj j d|jj|j2|jj |jjyyt|tjrJ|jj j|jj jdyy)zInitialize the weights.g)meanstdN?) isinstancerrWrweightdatanormal_rinitializer_rangebiaszero_ Embedding padding_idxrfill_)r^modules r _init_weightsz!CTRLPreTrainedModel._init_weightss  fryy&1 2 MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r"N)rxryrzr__annotations__base_model_prefixrr"r rrs %*r"rceZdZfdZdZdZdZe ddee jdee e e jdee jdee jd ee jd ee jd ee jd ee d ee dee dee dee jdee e jeffdZxZS) CTRLModelc t|||j|_|j|_t |j|jtj|_ tj|j|j|_tj|j |_tj$t'|jDcgc]:}t)|j|j*|j,|j.|<c}|_tj2|j|j4|_|j9ycc}w)Nrr)rQrRn_embdrn_layer num_layersr5 n_positionsrfloatr4rr vocab_sizewr embd_pdropdropout ModuleListrangern_headr resid_pdrophrlayer_norm_epsilon layernorm post_init)r^rrr_s r rRzCTRLModel.__init__s  "MM ../0B0BDDUDUW\WbWbcf//?zz&"3"34 v~~. V]]FMM6::vGYGYefg   fmm9R9RS  s ?E,c|jSrPr)r^s r get_input_embeddingszCTRLModel.get_input_embeddingss vv r"c||_yrPr)r^new_embeddingss r set_input_embeddingszCTRLModel.set_input_embeddingss r"c|jD]-\}}|j|jj|/y)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrf)r^heads_to_prunelayerrcs r _prune_headszCTRLModel._prune_headss>+002 BLE5 FF5M . . : :5 A Br" input_idspast_key_valuesrCtoken_type_ids position_idsrD inputs_embedsrqrroutput_hidden_states return_dictrnreturnc | | n|jj} ||n|jj}| | n|jj} | | n|jj} | | t d|G|j |||j}|jd|d}|jd}n0|#|jdd}|jd}n t d| |jn |j}d}|rt%j&||d|zt$j(| }|j+d}||dkr t d |j|d}|j+d j+d }|j-|j. }d|z t%j0|j.j2z}|j5||jj6}|I|jd|d}|j9|}|t;j<|j>z}nd}||j9|}|d}t%j@t%jB||z||zd j-|}|t;j<|j>z}|jDj-||_"|jD|ddf}||z|z}|jG|}| rdnd}| rdnd}tI|jJD]1\}}| r||fz}||||||||| | }|d}| s)||d fz }3|jM|}| r||fz}|r|jO}| stQd||||fDStS||||S)aE input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer&rz5You have to specify either input_ids or inputs_embedsFzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `DynamicCache` instead, e.g. `past_key_values=DynamicCache.from_legacy_cache(past_key_values)`.T)r%devicez$batch_size has to be defined and > 0rrr$rrrc3&K|] }|| ywrPr).0rAs r z$CTRLModel.forward..sbcbos)last_hidden_stater hidden_states attentions)*rrrrqruse_return_dict ValueError%warn_if_padding_and_no_attention_maskr=viewr:rrr logger warning_oncer from_legacy_cacheget_seq_lengthrr)longr,r+r%finfomin get_head_maskrrr;r<rtriuonesr4r enumeraterrto_legacy_cachetupler )r^rrrCrrrDrrqrrrrrnkwargs input_shaperkrreturn_legacy_cache past_lengthtoken_type_embedsseq_lenrB pos_embedsrall_hidden_statesall_attentionsrrrs r rvzCTRLModel.forwards=b2C1N-TXT_T_TqTq!*!6IDKKcd d  "  6 6y. Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U%.%:!! @T@T# Z?   U  #' *<<_MO:I:Uo446[\   << [_{5RZ_ZdZdmstL'11!4L  %Q !GHH+00R@N ,55a8BB1EN,..TZZ.@N!N2ekk$**6M6Q6QQN&&y$++2E2EF  %+00[_EN $~ 6  ):):!; ;  !    FF9-Mb/zz%**W{%:Gkg $D>g 'tng d^g !.g  uU\\"$;; <g g r"rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc eZdZdgZfdZe ddeejdee e ejdeejdeejdeejdeejd eejd eejd ee d ee d ee dee deejde e ejeffdZddZxZS)CTRLLMHeadModelzlm_head.weightct||t||_t j |j |jd|_|jy)NTr) rQrRrrrrWrrlm_headrr^rr_s r rRzCTRLLMHeadModel.__init__sG  $V,yy0A0AM  r"rrrCrrrDrlabelsrqrrrrrnrc ~| | n|jj} |j|||||||| | | | |  }|d}|j|}d}|*|j||fd|jj i|}| s|f|ddz}||f|zS|St |||j|j|jS)a  input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N) rrCrrrDrrqrrrrrnrrr)losslogitsrrr) rrrr loss_functionrrrrr)r^rrrCrrrDrrrqrrrrrnrtransformer_outputsr lm_logitsrrKs r rvzCTRLLMHeadModel.forwardsx&1%c $D>c 'tnc d^c !.c uU\\"$:: ;!c c Jdr"ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZfdZe ddeejdeeeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d ee de eeje ffdZxZS)CTRLForSequenceClassificationct|||j|_t||_t j |j|jd|_|jy)NFr) rQrR num_labelsrrrrWr classifierrrs r rRz&CTRLForSequenceClassification.__init__OsR   ++$V,))FMM4??O r"rrrCrrrDrrrqrrrrrc | | n|jj} |j|||||||| | | |  } | d}|j|}||jdd\}}n|jdd\}}|jj |dk7r t d|jj d}n|||jj k7j|jtj}tj|jd|jtj}||zjd}n.d}tj|jj d |tj||j |f}d}||jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}| s|f| ddz}||f|zS|St7||| j8| j:S)a2 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```N) rrCrrrDrrqrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r&)rr%z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rr rr)rrrrr: pad_token_idrr+rrint32r)argmaxrrr_rx problem_typerr%rrUrsqueezerrrrrr)r^rrrCrrrDrrrqrrrrr rr rksequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrKs r rvz%CTRLForSequenceClassification.forwardXsP&1%!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6#%(;AB(??F)-)9TGf$ Ev E' -;;*55   r"r)rxryrzrRrrrrrrrrrrrvr{r|s@r rrCsW15EI6:59371559-1$(,0/3&*p E,,-p "%e.?.?(@"ABp !!2!23 p !!1!12 p u//0 p E--.p   1 12p ))*p D>p $D>p 'tnp d^p  uU\\"$<< =p p r"r)rrrrr)/__doc__typingrrnumpyr;rrtorch.nnrrr cache_utilsr r generationr modeling_outputsr rrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_ctrlr get_loggerrxrr!r5rLModulerNrrrrrr__all__rr"r r4s$ " AA.)ii-YY+   H % %4DDN`/299/d*/***L #L L ^ Ad)?Ad  AdH { $7{  { | cr"