rhdZddlZddlmZddlmZddlmZmZddl Z ddl Z ddl m Z ddl m Z mZmZdd lmZmZerdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddlm'Z'ddl(m)Z)e'jTe+Z,dZ-dZ.dZ/Gdde j`Z1Gdde j`Z2Gdde j`Z3Gdde j`Z4Gdde j`Z5Gd d!e j`Z6Gd"d#eZ7Gd$d%e j`Z8Gd&d'e j`Z9Gd(d)e j`Z:Gd*d+e j`Z;Gd,d-e j`Z<Gd.d/e j`Z=Gd0d1e j`Z>eGd2d3e$Z?eed45Gd6d7eZ@eGd8d9e?ZAed:5Gd;de?ZCed?5Gd@dAe?ZDedB5GdCdDe?ZEeGdEdFe?ZFeGdGdHe?ZGeGdIdJe?ZHgdKZIy)LzPyTorch FNet model.N) dataclass)partial)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)auto_docstringis_scipy_available)linalg)ACT2FN)GradientCheckpointingLayer) BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)logging) FNetConfigc|jd}|d|d|f}|jtj}tjd|||S)z4Applies 2D matrix multiplication to 3D input arrays.rNzbij,jk,ni->bnk)shapetypetorch complex64einsum)xmatrix_dim_onematrix_dim_two seq_lengths y/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/fnet/modeling_fnet.py_two_dim_matmulr*8sNJ#KZK*$<=N uA <<(!^^ LLct|||SN)r*)r%r&r's r)two_dim_matmulr.As 1nn ==r+c|}tt|jddD]#}tjj ||}%|S)z Applies n-dimensional Fast Fourier Transform (FFT) to input array. Args: x: Input n-dimensional array. Returns: n-dimensional Fourier transform of input n-dimensional array. rN)axis)reversedrangendimr"fft)r%outr0s r)fftnr6FsG Cqvvqr*+,iimmCdm+, Jr+c*eZdZdZfdZddZxZS)FNetEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cxt|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j |j |_tj |j"|_|j'dt)j*|jj-dd|j'dt)j.|j0j3t(j4dy) N) padding_idxeps position_ids)rF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsLinear projectionDropouthidden_dropout_probdropoutregister_bufferr"arangeexpandzerosr=sizelongselfconfig __class__s r)rDzFNetEmbeddings.__init__Ys< !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>ST))F$6$68J8JKzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r+cX||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} || z} |j|} | | z } |j| } |j| } |j| } | S)Nr>rr@rrBdevice)rYr=hasattrr@rWr"rXrZrarIrMrKrNrQrT) r\ input_idsr@r= inputs_embeds input_shaper(buffered_token_type_ids buffered_token_type_ids_expandedrM embeddingsrKs r)forwardzFNetEmbeddings.forwardos=  #..*K',,.s3K ^  ,,Q  ^ J"%:: "66|D)) ^^J/ __Z0 \\*- r+)NNNN)__name__ __module__ __qualname____doc__rDri __classcell__r^s@r)r8r8VsQ ,!r+r8c*eZdZfdZdZdZxZS)FNetBasicFourierTransformcDt||j|yr-)rCrD_init_fourier_transformr[s r)rDz"FNetBasicFourierTransform.__init__s  $$V,r+c|js+ttjjd|_y|j dkrtr|jdtjtj|jtj|jdtjtj|jtjtt|j |j"|_yt%j&dt|_yt|_y) N)rdimdft_mat_hiddenrA dft_mat_seq)r&r'zpSciPy is needed for DFT matrix calculation and is not found. Using TPU optimized fast fourier transform instead.)use_tpu_fourier_optimizationsrr"r4r6fourier_transformrJr rUtensorrdftrGr#tpu_short_seq_lengthr.rzryrwarning)r\r]s r)rsz1FNetBasicFourierTransform._init_fourier_transforms33%,UYY^^%HD "  + +t 3!#$$$ell6::f>P>P3QY^YhYh&i$$!5<< 6;V;V0W_d_n_n#o*1"43C3CTXTgTg*&**.&%)D "r+c>|j|j}|fSr-)r|real)r\ hidden_statesoutputss r)riz!FNetBasicFourierTransform.forwards" ((7<<zr+)rjrkrlrDrsrirnros@r)rqrqs-*.r+rqc$eZdZfdZdZxZS)FNetBasicOutputct|tj|j|j |_yNr;)rCrDrrNrGrOr[s r)rDzFNetBasicOutput.__init__s, f&8&8f>S>STr+c.|j||z}|Sr-)rNr\r input_tensors r)rizFNetBasicOutput.forwards|m'CD r+rjrkrlrDrirnros@r)rrsUr+rc$eZdZfdZdZxZS)FNetFourierTransformcbt|t||_t ||_yr-)rCrDrqr\routputr[s r)rDzFNetFourierTransform.__init__s& -f5 %f- r+cX|j|}|j|d|}|f}|SNr)r\r)r\r self_outputsfourier_outputrs r)rizFNetFourierTransform.forwards1yy/ \!_mD!#r+rros@r)rrs . r+rcVeZdZfdZdej dej fdZxZS)FNetIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr-) rCrDrrPrGintermediate_sizedense isinstance hidden_actstrrintermediate_act_fnr[s r)rDzFNetIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r+rreturncJ|j|}|j|}|Sr-)rrr\rs r)rizFNetIntermediate.forwards& =1 00? r+rjrkrlrDr"Tensorrirnros@r)rrs#9U\\ellr+rcneZdZfdZdej dej dej fdZxZS) FNetOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) rCrDrrPrrGrrNrOrRrSrTr[s r)rDzFNetOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r+rrrcr|j|}|j|}|j||z}|Sr-)rrTrNrs r)rizFNetOutput.forwards7 =1  ]3 }|'CD r+rros@r)rrs1> U\\RWR^R^r+rc*eZdZfdZdZdZxZS) FNetLayerct||j|_d|_t ||_t ||_t||_ yNr) rCrDchunk_size_feed_forward seq_len_dimrfourierr intermediaterrr[s r)rDzFNetLayer.__init__sI '-'E'E$+F3 ,V4 ( r+c|j|}|d}t|j|j|j|}|f}|Sr)rrfeed_forward_chunkrr)r\rself_fourier_outputsr layer_outputrs r)rizFNetLayer.forwardsO#||M:-a00  # #T%A%A4CSCSUc  /r+cL|j|}|j||}|Sr-)rr)r\rintermediate_outputrs r)rzFNetLayer.feed_forward_chunks*"//?{{#6G r+)rjrkrlrDrirrnros@r)rrs) r+rc&eZdZfdZddZxZS) FNetEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rCrDr]r ModuleListr2num_hidden_layersrlayergradient_checkpointing)r\r]_r^s r)rDzFNetEncoder.__init__ sN  ]]uVE]E]?^#_!If$5#_` &+#$`sA#c|rdnd}t|jD]\}}|r||fz}||}|d}|r||fz}|std||fDSt||S)Nrc3&K|] }|| ywr-r).0vs r) z&FNetEncoder.forward.. sXq!-Xs)last_hidden_stater) enumeratertupler)r\routput_hidden_states return_dictall_hidden_statesi layer_module layer_outputss r)rizFNetEncoder.forwards"6BD(4 -OA|#$58H$H!(7M)!,M  - 1]4D D X]4E$FXX XN_``r+)FTrros@r)rr s, ar+rcVeZdZfdZdej dej fdZxZS) FNetPoolerct|tj|j|j|_tj |_yr-)rCrDrrPrGrTanh activationr[s r)rDzFNetPooler.__init__'s9 YYv1163E3EF '')r+rrc\|dddf}|j|}|j|}|Sr)rr)r\rfirst_token_tensor pooled_outputs r)rizFNetPooler.forward,s6+1a40 #56  6 r+rros@r)rr&s#$ U\\ellr+rcVeZdZfdZdej dej fdZxZS)FNetPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) rCrDrrPrGrrrrrtransform_act_fnrNrOr[s r)rDz$FNetPredictionHeadTransform.__init__7s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr+rrcl|j|}|j|}|j|}|Sr-)rrrNrs r)riz#FNetPredictionHeadTransform.forward@s4 =1 --m< }5 r+rros@r)rr6s$UU\\ellr+rc,eZdZfdZdZddZxZS)FNetLMPredictionHeadcDt|t||_t j |j |j|_t jtj|j|_ |j|j_ yr-) rCrDr transformrrPrGrFdecoder Parameterr"rXbiasr[s r)rDzFNetLMPredictionHead.__init__Hsi 4V<yy!3!3V5F5FG LLV->->!?@  II r+cJ|j|}|j|}|Sr-)rrrs r)rizFNetLMPredictionHead.forwardSs$}5  ]3 r+c|jjjjdk(r|j|j_y|jj|_y)Nmeta)rrrar!r\s r) _tie_weightsz!FNetLMPredictionHead._tie_weightsXsC <<   # # ( (F 2 $ DLL  ))DIr+)rN)rjrkrlrDrirrnros@r)rrGs & *r+rc$eZdZfdZdZxZS)FNetOnlyMLMHeadcBt|t||_yr-)rCrDr predictionsr[s r)rDzFNetOnlyMLMHead.__init__bs /7r+c(|j|}|Sr-)r)r\sequence_outputprediction_scoress r)rizFNetOnlyMLMHead.forwardfs ,,_=  r+rros@r)rras 8!r+rc$eZdZfdZdZxZS)FNetOnlyNSPHeadclt|tj|jd|_yNru)rCrDrrPrGseq_relationshipr[s r)rDzFNetOnlyNSPHead.__init__ms'  " &*<*.  - KK   " " $ MM   $ $S ).r+N)rjrkrlr__annotations__base_model_prefixsupports_gradient_checkpointingrrr+r)rrs &*#*r+rz0 Output type of [`FNetForPreTraining`]. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<y)FNetForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsr) rjrkrlrmr rr" FloatTensorrr r rrrr+r)r r sd )-D(5$$ %,59x 1 129;?Xe&7&78?8z5You have to specify either input_ids or inputs_embedsrxzThe `tpu_short_seq_length` in FNetConfig should be set equal to the sequence length being passed to the model when using TPU optimizations.r@r`)rcr=r@rd)rrrr)r pooler_outputr)r]ruse_return_dict ValueErrorrYr{rrarbrhr@rWr"rXrZrrrr)r\rcr@r=rdrrre batch_sizer(rarfrgembedding_outputencoder_outputsrrs r)rizFNetModel.forwards%9$D $++JjJj &1%cd d  "#..*K%0 "J  &',,.s3K%0 "J TU U KK 5 5d" 00J>;  &/%:!! @T@T  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z??%)' +  ,, !5#'  *!,8< 8O O4UY #]3oab6II I)-')77  r+)T)NNNNNN)rjrkrlrmrDrrr rr" LongTensorrboolrrrrirnros@r)rrs /015593759/3&*C E,,-C !!1!12C u//0 C   1 12 C 'tn C d^C  uo% &C C r+rz FNet Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c,eZdZddgZfdZdZdZe ddee jdee jdee jd ee jd ee jd ee jd ee d ee de e effdZxZS)FNetForPreTrainingcls.predictions.decoder.biascls.predictions.decoder.weightct||t||_t ||_|j yr-)rCrDrrrclsrr[s r)rDzFNetForPreTraining.__init__"s4  f% '/ r+cB|jjjSr-r*rrrs r)get_output_embeddingsz(FNetForPreTraining.get_output_embeddings+xx##+++r+c||jj_|j|jj_yr-r*rrrr\new_embeddingss r)set_output_embeddingsz(FNetForPreTraining.set_output_embeddings.,'5$$2$7$7!r+rcr@r=rdlabelsnext_sentence_labelrrrc ||n|jj}|j||||||} | dd\} } |j| | \} } d}|u|st }|| j d|jj |j d}|| j dd|j d}||z}|s| | f| ddz}||f|zS|St|| | | jS)aH labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForPreTraining.from_pretrained("google/fnet-base") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Nr@r=rdrrrur>)r r r r) r]rrr*r viewrFr r)r\rcr@r=rdr5r6rrrrrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossrs r)rizFNetForPreTraining.forward2s5J&1%rur logitsr) r]rrr*r r9rFrr)r\rcr@r=rdr5rrrrrr<r;rs r)rizFNetForMaskedLM.forwards"&1%:K[b[p[pqqr+NNNNNNN)rjrkrlr?rDr-r3r rr"rr$rrrrirnros@r)rArAxs8:Z[,8-115/304)-/3&*'rELL)'r!.'ru||, 'r   - 'r & 'r'tn'rd^'r un$ %'r'rr+rAzT FNet Model with a `next sentence prediction (classification)` head on top. ceZdZfdZe d deejdeejdeejdeejdeejdeedeed e e e ffd Z xZ S) FNetForNextSentencePredictionct||t||_t ||_|j yr-)rCrDrrrr*rr[s r)rDz&FNetForNextSentencePrediction.__init__rCr+rcr@r=rdr5rrrc d|vr+tjdt|jd}||n|jj }|j ||||||} | d} |j| } d} |2t} | | jdd|jd} |s| f| ddz}| | f|zS|St| | | jS) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForNextSentencePrediction.from_pretrained("google/fnet-base") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ```r6zoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nr8rr>rurG) warningswarn FutureWarningpopr]rrr*r r9rr)r\rcr@r=rdr5rrkwargsrrseq_relationship_scoresr=r;rs r)riz%FNetForNextSentencePrediction.forwards H !F * MM%  ZZ 56F%0% 1` a classification loss is computed (Cross-Entropy). Nr8r regressionsingle_label_classificationmulti_label_classificationr>rurG)r]rrrTrY problem_typerXrBr"rZintr squeezer r9rrr)r\rcr@r=rdr5rrrrrHr r;rs r)riz%FNetForSequenceClassification.forward#s"&1%r8rurG) r]rr r9rYrrTrYr rr)r\rcr@r=rdr5rr num_choicesrrrHreshaped_logitsr r;rs r)rizFNetForMultipleChoice.forwardlsR&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  )) )%'!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E(d?ZaZoZoppr+rI)rjrkrlrDr rr"rr$rrrrirnros@r)rbrb`s-115/304)-/3&*LqELL)Lq!.Lqu||, Lq   - Lq & Lq'tnLqd^Lq u// 0LqLqr+rbceZdZfdZe d deejdeejdeejdeejdeejdeedeed e e e ffd Z xZ S) FNetForTokenClassificationc,t|||j|_t||_t j |j|_t j|j|j|_ |jyr-rWr[s r)rDz#FNetForTokenClassification.__init__si   ++f% zz&"<"<= ))F$6$68I8IJ r+rcr@r=rdr5rrrc||n|jj}|j||||||}|d} |j| } |j | } d} |rurG) r]rrrTrYr r9rXrr)r\rcr@r=rdr5rrrrrHr r;rs r)riz"FNetForTokenClassification.forwards&1%rv) ignore_indexru)r  start_logits end_logitsr) r]rrrosplitr` contiguouslenrYclampr rr)r\rcr@r=rdrprqrrrrrHrtrur: ignored_indexr; start_lossend_lossrs r)riz FNetForQuestionAnswering.forwards&1%)rjrkrlrDr rr"rr$rrrrirnros@r)rmrms -115/3042604/3&*4 ELL)4 !.4 u||, 4   - 4 "%,,/ 4   -4 'tn4 d^4  u22 34 4 r+rm) rArbrKr&rmrUrirrr)JrmrN dataclassesr functoolsrtypingrrr"torch.utils.checkpointrtorch.nnrr r utilsr r scipyr activationsrmodeling_layersrmodeling_outputsrrrrrrrrrmodeling_utilsr pytorch_utilsrrconfiguration_fnetr get_loggerrjloggerr*r.r6Moduler8rqrrrrrrrrrrrrrr rr&rArKrUrbrirm__all__rr+r)rs!" AA7!9   .6*   H %M>  :RYY:z# #Lbii 299  ryy   *6a"))a8   "))"*299*4!bii!&bii& 9299 9*/**.  ={=  =$b #b b J V ,V  V r;r);r;r| R $7R  R j Fg$7Fg  FgRXq/XqXqv6d!46d6drA 2A A H r+