rh8dZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl m Z ddl m Z mZmZddlmZmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZm Z m!Z!dd l"m#Z#m$Z$ddl%m&Z&e$jNe(Z)dZ*Gdde jVZ,e#GddeZ-Gdde jVZ.Gdde jVZ/Gdde jVZ0Gdde jVZ1Gdde jVZ2Gdd e jVZ3Gd!d"e jVZ4Gd#d$eZ5Gd%d&e jVZ6Gd'd(e jVZ7Gd)d*e jVZ8e#Gd+d,e-Z9Gd-d.e jVZ:e#Gd/d0e-Z;Gd1d2e jVZ<e#d34Gd5d6e-Z=e#Gd7d8e-Z>e#Gd9d:e-Z?e#Gd;dzPyTorch ConvBERT model.N) attrgetter)CallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNget_activation)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging)ConvBertConfigc ddl}tj j |}tjd||jj|}i}|D]A\}}tjd|d||jj||} | ||<Cddd d d d d d} |jdkDrd} nd} t|jD]:} d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d | d| d!<d| d"| d| d#<d| d$| d| d%<d| d&| d| d'<d| d(| d| d)<d| d*| d| d+<d| d,| d| d-<d| d.| d| d/<d| d0| d| d1<d| d2| d| d3<d| d4| d| d5<d| d6| d7| d| d8<d| d6| d9| d| d:<d| d;| d7| d| d<<d| d;| d9| d| d=<d| d>| d| d?<d| d@| d| dA<=|jD]} | d}t|}||}| |}t!j"||}tjdB|dC|dD|j%d7r.|j%dEs|j%dFs |j&}|j%dGr|j)ddHd}|j%dIr|j)dHdd}|j%dJr|j+dK}||_|S#t$rtjdwxYw)Lz'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape z"electra/embeddings/word_embeddingsz&electra/embeddings/position_embeddingsz(electra/embeddings/token_type_embeddingsz"electra/embeddings/LayerNorm/gammaz!electra/embeddings/LayerNorm/betaz!electra/embeddings_project/kernelzelectra/embeddings_project/bias)z!embeddings.word_embeddings.weightz%embeddings.position_embeddings.weightz'embeddings.token_type_embeddings.weightzembeddings.LayerNorm.weightzembeddings.LayerNorm.biaszembeddings_project.weightzembeddings_project.biasrg_densedensezelectra/encoder/layer_z/attention/self/query/kernelzencoder.layer.z.attention.self.query.weightz/attention/self/query/biasz.attention.self.query.biasz/attention/self/key/kernelz.attention.self.key.weightz/attention/self/key/biasz.attention.self.key.biasz/attention/self/value/kernelz.attention.self.value.weightz/attention/self/value/biasz.attention.self.value.biasz./attention/self/conv_attn_key/depthwise_kernelz4.attention.self.key_conv_attn_layer.depthwise.weightz./attention/self/conv_attn_key/pointwise_kernelz4.attention.self.key_conv_attn_layer.pointwise.weightz"/attention/self/conv_attn_key/biasz(.attention.self.key_conv_attn_layer.biasz'/attention/self/conv_attn_kernel/kernelz(.attention.self.conv_kernel_layer.weightz%/attention/self/conv_attn_kernel/biasz&.attention.self.conv_kernel_layer.biasz&/attention/self/conv_attn_point/kernelz%.attention.self.conv_out_layer.weightz$/attention/self/conv_attn_point/biasz#.attention.self.conv_out_layer.biasz/attention/output/dense/kernelz.attention.output.dense.weightz!/attention/output/LayerNorm/gammaz".attention.output.LayerNorm.weightz/attention/output/dense/biasz.attention.output.dense.biasz /attention/output/LayerNorm/betaz .attention.output.LayerNorm.biasz/intermediate/z/kernelz.intermediate.dense.weightz/biasz.intermediate.dense.biasz/output/z.output.dense.weightz.output.dense.biasz/output/LayerNorm/gammaz.output.LayerNorm.weightz/output/LayerNorm/betaz.output.LayerNorm.biaszTF: z, PT:  z/intermediate/g_dense/kernelz/output/g_dense/kernelz/depthwise_kernelz/pointwise_kernelz/conv_attn_key/bias) tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variable num_groupsrangenum_hidden_layersnamed_parametersrtorch from_numpyendswithTpermute unsqueezedata)modelconfigtf_checkpoint_pathtftf_path init_varstf_datanameshapearray param_mappinggroup_dense_namejparam param_name retrieverresulttf_namevalues /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/convbert/modeling_convbert.pyload_tf_weights_in_convbertrM1soggoo01G KK8 BC''0IG  e (l5'BC&&w5  .R1Y3]'K%H%H#DM1$" 6++ ,Cw$QC'C D qc)EFG%QC'A B qc)CDE%QC'A B qc)CDE%QC'? @ qc)ABC%QC'C D qc)EFG%QC'A B qc)CDE%QC'U V qc)]^_%QC'U V qc)]^_%QC'I J qc)QRS%QC'N O qc)QRS%QC'L M qc)OPQ%QC'M N qc)NOP%QC'K L qc)LMN%QC'E F qc)GHI%QC'H I qc)KLM%QC'C D qc)EFG%QC'G H qc)IJK%QC~6F5Gw O qc)CDE%QC~6F5Gu M qc)ABC%QCx0@/A I qc)=>?%QCx0@/A G qc);<=%QC'> ? qc)ABCG]]^\__uDv qc)?@AGCwJ'')1X z* 5! +  !12 d7)6*Q78   I &##$BC''(@A!GGE   / 0MM!Q*E   / 0MM!Q*E   1 2OOB'E #$ Lk   Q   s L++ M c eZdZdZfdZ d deejdeejdeejdeejdejf dZ xZ S) ConvBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_ids)rr"F) persistenttoken_type_idsdtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr2arangeexpandzerosrTsizelongselfr: __class__s rLrZzConvBertEmbeddings.__init__s !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&;&;AVAVWzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  input_idsrVrT inputs_embedsreturnc2||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} |j|} || z| z} |j| } |j| } | S)Nr"rrVrrXdevice)rmrThasattrrVrkr2rlrnrxr_rarcrdrh) rprsrVrTrt input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrarc embeddingss rLforwardzConvBertEmbeddings.forwards,  #..*K',,.s3K ^  ,,Q  ^ J"%88;PP ^^J/ \\*- rr)NNNN) __name__ __module__ __qualname____doc__rZrr2 LongTensor FloatTensorr __classcell__rqs@rLrOrOsQ (15593759 $E,,-$!!1!12$u//0 $   1 12 $    $rrrOc*eZdZUeed<eZdZdZdZ y)ConvBertPreTrainedModelr:convbertTclt|tjtjfrm|jj j d|jj|j%|jj jyyt|tjrz|jj j d|jj|j2|jj |jjyyt|tjrJ|jj j|jj jdyt|tr%|jj jyt|t r`|jj j d|jj|jj jyy)zInitialize the weightsmeanstdNg?) isinstancerLinearConv1dweightr8normal_r:initializer_rangebiaszero_r[rQrdfill_SeparableConv1DGroupedLinearLayer)rpmodules rL _init_weightsz%ConvBertPreTrainedModel._init_weightssw fryy"))4 5 MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S )  0 KK   " " $  2 3 MM   & &CT[[5R5R & S KK   " " $4rrN) rrrr__annotations__rMload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrrrrLrrs 1O"&*#%rrrcZeZdZdZfdZdej dej fdZxZS)rzSThis class implements separable convolution, i.e. a depthwise and a pointwise layerc t|tj|||||dzd|_tj||dd|_tj tj|d|_ |jjjjd|j|j jjjd|jy)Nr!F) kernel_sizegroupspaddingrr)rrrr)rYrZrr depthwise pointwise Parameterr2rlrrr8rr)rpr: input_filtersoutput_filtersrkwargsrqs rLrZzSeparableConv1D.__init__ s   # 1$  =.aV[\LL^Q!?@  ""**9Q9Q*R ""**9Q9Q*Rrr hidden_statesruch|j|}|j|}||jz }|SN)rrr)rprxs rLrzSeparableConv1D.forwards0 NN= ) NN1  TYYrr rrrrrZr2Tensorrrrs@rLrrs']S U\\ellrrrceZdZfdZ d dej deejdeejdeej deede ej eej ff dZ xZ S) ConvBertSelfAttentioncjt||j|jzdk7r2t |ds&t d|jd|jd|j|j z}|dkr|j|_d|_n||_|j |_|j|_|j|jzdk7r t d|j|jzdz|_|j|jz|_ tj|j|j|_ tj|j|j|_ tj|j|j|_t||j|j|j|_tj|j|j|jz|_tj|j|j|_tj&|jdgt)|jdz dz dg |_tj,|j.|_y) Nrr]zThe hidden size (z6) is not a multiple of the number of attention heads ()rz6hidden_size should be divisible by num_attention_headsr!)rr)rYrZ hidden_sizenum_attention_headsry ValueError head_ratioconv_kernel_sizeattention_head_size all_head_sizerrquerykeyrKrkey_conv_attn_layerconv_kernel_layerconv_out_layerUnfoldintunfoldrfattention_probs_dropout_probrh)rpr:new_num_attention_headsrqs rLrZzConvBertSelfAttention.__init__!s>     : : :a ?PVXhHi#F$6$6#78 445Q8  #)"<"<@Q@Q"Q "Q &$88DO'(D $'>D $$//DO & 7 7    8 8 8A =UV V$*$6$6$:R:R$RWX#X !558P8PPYYv1143E3EF 99V//1C1CDYYv1143E3EF #2 F&&(:(:D V u||Xell33 4VrrrcneZdZfdZdej dej dej fdZxZS)ConvBertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNrR) rYrZrrrrrdrerfrgrhros rLrZzConvBertSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rrr input_tensorrucr|j|}|j|}|j||z}|Srrrhrdrprrs rLrzConvBertSelfOutput.forward7 =1  ]3 }|'CD rrrrrrZr2rrrrs@rLrrs1> U\\RWR^R^rrrceZdZfdZdZ d dej deejdeejdeej dee de ej eejff d Z xZ S) ConvBertAttentionct|t||_t ||_t |_yr)rYrZrrproutputset pruned_headsros rLrZzConvBertAttention.__init__s0 )&1 (0 Errc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrrprrrrrrrKrrrunion)rpheadsindexs rL prune_headszConvBertAttention.prune_headss u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rrrrrrrrucl|j|||||}|j|d|}|f|ddz}|S)Nrr)rpr) rprrrrr self_outputsattention_outputrs rLrzConvBertAttention.forwardsQyy    !    ;;|A F#% QR(88rrr) rrrrZrr2rrrrrrrrs@rLrrs" ;*7;158<,1 ||!!2!23E--.  ( 5  $D>  u||Xe&7&788 9rrrcVeZdZfdZdej dej fdZxZS)rct|||_||_||_|j|jz|_|j|jz|_tjtj|j|j |j |_ tjtj||_ yr) rYrZ input_size output_sizer. group_in_dim group_out_dimrrr2emptyrr)rpr r r.rqs rLrZzGroupedLinearLayer.__init__s $&$ OOt>!--@ll5;;t@Q@QSWSeSe#fg LL[!9: rrrruct|jd}tj|d|j|j g}|j ddd}tj||j}|j ddd}tj||d|jg}||jz}|S)Nrr"rr!) listrmr2rr.r r6rrr r)rprrrs rLrzGroupedLinearLayer.forwards-,,./2 MM-"doot?P?P)Q R IIaA  LLDKK ( IIaA  MM!j"d.>.>? @  Mrrrrs@rLrrs#;U\\ellrrrcVeZdZfdZdej dej fdZxZS)ConvBertIntermediatect||jdk(r0tj|j |j |_n1t|j |j |j|_t|jtrt|j|_ y|j|_ y)Nrr r r.)rYrZr.rrrintermediate_sizerrr hidden_actstrr intermediate_act_fnros rLrZzConvBertIntermediate.__init__s     !6#5#5v7O7OPDJ+!--6;S;S`f`q`qDJ f'' -'-f.?.?'@D $'-'8'8D $rrrrucJ|j|}|j|}|Sr)rrrprs rLrzConvBertIntermediate.forwards& =1 00? rrrrs@rLrrs# 9U\\ellrrrcneZdZfdZdej dej dej fdZxZS)ConvBertOutputct||jdk(r0tj|j |j |_n1t|j |j |j|_tj|j |j|_ tj|j|_ y)NrrrR)rYrZr.rrrrrrrdrerfrgrhros rLrZzConvBertOutput.__init__s     !6#;#;V=O=OPDJ+!33ASAS`f`q`qDJf&8&8f>S>STzz&"<"<= rrrrrucr|j|}|j|}|j||z}|Srrrs rLrzConvBertOutput.forwardrrrrrs@rLrrs1 >U\\RWR^R^rrrceZdZfdZ d dej deejdeejdeej deej deede ej eejffd Z d Z xZ S) ConvBertLayercbt||j|_d|_t ||_|j |_|j|_|jr*|j st|dt ||_ t||_ t||_ y)Nrz> should be used as a decoder model if cross attention is added)rYrZchunk_size_feed_forward seq_len_dimr attention is_decoderadd_cross_attention TypeErrorcrossattentionr intermediaterrros rLrZzConvBertLayer.__init__s '-'E'E$*62 ++#)#=#=  # #??4&(f ghh"3F";D 08$V, rrrrrrencoder_attention_maskrruc>|j||||}|d}|dd} |jr?|=t|dstd|d|j |||||} | d}| | ddz} t |j |j|j|} | f| z} | S)N)rrrr)z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) r%r&ryAttributeErrorr)rfeed_forward_chunkr#r$) rprrrrr+rself_attention_outputsrrcross_attention_outputs layer_outputs rLrzConvBertLayer.forward(s"&   / "0"  2!4(, ??4@4!12$=dVDDD'+&9&9 &%! ' # 7q9  7 ;;G0  # #T%A%A4CSCSUe  /G+rrcL|j|}|j||}|Sr)r*r)rprintermediate_outputr1s rLr.z ConvBertLayer.feed_forward_chunkPs,"//0@A{{#68HI rr)NNNNF) rrrrZr2rrrrrrr.rrs@rLr!r!s -"7;158<9=,1&||&!!2!23&E--. & ( 5 & !) 6 &$D>& u||Xe&7&788 9&Prrr!ceZdZfdZ d dej deejdeejdeej deej deedeed eed e e e ffd Z xZ S) ConvBertEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rYrZr:r ModuleListr/r0r!layergradient_checkpointing)rpr:rrqs rLrZzConvBertEncoder.__init__WsN  ]]5IaIaCb#caM&$9#cd &+#$dsA#rrrrr+routput_hidden_states return_dictruc |rdnd} |rdnd} |r|jjrdnd} t|jD]T\} } |r| |fz} ||| nd}| ||||||}|d}|s,| |dfz} |jjsL| |dfz} V|r| |fz} |st d|| | | fDSt || | | S)Nrrrr!c3$K|]}|| ywrr).0vs rL z*ConvBertEncoder.forward..s=s)last_hidden_stater attentionscross_attentions)r:r' enumerater8rr)rprrrrr+rr:r;all_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_mask layer_outputss rLrzConvBertEncoder.forward]s$#7BD$5b4%64;;;Z;Zr`d(4 VOA|#$58H$H!.7.CilO(%&! M*!,M &9]1=M0 'tn0 d^0  u88 90 rrr5cVeZdZfdZdej dej fdZxZS)ConvBertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) rYrZrrrrrrrr transform_act_fnrdreros rLrZz(ConvBertPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrrrrucl|j|}|j|}|j|}|Sr)rrOrdrs rLrz'ConvBertPredictionHeadTransform.forwards4 =1 --m< }5 rrrrs@rLrMrMs$UU\\ellrrrMceZdZdZdeffd Z ddejdeejdejfdZ xZ S) ConvBertSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`ConvBertConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. r:cft|t|dd|_|jdk(rtt j |_t|drq|jret|dr(|jr|jdkDr |j}n |j}t j|j||_t|dd}|r t|nt j |_t j |_t|dr3|j"dkDr$t j$|j"|_t j |_t|d r5|j(dkDr%t j$|j(|_yyy) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrsummary_activationsummary_first_dropoutsummary_last_dropout)rYrZgetattrrTNotImplementedErrorrIdentitysummaryryrWrX num_labelsrrr activation first_dropoutrZrf last_dropoutr[)rpr: num_classesactivation_stringrqs rLrZz ConvBertSequenceSummary.__init__sU #FNFC    && %{{} 6- .63J3Jv78V=Z=Z_e_p_pst_t$// $00 99V%7%7EDL#F,@$GIZN3D$E`b`k`k`m[[] 62 38T8TWX8X!#F,H,H!ID KKM 61 2v7R7RUV7V " 6+F+F GD 8W 2rrr cls_indexruc|jdk(r |dddf}n|jdk(r |dddf}n|jdk(r|jd}n|jd k(r|Atj|d ddddf|jd dz tj }nX|j dj d}|jd |jdz z|jdfz}|jd |jd }n|jdk(rt|j}|j|}|j|}|j!|}|S)ak Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. rUNr"firstrrrrrf.rrW)r"rV)rTrr2 full_likerArnr7rkrrmgathersqueezer]rbr_rarc)rprrfrs rLrzConvBertSequenceSummary.forwardsn    &"1b5)F   ' )"1a4(F   & ("''A'.F   + - !OO!#rr1*-!''+a/** &//3==bA %,,Uimmo6I-JmN`N`acNdMf-fg "))"i8@@DF   & (% %##F+f%(""6* rrr) rrrrrrZr2rrrrrrs@rLrRrRsQ2H~H<Y])"..);CEDTDT;U)   )rrrRc6eZdZfdZdZdZdZe ddee jdee jdee jdee jd ee jd ee jd ee d ee d ee de eeffdZxZS) ConvBertModelc"t||t||_|j|j k7r/t j|j|j |_t||_ ||_ |jyr) rYrZrOr~r]rrrembeddings_projectr5encoderr: post_initros rLrZzConvBertModel.__init__sl  ,V4  F$6$6 6&(ii0E0EvGYGY&ZD #&v.   rrc.|jjSrr~r_rps rLget_input_embeddingsz"ConvBertModel.get_input_embeddingss...rrc&||j_yrrs)rprKs rLset_input_embeddingsz"ConvBertModel.set_input_embeddingss*/'rrc|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrpr8r%r)rpheads_to_pruner8rs rL _prune_headszConvBertModel._prune_headssE +002 CLE5 LL  u % / / ; ;E B CrrrsrrVrTrrtrr:r;ruc ||n|jj}||n|jj}| | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \} } | |jn |j} |tj| | }|pt|jdr4|jjddd| f}|j| | }|}n&tj| tj| }|j!|| }|j#||jj$}|j||||}t|dr|j'|}|j)|||||| }|S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer"z5You have to specify either input_ids or inputs_embeds)rxrVrw)rsrTrVrtro)rrrr:r;)r:rr:use_return_dictr%warn_if_padding_and_no_attention_maskrmrxr2onesryr~rVrkrlrnget_extended_attention_mask get_head_maskr0rorp)rprsrrVrTrrtrr:r;rzrr{rxr|r}extended_attention_maskrs rLrzConvBertModel.forward!s2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T  !"ZZ FCN  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z"&"B"B>S^"_&&y$++2O2OP l>iv(  4- . 33MBM 2/!5# % rr) NNNNNNNNN)rrrrZrurwr{rrr2rrrrrrrrrs@rLrmrms /0C156:59371559,0/3&*<E,,-<!!2!23<!!1!12 < u//0 < E--. <  1 12<$D><'tn<d^< u88 9<<rrrmcZeZdZdZfdZdej dej fdZxZS)ConvBertGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.ct|td|_t j |j |j|_t j|j|j |_ y)NgelurR) rYrZr rarrdr]rerrrros rLrZz%ConvBertGeneratorPredictions.__init__dsV (0f&;&;AVAVWYYv1163H3HI rrgenerator_hidden_statesrucl|j|}|j|}|j|}|Sr)rrard)rprrs rLrz$ConvBertGeneratorPredictions.forwardks3 #:;  6 }5 rr) rrrrrZr2rrrrs@rLrras+KJu/@/@UEVEVrrrcVeZdZdgZfdZdZdZe ddee jdee jdee jdee jd ee jd ee jd ee jd ee d ee dee de eeffdZxZS)ConvBertForMaskedLMzgenerator.lm_head.weightct||t||_t ||_t j|j|j|_ |jyr) rYrZrmrrgenerator_predictionsrrr]r\generator_lm_headrqros rLrZzConvBertForMaskedLM.__init__wsR  %f- %A&%I"!#6+@+@&BSBS!T rrc|jSrrrts rLget_output_embeddingsz)ConvBertForMaskedLM.get_output_embeddingss%%%rrc||_yrr)rpr_s rLset_output_embeddingsz)ConvBertForMaskedLM.set_output_embeddingss !0rrrsrrVrTrrtlabelsrr:r;ruc | | n|jj} |j|||||||| | } | d} |j| } |j | } d}|Pt j }|| jd|jj|jd}| s| f| ddz}||f|zS|St|| | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr"rlosslogitsrrB) r:r}rrrrr rr\rrrB)rprsrrVrTrrtrrr:r;rgenerator_sequence_outputprediction_scoresrloss_fctrs rLrzConvBertForMaskedLM.forwards(&1%! 667PQ 223DE  **,H-222t{{7M7MNPVP[P[\^P_`D'),CAB,GGF)-)9TGf$ Ev E$1??.99   rr NNNNNNNNNN)rrr_tied_weights_keysrZrrrrr2rrrrrrrrrs@rLrrss45&1156:59371559-1,0/3&*4 E,,-4 !!2!234 !!1!12 4 u//0 4 E--. 4   1 124 ))*4 $D>4 'tn4 d^4  un$ %4 4 rrrcZeZdZdZfdZdej dej fdZxZS)ConvBertClassificationHeadz-Head for sentence-level classification tasks.cht|tj|j|j|_|j |j n |j}tj||_ tj|j|j|_ ||_ yr) rYrZrrrrclassifier_dropoutrgrfrhr`out_projr:rpr:rrqs rLrZz#ConvBertClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH  rrrruc |dddddf}|j|}|j|}t|jj|}|j|}|j |}|S)Nr)rhrr r:rr)rprrrs rLrz"ConvBertClassificationHead.forwardse !Q' " LLO JJqM 4;;)) *1 - LLO MM! rrrrs@rLrrs&7 U\\ rrrz ConvBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcDeZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd ee d ee d ee d e e e ffd Z xZS)!ConvBertForSequenceClassificationct|||j|_||_t ||_t ||_|jyr) rYrZr`r:rmrr classifierrqros rLrZz*ConvBertForSequenceClassification.__init__sH   ++ %f- 4V< rrrsrrVrTrrtrrr:r;ruc | | n|jj} |j|||||||| |  } | d} |j| } d}||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt}|j dk(r&|| j|j}n|| |}n|jjdk(r=t}|| jd|j |jd}n,|jjdk(rt}|| |}| s| f| ddz}||f|zS|St|| | j | j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrVrTrrtrr:r;rr regressionsingle_label_classificationmulti_label_classificationr"r)r:r}rr problem_typer`rXr2rnrr rkr rrrrrBrprsrrVrTrrtrrr:r;rsequence_outputrrrrs rLrz)ConvBertForSequenceClassification.forwards(&1%D 'tnD d^D  u.. /D D rrrcDeZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd ee d ee d ee d e e e ffd Z xZS)ConvBertForMultipleChoicect||t||_t ||_t j|jd|_ |jy)Nr) rYrZrmrrRsequence_summaryrrrrrqros rLrZz"ConvBertForMultipleChoice.__init__2sM  %f- 7 ?))F$6$6: rrrsrrVrTrrtrrr:r;ruc L| | n|jj} ||jdn|jd} |!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|!|jd|j dnd}|1|jd|j d|j dnd}|j |||||||| |  } | d} |j | }|j|}|jd| }d}|t}|||}| s|f| ddz}||f|zS|St||| j| jS)a\ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr"rrrr) r:r}rArrmrrrr rrrB)rprsrrVrTrrtrrr:r; num_choicesrr pooled_outputrreshaped_logitsrrrs rLrz!ConvBertForMultipleChoice.forward<sZ&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  -- ))%'/!5#  "!*--o> / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rrr)rrrrZrrr2rrrrrrrrrs@rLrr0s156:59371559-1,0/3&*Y E,,-Y !!2!23Y !!1!12 Y u//0 Y E--. Y   1 12Y ))*Y $D>Y 'tnY d^Y  u// 0Y Y rrrcDeZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd ee d ee d ee d e e e ffd Z xZS)ConvBertForTokenClassificationc`t|||j|_t||_|j |j n |j }tj||_ tj|j|j|_ |jyr)rYrZr`rmrrrgrrfrhrrrrqrs rLrZz'ConvBertForTokenClassification.__init__s   ++%f- )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rrrsrrVrTrrtrrr:r;ruc | | n|jj} |j|||||||| |  } | d} |j| } |j | } d}|2 'tn2 d^2  u++ ,2 2 rrrcdeZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e e ffdZ xZS)ConvBertForQuestionAnsweringct|||j|_t||_t j |j|j|_|jyr) rYrZr`rmrrrr qa_outputsrqros rLrZz%ConvBertForQuestionAnswering.__init__sS   ++%f- ))F$6$68I8IJ rrrsrrVrTrrtstart_positions end_positionsrr:r;ruc (| | n|jj} |j||||||| | |  } | d} |j| }|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| jS) Nrrrr"r) ignore_indexr!)r start_logits end_logitsrrB)r:r}rrsplitrkrrrmclampr rrrB)rprsrrVrTrrtrrrr:r;rrrrr total_loss ignored_indexr start_lossend_lossrs rLrz$ConvBertForQuestionAnswering.forwards&1% E,,-> !!2!23> !!1!12 > u//0 > E--. >   1 12> "%"2"23>   0 01> $D>> 'tn> d^>  u22 3> > rrr) rrrrrr!rmrrM)Brrr'operatorrtypingrrrr2torch.utils.checkpointrtorch.nnrr r activationsr r modeling_layersrmodeling_outputsrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_convbertr get_loggerrr%rMModulerOrrrrrrrrr!r5rMrRrmrrrrrrr__all__rrrrLrs; ,, AA19.ll3   H %yx99x%o%%8bii4~BII~B  * *Z,299(RYY&:.:z7 bii7 tbii$`bii`FX+XXv299$H 1H H V0 P (?P  P fe 7e e PB %<B B JJ #:J J Z rr