rhTdZddlZddlmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZdd lmZmZmZmZmZdd lmZdd lmZmZmZdd lmZmZd dlm Z ejBe"Z#GddejHZ%GddejHZ&GddejHZ'GddejHZ(GddejHZ)GddejHZ*GddejHZ+GddeZ,Gdd ejHZ-Gd!d"ejHZ.eGd#d$eZ/eGd%d&e/Z0ed'(Gd)d*e/Z1eGd+d,e/Z2Gd-d.ejHZ3eGd/d0e/Z4gd1Z5y)2zPyTorch LiLT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) LiltConfigc:eZdZfdZ ddZdZdZxZS)LiltTextEmbeddingsc8t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)ddt+|dd|_|j |_tj|j|j |j.|_ y) N padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr#rselfconfig __class__s y/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/lilt/modeling_lilt.pyr&zLiltTextEmbeddings.__init__,s2 !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$"..#%<<  * *F,>,>DL\L\$  cD|I|6|j||jj|j}n|j |}||j }n|j dd}|:t j|t j|jj}||j|}|j|}||z}|jdk(r|j|}||z }|j|}|j|}||fS)Nr!dtypedevicer$)"create_position_ids_from_input_idsrtorC&create_position_ids_from_inputs_embedssizer6zeroslongr r+r/r#r-r0r4) r; input_idstoken_type_idsr inputs_embeds input_shaper/ embeddingsr-s r>forwardzLiltTextEmbeddings.forwardCs!  $#FFyRVRbRbcff$$   $JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN   00;M $ : :> J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- <''r?c|j|j}tj|dj ||z}|j |zS)a Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr6cumsumtype_asrI)r;rJrmaskincremental_indicess r>rDz5LiltTextEmbeddings.create_position_ids_from_input_idsgsP||K(,,.$||Da8@@F$N"'')K77r?c|jdd}|d}tj|jdz||jzdztj|j }|j dj|S)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr!rrAr)rGr6r7rrIrC unsqueezer8)r;rLrMsequence_lengthr s r>rFz9LiltTextEmbeddings.create_position_ids_from_inputs_embedsts$((*3B/ %a.||   q /D4D4D"Dq"HPUPZPZcpcwcw %%a(// <rr+s& 2 "(H 8 =r?rc&eZdZfdZddZxZS)LiltLayoutEmbeddingsct|tj|j|j dz|_tj|j|j dz|_tj|j|j dz|_tj|j|j dz|_ |j|_ tj|j|j |jz|j|_tj|j |j |jz|_tj"|j |jz|j$|_tj&|j(|_y)Nr) in_features out_featuresr)r%r&rr'max_2d_position_embeddingsr)x_position_embeddingsy_position_embeddingsh_position_embeddingsw_position_embeddingsr*rr,channel_shrink_ratiobox_position_embeddingsLinearbox_linear_embeddingsr0r1r2r3r4r:s r>r&zLiltLayoutEmbeddings.__init__s^ &(\\&2S2SU[UgUgklUl%m"%'\\&2S2SU[UgUgklUl%m"%'\\&2S2SU[UgUgklUl%m"%'\\&2S2SU[UgUgklUl%m"!..')||  * *   &"="= =((( $ &(YY**9K9KvOjOj9j& "f&8&8FrOzLiltLayoutEmbeddings.forwardsw c'+'A'A$q!Qw-'P $(,(B(B41a=(Q %(,(B(B41a=(Q %(,(B(B41a=(Q %!% : :41a=4PQSTVWPW=;X Y $ : :41a=4PQSTVWPW=;X Y&+ii()))%%   ' #'+&@&@A\&]#"&">">|"L&AD[&[#&*nn5P&Q#&*ll3N&O#**3 cZ[ab b csA,D&& E/ D;;ENN)r\r]r^r&rOr_r`s@r>rbrbs >*+r?rbc6eZdZdfd ZddZ ddZxZS)LiltSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|jz|j|jz|_tj|j|jz|j|jz|_tj|j|jz|j|jz|_tj$|j&|_|xs t+|dd|_|j,dk(s|j,d k(rF|j.|_tj0d |j.zd z |j|_|j|_||_y) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r#r$ relative_keyrelative_key_queryrqr)r%r&r)num_attention_headshasattr ValueErrorrTattention_head_size all_head_sizerrnquerykeyvaluerl layout_query layout_key layout_valuer2attention_probs_dropout_probr4r9r#r,r'distance_embedding layer_idx)r;r<r#rr=s r>r&zLiltSelfAttention.__init__sU     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #$*$?$?!"r?c|jdd|j|j|zfz}|j|}|j ddddS)Nr!rrqrr )rGrrviewpermute)r;xr new_x_shapes r>transpose_for_scoresz&LiltSelfAttention.transpose_for_scoressTffhsmt'?'?AYAY]^A^&__ AFFK yyAq!$$r?c|j|j||j}|j|j||j}|j|j ||j}|j |} |j|j |} |j|j|} |j| } tj| | jdd} tj||jdd}|jdk(s|jdk(rF|jd}tj|tj|jj!dd}tj|tj|jj!dd}||z }|j#||j$zdz }|j'| j(}|jdk(rtj*d | |}| |z} nE|jdk(r6tj*d | |}tj*d | |}| |z|z} | t-j.|j0z }|t-j.|j0|jzz }||z} ||z}|||z}t3j4d |}|j7|}|||z}tj||}|j9d d ddj;}|jdd|j<|jzfz}|j |}|| |z} t3j4d | }|j7|}|||z}tj|| }|j9d d ddj;}|jdd|j<fz}|j |}|r||f|f}|S||ff}|S)N)rr!rrrrA)rBzbhld,lrd->bhlrzbhrd,lrd->bhlrrQrrqr )rrrlrrrrrr6matmul transposer#rGr7rIrCrrr,rErBeinsummathsqrtrrSoftmaxr4r contiguousr)r; hidden_states layout_inputsattention_mask head_maskoutput_attentionslayout_value_layerlayout_key_layerlayout_query_layermixed_query_layer key_layer value_layer query_layerattention_scoreslayout_attention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keytmp_attention_scorestmp_layout_attention_scoreslayout_attention_probslayout_context_layernew_context_layer_shapeattention_probs context_layeroutputss r>rOzLiltSelfAttention.forwardsd"66t7H7H7W[_[t[t6u44T__]5SW[WpWp4q!66t7H7H7W[_[t[t6u JJ}5--dhh}.EF // =0IJ //0AB  << Y5H5HR5PQ"',,/ACSC]C]^`bdCe"f  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||,,7  - 9O _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD !1 2O D  !"679  r?r{)rNNF)r\r]r^r&rrOr_r`s@r>r}r}s##J% \r?r}cneZdZfdZdej dej dej fdZxZS)LiltSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr) r%r&rrnr)denser0r1r2r3r4r:s r>r&zLiltSelfOutput.__init__Hs` YYv1163E3EF f&8&8f>S>STzz&"<"<= r?r input_tensorreturncr|j|}|j|}|j||z}|SNrr4r0r;rrs r>rOzLiltSelfOutput.forwardN7 =1  ]3 }|'CD r?r\r]r^r&r6TensorrOr_r`s@r>rrG1> U\\RWR^R^r?rceZdZd fd ZdZ d dej dej deejdeejdee de ej f d Z xZ S) LiltAttentionct|t||||_t ||_t |_|j}|j|jz|_t ||_ ||_y)N)r#r) r%r&r}r;routputset pruned_headsr)rl layout_output)r;r<r#rori_hidden_sizer=s r>r&zLiltAttention.__init__Vso %fF]irs $V, E ,,#//63N3NN+F3,r?c>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)NrrrQ)lenrr;rrrrrrrrrrunion)r;headsindexs r> prune_headszLiltAttention.prune_headsbs u:? 7 49900$))2O2OQUQbQb u -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r?rrrrrrc|j|||||}|j|dd|}|j|dd|}||ff|ddz} | S)Nrr)r;rr) r;rrrrr self_outputsattention_outputlayout_attention_outputrs r>rOzLiltAttention.forwardtszyy        ;;|Aq'9=I"&"4"4\!_Q5G"W$&=>@rrUs -;,7;15,1 ||||!!2!23  E--.  $D>  u|| r?rcVeZdZfdZdej dej fdZxZS)LiltIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r%r&rrnr)intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr:s r>r&zLiltIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r?rrcJ|j|}|j|}|Sr)rr)r;rs r>rOzLiltIntermediate.forwards& =1 00? r?rr`s@r>rrs#9U\\ellr?rcneZdZfdZdej dej dej fdZxZS) LiltOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r%r&rrnrr)rr0r1r2r3r4r:s r>r&zLiltOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= r?rrrcr|j|}|j|}|j||z}|Srrrs r>rOzLiltOutput.forwardrr?rr`s@r>rrrr?rceZdZd fd Z d dej dej deejdeejdeede ej f dZ d Z d Z xZ S) LiltLayerct||j|_d|_t |||_t ||_t||_ |j}|j}|j|jz|_ |j|jz|_ t ||_ t||_||_ ||_ y)Nrr)r%r&chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr)rrllayout_intermediater)r;r<rrori_intermediate_sizer=s r>r&zLiltLayer.__init__s '-'E'E$&vC,V4 (  ,, & 8 8#//63N3NN#)#;#;v?Z?Z#Z #3F#; '/,#8 r?rrrrrrc|j|||||}|dd}|dd}|dd} t|j|j|j|} t|j |j|j|} | | ff| z} | S)N)rrr)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r;rrrrrself_attention_outputsrrr layer_outputlayout_layer_outputs r>rOzLiltLayer.forwards"&    / "0" 2!4Q7"8";A">(,0  # #T%A%A4CSCSUe 8  * *D,H,H$JZJZ\s !"5687Br?cL|j|}|j||}|Sr)rrr;rintermediate_outputrs r>rzLiltLayer.feed_forward_chunks,"//0@A{{#68HI r?cL|j|}|j||}|Sr)rrrs r>rz#LiltLayer.layout_feed_forward_chunks."667GH))*=?OP r?rr)r\r]r^r&r6rrrrrrOrrr_r`s@r>rrs9*7;15,1 ||||!!2!23  E--.  $D>  u|| > r?rceZdZd fd Z d dej dej deejdeejdeedeedeed e e ej e ffd Z xZ S) LiltEncoderc t|||_tjt |j Dcgc]}t||c}|_d|_ ycc}w)NrF) r%r&r<r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r;r<rir=s r>r&zLiltEncoder.__init__sQ  ]]ERXRjRjLk#lqIf$B#lm &+#$msA%rrrrroutput_hidden_states return_dictrc |rdnd}|rdnd} t|jD]>\} } |r||fz}||| nd} | |||| |} | dd}| dd}|s6| | dfz} @|r||fz}|std||| fDSt||| S)Nrrc3$K|]}|| ywrr).0vs r> z&LiltEncoder.forward..s = s)last_hidden_stater attentions) enumerater rr )r;rrrrrrrall_hidden_statesall_self_attentionsr  layer_modulelayer_head_mask layer_outputss r>rOzLiltEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO(! M*!,Q/M)!,Q/M &9]1=Mrrs,7;15,1/4&*2 ||2 ||2 !!2!23 2 E--. 2 $D> 2 'tn2 d^2  uU\\"O3 42 r?rcVeZdZfdZdej dej fdZxZS) LiltPoolerct|tj|j|j|_tj |_yr)r%r&rrnr)rTanh activationr:s r>r&zLiltPooler.__init__#s9 YYv1163E3EF '')r?rrc\|dddf}|j|}|j|}|SNr)rr")r;rfirst_token_tensor pooled_outputs r>rOzLiltPooler.forward(s6+1a40 #56  6 r?rr`s@r>rr"s#$ U\\ellr?rc*eZdZUeed<dZdZgZdZy)LiltPreTrainedModelr<liltTct|tjrm|jjj d|j j|j%|jjjyyt|tjrz|jjj d|j j|j2|jj|jjyyt|tjrJ|jjj|jjjdyy)zInitialize the weightsg)meanstdNg?)rrrnweightdatanormal_r<initializer_rangebiaszero_r'rr0fill_)r;modules r> _init_weightsz!LiltPreTrainedModel._init_weights8s fbii ( MM   & &CT[[5R5R & S{{&   &&('  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - KK   " " $ MM   $ $S ).r?N) r\r]r^r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr5rr?r>r(r(1s &*#*r?r(creZdZdfd ZdZdZdZe ddee jdee jdee jdee jd ee jd ee jd ee jd ee d ee dee de e e jeffdZxZS) LiltModelct||||_t||_t ||_t||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r%r&r<rrNrblayout_embeddingsrencoderrpooler post_init)r;r<add_pooling_layerr=s r>r&zLiltModel.__init__KsZ   ,V4!5f!="6* ,=j(4  r?c.|jjSrrNr+)r;s r>get_input_embeddingszLiltModel.get_input_embeddings\s...r?c&||j_yrrC)r;rs r>set_input_embeddingszLiltModel.set_input_embeddings_s*/'r?c|jD]7\}}|jj|jj |9y)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr>r rr)r;heads_to_pruner rs r> _prune_headszLiltModel._prune_headsbsE +002 CLE5 LL  u % / / ; ;E B Cr?rJrtrrKr rrLrrrrc ||n|jj}| | n|jj} | | n|jj} | | t d|#|j |||j } n!||j dd} n t d| \} } | |jn |j}|)tj| dztj|}|tj| | f|}|pt|jdr4|jjddd| f}|j| | }|}n&tj| tj|}|j!|| }|j#||jj$}|j|||| \}}|j'|| }|j)|||||| | }|d }|j*|j+|nd}| s ||f|d dzSt-|||j.|j0S)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer!z5You have to specify either input_ids or inputs_embeds)rA)rCrK)rJr rKrL)rtr )rrrrrrr)r pooler_outputrr)r<rruse_return_dictr%warn_if_padding_and_no_attention_maskrGrCr6rHrIonesrrNrKr8get_extended_attention_mask get_head_maskr r=r>r?r rr)r;rJrtrrKr rrLrrrrM batch_sizerrCbuffered_token_type_ids buffered_token_type_ids_expandedextended_attention_maskembedding_outputlayout_embedding_outputencoder_outputssequence_outputr&s r>rOzLiltModel.forwardjssP2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T <;;{T1FSD  !"ZZ*j)A6RN  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z150P0PQ_al0m&&y$++2O2OP )-%)' *9* &,#'"8"8dQ]"8"^,,  #2/!5#' *!,8< 8O O4UY #]3oab6II I)-')77&11   r?)T) NNNNNNNNNN)r\r]r^r&rDrFrJrrr6rrrrr rOr_r`s@r>r;r;Is$"/0C-1'+1515/3,004,0/3&*p ELL)p u||$p !. p !. p u||, p ELL)p   -p $D>p 'tnp d^p  uU\\"$>> ?p p r?r;z LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introc~eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e eje ffdZxZS)LiltForSequenceClassificationct|||j|_||_t |d|_t ||_|jyNF)rA) r%r& num_labelsr<r;r)LiltClassificationHead classifierr@r:s r>r&z&LiltForSequenceClassification.__init__sJ   ++ f> 08 r?rJrtrrKr rrLlabelsrrrrc V| | n|jj} |j|||||||| | |  } | d} |j| }d}||j |j }|jj |jdk(rd|j_nl|jdkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|jdk(r&||j|j}n|||}n|jj dk(r=t}||jd|j|jd}n,|jj dk(rt!}|||}| s|f| d dz}||f|zS|St#||| j$| j& S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```N rtrrKr rrLrrrrr regressionsingle_label_classificationmulti_label_classificationr!rqlosslogitsrr)r<rNr)rbrErC problem_typer`rBr6rIrTrsqueezerrrrrrr;rJrtrrKr rrLrcrrrrrZrkrjloss_fctrs r>rOz%LiltForSequenceClassification.forwards\&1%r]r]s9 15'+6:59371559-1,0/3&*` E,,-` u||$` !!2!23 ` !!1!12 ` u//0 ` E--.`   1 12` ))*` $D>` 'tn` d^`  uU\\"$<< =` ` r?r]c~eZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e eje ffdZxZS)LiltForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyr_)r%r&r`r;r)classifier_dropoutr3rr2r4rnr)rbr@r;r<rur=s r>r&z#LiltForTokenClassification.__init__Xs   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r?rJrtrrKr rrLrcrrrrc | | n|jj} |j|||||||| | |  } | d} |j| } |j | }d}|W|j |j }t}||jd|j|jd}| s|f| ddz}||f|zS|St||| j| jS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nrerr!rqri) r<rNr)r4rbrErCrrr`rrrrns r>rOz"LiltForTokenClassification.forwardfs V&1%rsrsUs; 15+/6:59371559-1,0/3&*N E,,-N u''(N !!2!23 N !!1!12 N u//0 N E--.N   1 12N ))*N $D>N 'tnN d^N  uU\\"$99 :N N r?rsc(eZdZdZfdZdZxZS)raz-Head for sentence-level classification tasks.cZt|tj|j|j|_|j |j n |j}tj||_ tj|j|j|_ yr) r%r&rrnr)rrur3r2r4r`out_projrvs r>r&zLiltClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH r?c |dddddf}|j|}|j|}tj|}|j|}|j |}|Sr$)r4rr6tanhrz)r;featureskwargsrs r>rOzLiltClassificationHead.forwardsY Q1W  LLO JJqM JJqM LLO MM! r?)r\r]r^__doc__r&rOr_r`s@r>raras7Ir?raceZdZfdZe ddeejdeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e eje ffdZxZS)LiltForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyr_) r%r&r`r;r)rrnr) qa_outputsr@r:s r>r&z!LiltForQuestionAnswering.__init__sU   ++f> ))F$6$68I8IJ r?rJrtrrKr rrLstart_positions end_positionsrrrrc *| | n|jj} |j|||||||| | |  } | d}|j|}|j dd\}}|j dj }|j dj }d}|| t|jdkDr|j d}t| jdkDr| j d} |jd}|jd|}| jd|} t|}|||}||| }||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```Nrerrr!rQ) ignore_indexrq)rj start_logits end_logitsrr)r<rNr)rsplitrmrrrGclamprrrr)r;rJrtrrKr rrLrrrrrrrZrkrr total_loss ignored_indexro start_lossend_lossrs r>rOz LiltForQuestionAnswering.forwards^&1%rrsT15+/6:593715596:48,0/3&*_ E,,-_ u''(_ !!2!23 _ !!1!12 _ u//0 _ E--._   1 12_ "%"2"23_   0 01_ $D>_ 'tn_ d^_  uU\\"$@@ A_ _ r?r)rr]rsr;r()6rrtypingrrr6torch.utils.checkpointrtorch.nnrrr activationsr modeling_layersr modeling_outputsr r rrrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_liltr get_loggerr\loggerModulerrbr}rrrrrrrr(r;r]rsrar__all__rr?r>rs " AA!9.ll,*   H %V=V=r5+2995+pG GV RYY 1BII1j ryy   9*9x: ")): |  */**.Q #Q Q h n $7n  n b_ !4_ _ FRYY,l 2l l ^ r?