rh$ d Z ddlZddlZddlmZ ddlmZmZ ddlm Z ddl
m Z m Z m
Z
ddlmZmZ dd lmZ G d
d e d
Z ej( e Z G d de ZdgZy)z
Processor class for Donut.
N)contextmanager)OptionalUnion )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)loggingc e Zd Zi Zy)DonutProcessorKwargsN)__name__
__module__ __qualname__ _defaults }/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/donut/processing_donut.pyr r s Ir r F)totalc
e Zd ZdZddgZdZdZd fd Z ddede e
e e e e
ef d ee fd
Zd Zd Zed
ZddZed Zed Z xZS )DonutProcessora
Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single
processor.
[`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and
[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and
[`~DonutProcessor.decode`] for more information.
Args:
image_processor ([`DonutImageProcessor`], *optional*):
An instance of [`DonutImageProcessor`]. The image processor is a required input.
tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*):
An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input.
image_processor tokenizerAutoImageProcessor
AutoTokenizerc d }d|v r+t j dt |j d }||n|}| t d | t d t
| || | j | _ d| _ y )Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.z)You need to specify an `image_processor`.z"You need to specify a `tokenizer`.F)
warningswarn
FutureWarningpop
ValueErrorsuper__init__r current_processor_in_target_context_manager)selfr r kwargsr __class__s r r& zDonutProcessor.__init__9 s & ( MM
!'
+> ? -<-H/N_ " HI I AB B
)4!%!5!5 */'r imagestextr* c | | j r | j ||fi |S |
| t d | j t fd| j
j i|}| | j |fi |d }|-||d j dd | j
|fi |d }|S |S d d< |d |d< |S ) a
When used in normal mode, this method forwards all its arguments to AutoImageProcessor's
[`~AutoImageProcessor.__call__`] and returns its output. If used in the context
[`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's
[`~DonutTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information.
zBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs
images_kwargs text_kwargsadd_special_tokensF input_idslabels) r( r' r$
_merge_kwargsr r init_kwargsr
setdefault) r) r, r- audiovideosr*
output_kwargsinputs encodingss r __call__zDonutProcessor.__call__M s * *)4))&$A&A A >dl ab b***
"&.."<"<
)T))&SM/4RSF ! m,778LeT&tL}]/KLI <M
^ ( 5F8 "+K"8F; Mr c : | j j |i |S )z
This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.batch_decode`]. Please refer
to the docstring of this method for more information.
)r batch_decoder) argsr* s r r? zDonutProcessor.batch_decodew s
+t~~**D;F;;r c : | j j |i |S )z
This method forwards all its arguments to DonutTokenizer's [`~PreTrainedTokenizer.decode`]. Please refer to the
docstring of this method for more information.
)r decoder@ s r rC zDonutProcessor.decode~ s
%t~~$$d5f55r c # K t j d d| _ | j | _ d | j
| _ d| _ yw)z
Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR.
z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r! r( r r' r r) s r as_target_processorz"DonutProcessor.as_target_processor sH
9
+/'!%
!%!5!5 */'s A A
c || j j }i }|r[t j d|t j }|n2||j d }d|vrn|d|j
d dz }|t d t d }t j | }t j d| d|t j } | |j |d }n| j } t j | }
t j | } t j |
d| |t j t j z } | | j d j } d| v r3d| v r/| j | d| }
|
rt |
dk( r|
d
}
|
||< ntg ||< | j d D ]? }|j }||v r|d
d k( r
|d
d dk( r|dd
}|| j | A t || dk( r || d
||< ||j! | t | z d j }|dd d k( r|g| j |dd d| z S |r[|r|r|gS |S |rg S d|iS )zS
Convert a (generated) token sequence into an ordered JSON format.
Nz z
text_sequence)r get_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r) tokensrK rL outputpotential_start start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafs r r] zDonutProcessor.token2json s ..88:K ii
FO & !6!6!8!:;K+%%&B (9(9#(>(BCK c%jCH95C ))C.K T+ a"8&"--PI R8 %OO - &(ii &<#$&IIi$8! ))*+51B0CDfbmm^`^g^gNg & %mmA . 4 4 6G (W-? $Ze f "5zQ(-a*/F3K&(s $+MM)$< 5D#'::r sc
% " % H H C +5
H %x$^ x$v
r