rh'*dZddlmZddlZddlmZddlmZGdd ejjjZ Gd d ejjjZ Gd d ejjjZy)a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalN) shape_list) IdeficsConfigc~eZdZdededededededdffd Zfd Zd ejdejfd Z xZ S) TFIdeficsPerceiverResamplerconfig embed_dimdepthn_headshead_dim n_latentsreturnNc t |d i|||||f\|_|_|_|_|j j|_t|jds|jdzn|jjdz|_ g|_ t|D]s}|jjt|j|j|j|jd|dt!|j|d|dgut"j$j&j)dd |_y ) ao Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). r zblocks.z.0namez.1h㈵> layer_normepsilonrN)super__init__r r rrperceiver_configqk_layer_norms_perceiverqk_layer_normshasattr vision_configintermediate_dimblocksrangeappendTFIdeficsPerceiverAttention TFIdeficsMLPtfkeraslayersLayerNormalizationr) selfr r r r rrkwargsi __class__s {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/idefics/perceiver_tf.pyrz$TFIdeficsPerceiverResampler.__init__1s+( "6"FOQXZbdmFmC dmT^$55NN6//= NNQ %%//!3   u A KK  / dmmTEXEXahijhkkm_n!!6!6wqcQS_U   ((//<>>4>>2[_fo'    k"r0contextctj|jd}tj|tj|dddg}|j D]\}}||||z}|||z}|j |S)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsraxisr)r' expand_dimsr3tiler4r"r)r+r:r3attnffs r/callz TFIdeficsPerceiverResampler.callcs..A6'''BHHW$5a$8!Q#?@  ,HD"7G,w6GkG+G ,w''r0) __name__ __module__ __qualname__rintrr8r'TensorrB __classcell__r.s@r/r r 0si)^#)^03)^ let long-form inputs be `context`, resampled embeddings be `latents`rcontext_layer_normrlatents_layer_norm q_layer_norm k_layer_normgFq_projuse_biasrk_projv_proj output_projNr)rrr r rrr'r(r)r*rLrMrNrOqk_scaleDenserPrSrTrU)r+r r rrr,r.s r/rz$TFIdeficsPerceiverAttention.__init__ps "6"6?(3 dm,"$((//"D"DTXl"D"m"$((//"D"DTXl"D"m    " B B4Vd B eD  " B B4Vd B eD  t+ hhoo++DLL4==,HSX_g+h hhoo++DLL4==,HSX_g+h hhoo++DLL4==,HSX_g+h 88??00UQ^0_r0r:r3c |j|}|j|}t|\}}}|j|}|j t j ||gd}|jt j ||gd}|||fD cgc]T} t jt j| || jd|j|jfgdVc} \}}}|jr"|j|}|j|}t j d||j"z|} | t j$| dd z } t j&j)| d} t j d | |} |j+t jt j| gd|d|j|jzfScc} w) a= Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`tf.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`tf.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `tf.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. r<r)rrr)permz... i d, ... j d -> ... i jT)r=keepdimsz... i j, ... j d -> ... i d)rLrMrrPrSr'concatrT transposereshaper4r rrrNrOeinsumrV reduce_maxnnsoftmaxrU)r+r:r3 batch_size seq_lengthr qkvxscoresstabilized_scoresr@ resampleds r/rBz TFIdeficsPerceiverAttention.calls))'2))'2,6w,?) J  KK  KK 7G"42> ? KK 7G"42> ? AY  LLA AGGAJ dmm'\]dp q 1a   !!!$A!!!$A8!dmm:KQO"R]]6T%RRuu}}.R}8II;T1E  JJr||ILAJPRTXT`T`cgcpcpTpCq r   sAG5) rCrDrErFboolrr'rGrBrHrIs@r/r%r%osY`#``s`TX`gk`*+ BII+  + bii+ r0r%cheZdZdeffd ZdeeejdejfdZ xZ S)r&r c t|d i||jj|_tj j jdd|_tj j j|dd|_ tj j jd|_ tj j j|jdd |_ y ) z:Simple MLP block with intermediate_size and embedding sizerlnrFfcrQactrc_projNr)rrr r r'r(r)r*rqrWrrReLUrsrt)r+intermediate_sizer r,r.s r/rzTFIdeficsMLP.__init__s "6"--77((//44T4M((//''(9EPT'U88??''U'3hhoo++DNNUQY+Z r0 hidden_statesrc|j|}|j|}|j|}|j|}|S)N)rqrrrsrt)r+rws r/rBzTFIdeficsMLP.calls@ .  . /  M2 r0) rCrDrErrrtupler'rGrBrHrIs@r/r&r&s6[-[(5+;"<r0r&)__doc__typingr tensorflowr'modeling_tf_utilsrconfiguration_ideficsrr(r)Layerr r%r&rr0r/rsj4 +0<("((//"7"7<(~A "((//"7"7A H288??((r0