rhxUddlZddlZddlZddlZddlZddlZddlmZm Z ddl m Z ddl m Z ddlmZmZmZmZddlZddlmZddlmZdd lmZdd lmZmZmZmZerdd lm Z!ejDe#Z$Gd d eZ%Gdde%Z&Gdde%Z'Gdde'Z(Gdde(Z)GddZ*Gdde*Z+Gdde*Z,Gdde,Z-Gdde,Z.d ed!e/ej`ej`ffd"ed!e/ej`ej`fffd#Z1Gd$d%Z2Gd&d'Z3Gd(d)e3Z4ed*rd+e4fd,Z5d-ej4jljnfd.Z8ej4jljse4d/e8e4jtd0e4jFd12ejvjljye4d3Gd4d5e4Z=Gd6d7e3Z>Gd8d9e>Z?Gd:d;e3Z@Gd<d=e3ZAGd>d?eAZBGd@dAeBZCGdBdCe4ZDGdDdEeDZEGdFdGeDZFGdHdIe3ZGdJeeHddKeId"e/eIeIffdLZJ dddMeedNeeKdOeeKdPeejeMdfdQeejdReeIeKejfdSeeKdTeeKd"eIfdUZOe'e(e)dVZPeIeMeHd feQdW<e+e,e.dXZReIeMeHdfeQdY<GdZd[e3ZSe Gd\d]ZTe Gd^d_eTZUe Gd`daeTZVdbeMd"efdcZWy)eN)ABCabstractmethod)Iterable) dataclass)AnyCallableOptionalUnion)version)"is_torch_greater_or_equal_than_2_6)PretrainedConfig)is_hqq_availableis_optimum_quanto_availableis_torch_greater_or_equallogging) Quantizerc neZdZdZdZdZe ddejdejde e e e fde ejejffd Zeddefd Zedefd Zed ejde eeffd ZddZdej(de ejejffdZy)CacheLayerMixinz0Base, abstract class for a single layer's cache.Fc"d\|_|_y)N)NNkeysvaluesselfs k/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/cache_utils.py__init__zCacheLayerMixin.__init__!s!+ 4;N key_states value_states cache_kwargsreturncyNrrr r!s rupdatezCacheLayerMixin.update$s -0rcyr$r%rcache_positions rget_seq_lengthzCacheLayerMixin.get_seq_length,s:=rcyr$r%rs rget_max_cache_shapez#CacheLayerMixin.get_max_cache_shape/s*-rr*cyr$r%r)s rget_mask_sizeszCacheLayerMixin.get_mask_sizes2sORrcl|jj|jjy)z4Resets the cache values while preserving the objectsN)rzero_rrs rresetzCacheLayerMixin.reset5s   rbeam_idxc|jjrF|jj}|jjd|j ||_|j jrG|j j}|j jd|j ||_yy)z,Reorders this layer's cache for beam search.rN)rnumeldevice index_selecttor)rr3r6s r reorder_cachezCacheLayerMixin.reorder_cache:s 99?? YY%%F ..q(++f2EFDI ;;   [[''F++221hkk&6IJDK rr$r"N)__name__ __module__ __qualname____doc__is_compileablerrtorchTensorr dictstrrtupler'intr+r-r/r2 LongTensorr9r%rrrrs:N, 26 0LL0ll0tCH~. 0 u||U\\) * 00=S==-S--RU\\ReCHoRR Ke&6&6K5u||A[;\Krrc eZdZdZdZ ddej dej deee e fde ej ej ffdZ dde fd Zde fd Zd ej ddfd Zd e ddfdZde ddfdZdej ddfdZdej de e e ffdZedej dej ddfdZy) DynamicLayeraE A cache layer that grows dynamically as more tokens are generated. This is the default for generative models. It stores the Key and Value states as tensors with shape `[batch_size, num_heads, seq_len, head_dim]`. See `CacheLayerMixin` for details on common methods that are implemented by all cache layers. FNrr r!r"c|j||_||_nPtj|j|gd|_tj|j|gd|_|j|jfS)a Updates the cache with the new `key_states` and `value_states`. Parameters: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. cache_kwargs (`dict[str, Any]`, *optional*): Additional arguments for the cache subclass. No additional arguments are used in `DynamicLayer`. Return: A tuple containing the updated key and value states. dim)rrr@catr&s rr'zDynamicLayer.updateNse( 99 "DI&DK 499j"9rBDI))T[[,$?RHDKyy$++%%rc|j|jjdk(ry|jjdS)1Returns the sequence length of the cached states.rrJ)rr5shaper)s rr+zDynamicLayer.get_seq_lengthjs4 99   1Q 6yyr""rcy)zeReturns the maximum sequence length of the cache object. DynamicLayer does not have a maximum length.r%rs rr-z DynamicLayer.get_max_cache_shapepsrr3cd|j|jjr|jjd|j|jj|_|j jd|j|j j|_yyy)DReorders the cache for beam search, given the selected beam indices.Nr)rr5r7r8r6rrr3s rr9zDynamicLayer.reorder_cachetsw 99 TYY__%6 ..q(++dii>N>N2OPDI++221hkk$++BTBT6UVDK&7 r max_lengthc*|dkr|jt|z }|j|kry|jR|jjr7|jdd|ddf|_|jdd|ddf|_yyy)z Crop the past key values up to a new `max_length` in terms of tokens. `max_length` can also be negative to remove `max_length` tokens. rN.)r+absrr5r)rrVs rcropzDynamicLayer.cropzs >,,.Z@J    J .  99 TYY__%6 #{ {A"56DI++c;J;&9:DK&7 rrepeatsc|j`|jjrE|jj|d|_|jj|d|_yyy)z8Repeat the cache `repeats` times in the batch dimension.NrrK)rr5repeat_interleaverrrZs rbatch_repeat_interleavez$DynamicLayer.batch_repeat_interleavesW 99 TYY__%6 33G3CDI++77Q7GDK&7 rindicesc|jH|jjr-|j|df|_|j|df|_yyy)z is_slidingr@rAr rBrCrrDr'rEr+r-rFr9rYr^rbr/ classmethodrkr%rrrHrHDs)J 26 &LL&ll&tCH~. & u||U\\) * &8#S# SWe&6&6W4W ;s ;t ;HsHtH 4ELL4T4 $U\\$eCHo$ ell~rrHcdeZdZdZdZdZejddfdededed ed ejd e d e efd Z defdZ ddejdejde ee efdeejejffdZddefdZdej(ddfdZdejdeeeffdZy) StaticLayerab A static cache layer that stores the Key and Value states as static tensors with shape `[batch_size, num_heads, seq_len, head_dim]`. It allocates its full backing tensors up-front and mutates them in-place. Built for `torch.compile` support. See `CacheLayerMixin` for details on common methods that are implemented by all cache layers. TFcpuN max_cache_len batch_size num_headshead_dimdtyper6sliding_windowc||_||_||_||_||_||_t j|||j|f|||_t j|||j|f|||_ t jj|jt jj|jy)az Args: max_cache_len (`int`): Maximum number of tokens that can be stored, used for tensor preallocation. batch_size (`int`): Maximum batch size the cache is pre-allocated for. num_heads (`int`): Number of attention heads. head_dim (`int`): Per-head hidden dimension. dtype (`torch.dtype`, defaults to `torch.float32`): Data type of the cache tensors. device (`str` or `torch.device`, defaults to `"cpu"`): Device on which the cache tensors will be materialised. Notes: Static layers allocate their full backing tensors up-front and mutate them in-place. See the documentation of `Cache` for shared helper methods that operate uniformly across all layer types. rur6N) rqmax_batch_sizersrtrur6r@zerosrr_dynamomark_static_address)rrqrrrsrtrur6rvs rrzStaticLayer.__init__s<+("    KK D$6$6 A  kk D$6$6 A    ))$))4 ))$++6rr"c|jS)z+Return the maximum cache shape of the cacherqrs rr-zStaticLayer.get_max_cache_shapes!!!rrr r!cJ|r|jdnd}|j|jj}|j|jj}|j |j k7re|j |_|jj|j |_|jj|j |_|7|jj ||jj |n; |jjd|||jjd|||j|jfS#t$r/||jdddd|f<||jdddd|f<YOwxYw)a Update the static cache tensors in place. Args: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. cache_kwargs (`dict[str, Any]`, *optional*): Additional arguments for the cache. Returns: tuple[`torch.Tensor`, `torch.Tensor`]: The updated key and value states. r*N) getr8rrurr6copy_ index_copy_NotImplementedError)rrr r!r*s rr'zStaticLayer.updatesG"@L))*:;QU]]499??3 #t{{'8'89 ;;*++ +$++DK T[[1DI++..5DK  ! IIOOJ ' KK  l + A %%aD ''><H yy$++%% ' A2< !Q./4@ Aq.01 As:E**5F"!F"c|t|ddzS|j/|jdjdj}|Sd}|S)rOrRr )rrrKr)rEranysum)rr* seq_lengths rr+zStaticLayer.get_seq_length"sb  %~b)A-. .=AIIr?rlr@float32rErurCr rr-rArBrrDr'r+rFr9r/r%rrrorosNJ#]](,272727 27  27 {{ 2727! 27h"S"26 +&LL+&ll+&tCH~. +& u||U\\) * +&ZS@e&6&6@4@$U\\$eCHo$rroc eZdZdZdZfdZ d dejdejdee e e fde ejejffdZ d ejde eeffd ZxZS) SlidingWindowLayerz A static cache layer that implements sliding window attention caching. See `CacheLayerMixin` for details on common methods that are implemented by all cache layers. Tcv|jdd}| t||n|}t| g||d|i|y)z Args: sliding_window (`int`): Effective window size: number of tokens that are kept on each update call. rqN)popminsuperr)rrvargskwargsrq __class__s rrzSlidingWindowLayer.__init__BsH  ?D9 >K>WNM:]k  M$MdMmMfMrrr r!r"c|r|jdnd}| td|j|jk7re|j|_|jj |j|_|j j |j|_|j |jj }|j |j j }|jd|jkDrr|dddd|j dddf}|dddd|j dddf}|jj||j j|||fStj|j|j}|ddz}||jkD} || jz|jz} |jdddd| f} |j dddd| f} |jd|jdz } | jd | |}| jd | |}|jj||j j||j|j fS#t$r;| j!}| j!}||dddd| f<||dddd| f<YwxYw) a Update the sliding window cache tensors in place. Args: key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. cache_kwargs (`dict[str, Any]`, *optional*): Additional arguments for the cache. Returns: tuple[`torch.Tensor`, `torch.Tensor`]: The updated key and value states. r*Nz9`cache_position` must be provided for SlidingWindowLayer.r)r6rRr )rmaxr)r ValueErrorr6rr8rrurPrqrr@arangerclamp index_copyrclone)rrr r!r*new_knew_vslicingcurrent_seq_lento_shiftr_ k_out_shifted v_out_shiftedupdate_position k_out_updated v_out_updateds rr'zSlidingWindowLayer.updateLs"@L))*:;QU  !XY Y ;;*++ +$++DK T[[1DI++..5DK]]499??3 #t{{'8'89     "T%7%7 7q!d&8&8%8%:A=>E A(:(:':'rlrr@rAr rBrCrrDr'rEr/ __classcell__rs@rrr9s JN26 A&LLA&llA&tCH~. A& u||U\\) * A&F$U\\$eCHo$rrc eZdZdZfdZ d dej dej deee e fde ej ej ffdZ d fd Z d ej de eeffd ZxZS) ChunkedSlidingLayerz An extended SlidingWindowLayer that supports prefill chunking, originally implemented for Llama 4. See `SlidingWindowLayer` for details on common methods that are implemented by all cache layers. c2t||i|d|_yNr)rrcumulative_lengthrrrrs rrzChunkedSlidingLayer.__init__s $)&)!"rrr r!r"cb|r|jdnd}| td|j|jk7re|j|_|jj |j|_|j j |j|_|j }|xj |jdz c_||jk\}|rtj|jddddddddf|fd}tj|j ddddddddf|fd}|jddk(r1|jj||j j||j|j fS|s||jdz|jkDro|dk(r|}|}ntj|jddddd|ddf|fd}tj|j ddddd|ddf|fd}nS |jjd|||j jd|||j|j fS|jj|dddd|j dddf|j j|dddd|j dddf||fS#t$r/||jdddd|f<||j dddd|f<YwxYw)Nr*z:`cache_position` must be provided for ChunkedSlidingLayer.rJr rKrr)rrr6rr8rrrPrqr@rMrrr) rrr r!r*ris_fullfull_key_statesfull_value_statess rr'zChunkedSlidingLayer.updates @L))*:;QU  !YZ Z ;;*++ +$++DK T[[1DI++..5DK 22 *"2"22"66#t'9'99 #ii1aQ;)?(LRTUO % 4;;q!QR{+C\*RXZ [ #q( 0 !!"34yy$++--.1A1A!1DDtGYGYY A%",$0!"'))TYYq!=O>O=OQR7R-SU_,`fh"i$)IIt{{1aASBSASUV;V/WYe.fln$o! A %%aD ''><H99dkk) ) 1t/A/A.A.CQ(FGH +Aq43E3E2E2G,JKL 111' A2< !Q./4@ Aq.01 As6:K665L.-L.c0t|d|_yr)rr2rrrs rr2zChunkedSlidingLayer.resets  !"rr*c|jd}|d}|j}tj||z dzd}||k\r ||zdz }||fS||kr||z|kDr ||z}||fS|}||fS)Nrr r)rPrqr@r)rr*rerrvrdrgs rr/z"ChunkedSlidingLayer.get_mask_sizess%++A. -a0++KK 4~ E IqQ > 1&59I)## "N 27Kl7Z]k7k,|;I)##'I)##rr$r:)r;r<r=r>rr@rAr rBrCrrDr'r2rEr/rrs@rrrs #26 22LL22ll22tCH~. 22 u||U\\) * 22h#$U\\$eCHo$rrc4eZdZdZddZ ddddej dej d ed ee e e fde ej ej ff d Z dddd ej d ej d ed ee e e fde ej ej ff dZy)CacheProcessorz Base class for cache processors. It defines a pre-update and post-update methods that are called before and after the cache update. This class should be subclassed. cacheCacher"Nc Htd|jjd)a Initialize the processor and perform compatibility checks with the cache. Args: cache (`Cache`): The cache instance this processor will be applied to. **kwargs: Additional arguments that may be needed for initialization. z!Make sure to implement `init` in .)rrr;)rrrs rrzCacheProcessor.__init__s%"$EdnnF]F]E^^_"`aarrr layer_idxr!c ||fS)a Function called before the cache update. Can modify the key/value states. Args: cache (`Cache`): The cache instance. key_states (`torch.Tensor`): The new key states to cache. value_states (`torch.Tensor`): The new value states to cache. layer_idx (`int`): The index of the layer to cache the states for. cache_kwargs (`dict[str, Any]`, *optional*): Additional arguments for the cache. Returns: The modified key and value states. r%rrrr rr!s r pre_updatezCacheProcessor.pre_updates*<''r key_tensors value_tensorsc ||fS)a( Function called after the cache update. Can process the cached data. Args: cache (`Cache`): The cache instance. key_states (`torch.Tensor`): The key states that were cached. value_states (`torch.Tensor`): The value states that were cached. layer_idx (`int`): The index of the layer that was updated. cache_kwargs (`dict[str, Any]`, *optional*): Additional arguments for the cache. Returns: The final key and value states to return to the model. r%)rrrrrr!s r post_updatezCacheProcessor.post_updates*M))r)rrr"Nr$)r;r<r=r>rr@rArEr rBrCrrDrrr%rrrrs b 26 ((LL(ll (  ( tCH~. ( u||U\\) *(:26 **\\*|| *  * tCH~. * u||U\\) **rrceZdZdZddddeeejffdZ ddddejdejd e d e e ee fd eejejff d Zddd e fd Zddd e fdZddd e fdZy)OffloadedCacheProcessora A cache processor that offloads cache tensors to conserve accelerator memory. This processor manages moving cache tensors between accelerator and CPU memory, using asynchronous prefetching to minimize performance impact. Works with both dynamic and static layers. rroffload_devicec tj||_g|_d|_d|_tj jsNtddrtjjs#tdtddrdzdztd|jD|_ |jrt|jD]\}}|d k(r|jd n |j}|j j#||_|j$j#||_|jj'|jd t)||j*k7r t-d tddrtj.|_ytj j/|_y) z@Initialize the offload processor and check device compatibility.N2.7T accept_devz3OffloadedCacheProcessor can only be used with a GPUz or XPUc3<K|]}t|tywr$) isinstancero.0rjs r z3OffloadedCacheProcessor.__init__..HsVZ{;Vsrr6z?If static layers are used, all cache layers must be initialized)r@r6roriginal_deviceprefetch_streamr3cuda is_availablerxpu RuntimeErrorrlayers is_static enumeratelayer_init_kwargsrr8rappendlennum_hidden_layersrStream)rrrrirjr6s rrz OffloadedCacheProcessor.__init__8s#ll>:!#  JJ # # %)%DAeiiF\F\F^E 9%D Q9[ WY[  VVV >>%ell3 O5>?1f00:$J]J]"ZZ]]62 $||v6 $$++E,C,CH,MN  O 5zU444 !bcc8$OELLN UZU_U_UfUfUh rNrr rr!r"c$t||kr tdt||k(r;|jj|j|j ||||fSt ddr-tjjjn,tjjj|j |||j|||j||dzt|z||fS)z5Handles prefetching and eviction before cache update.zWOffloadedCache does not support model usage where layers are skipped. Use DynamicCache.rTrr )rrrrr6_evict_previous_layerrr@ acceleratorcurrent_stream synchronizer_ensure_layer_on_device_prefetch_layerrs rrz"OffloadedCacheProcessor.pre_updateVs u: !vw w Z9 $  ' ' (9(9 :  & &ui 8<'')4@!!002>>@ ))+779  & &ui 8  ( ( :  Q#e*(D E<''rc|t|krtddr |jn(tjj |j5|j |}|j|jj|d|j|_|j|jj|d|j|_ dddyy#1swYyxYw)z(Starts prefetching the next layer cache.rTr non_blockingN) rrrr@rstreamrrrr8r)rrrr6s rrz'OffloadedCacheProcessor._prefetch_layerrs s5z !-UtD$$ZZ&&t';';< n --i8/4||I/F/K/K/N/Nvdh/N/i Y',16i1H1O1O1R1RSYhl1R1m Y'. n n " n ns BC##C,c\t|dk\r|dz t|z}|j|jj|jd|j|_|j|j j|jd|j|_yy)z*Moves the previous layer cache to the CPU.rr TrN)rrrr8rr)rrrprev_layer_idxs rrz-OffloadedCacheProcessor._evict_previous_layers u:?'!ms5z9N05 ^0L0Q0Q0T0T##$1U1ELL ( -38,,~2N2U2U2X2X##$3Y3ELL ( / rc|t|kr|jj|j|jj |j ||_|j |jjd|j|j |_|j |jjd|j|j |_ yyy)z4Ensures the current layer is on the original device.Nr) rrrr3r8rrrr7r)rrrs rrz/OffloadedCacheProcessor._ensure_layer_on_devices s5z !  , , .}}( $ 0 01E1Ei1P Q /4||I/F/K/K/X/XYZ\`\i\i/j Y',16i1H1O1O1\1\]^`d`m`m1n Y'.) "rrpr$)r;r<r=r>r rCr@r6rrArEr rBrrDrrrrr%rrrr/s g uS%,,=N7O H26 ((LL(ll (  ( tCH~. ( u||U\\) *(8 nW n n 7 s  oW o orrceZdZdZddddddej dfdd d ed ed ed edededejdefdZ dZ d dd dejdejdede e eefdeejejff dZdejdedejfdZdejdejfdZdedefdZy)!QuantizedCacheProcessorz A cache processor that applies quantization to cache tensors to reduce memory usage. This processor quantizes cache tensors after they are stored, maintaining a residual length in original precision and quantizing older tokens. quantor@rprrbackendnbitsaxis_key axis_value q_group_sizeresidual_length compute_dtyper6c ||_||_||_||_||_||_||_| |_g|_g|_ |jd|_ t|jdts tdy)a Parameters: backend (`str`, defaults to `"quanto"`): Backend to use when performing quantization, Can be one of [`quanto`, `HQQ`] nbits (`int`, defaults to 4): Number of bits, can be 2 or 4 for the `quanto` backend and one of [1, 2, 3, 4, 8] for the `HQQ` backend. Defaults to 2. axis_key (`int`, defaults to 0): Axis over which to perform grouping for the key tensors. Can be [0, -1] for `quanto` backend and [0, 1] for `HQQ` backend. axis_value (`int`, defaults to 0): Axis over which to perform grouping for the value tensors. Can be [0, -1] for `quanto` backend and [0, 1] for `HQQ` backend. q_group_size (`int`, defaults to 64): Size of the quantization group, should be a divisor of the model's hidden dimension. Defaults to 64. residual_length (`int`, defaults to 128): Length of the residual cache which will always be stored in original precision. Defaults to 128. compute_dtype (`torch.dtype`, defaults to `torch.float16`): The default dtype used for computations in the model. Keys and Values will be cast to this dtype after dequantization. device (`str`, defaults to `"cpu"`): Device on which to perform computations, should be same as the model's device. rzr@float16rCrErurrrAr rBrrDrrrboolrr%rrrrsZ "%*]]1]1]1] 1]  1]  1]1]1]{{1]1]f2t26 ;0;0\\;0|| ;0  ;0 tCH~. ;0 u||U\\) *;0zZ ZCZELLZ\%,,\5<<\63646rrceZdZdZddddddej dfdd d ed ed ed edededejdeddffd Z dejdedejfdZ dejdejfdZ xZ S)QuantoQuantizedCacheProcessorz Quantized cache processor that uses `quanto` as a backend to perform quantization. Current implementation supports `int2` and `int4` dtypes only. rrrrrrprrrrrrrrrr6r"Nc `t|||||||||| |dk7rtd|trct j t jj d} | t j dkrtd| dddl m } m } m } |jd vrtd |j|jd vrtd |j|jd vrtd |j|jdk(r n |_ |_y)z-Initialize the quanto quantization processor.rzFQuantoQuantizedCacheProcessor only supports `quanto` backend, but got zoptimum-quantoz0.2.5zYou need optimum-quanto package version to be greater or equal than 0.2.5 to use `QuantoQuantizedCacheProcessor`. Detected version rr) MaxOptimizerqint2qint4)rrzA`nbits` for `quanto` backend has to be one of [`2`, `4`] but got )rrRzE`axis_key` for `quanto` backend has to be one of [`0`, `-1`] but got zG`axis_value` for `quanto` backend has to be one of [`0`, `-1`] but got rN)rrrrr parse importlibmetadata ImportErroroptimum.quantor(r)r*rrrqtype optimizer)rrrrrrrrrr6optimum_quanto_versionr(r)r*rs rrz&QuantoQuantizedCacheProcessor.__init__WsL  7E8ZXegm  h efmenop p & (%,]]93E3E3M3MN^3_%` "%w)??!Z[qZrrst B A ::V #`aeakak`lmn n == 'deiererdstu u ??' )YZ^ZiZiYjk #jjAoU5 %rrrctrTddlm}|j||j||j \}}|||j||||j }|Sy)z%Quantize tensor using quanto backend.r)quantize_weightN)rr/r4r1r0r)rrrr4scale zeropointqtensors rrz'QuantoQuantizedCacheProcessor._quantizesW & ( 6#~~fdjj$HYHYZ E9%fdjj$yRVRcRcdGN )rr7c"|jS)z'Dequantize tensor using quanto backend.) dequantize)rr7s rrz)QuantoQuantizedCacheProcessor._dequantizes!!##r)r;r<r=r>r@r#rCrErurrArrrrs@rr&r&Qs "%*]](((((( ((  ((  (((((({{(((( ((T CELL$5<<$ELL$rr&ceZdZdZddddddej dfdd d ed ed ed edededejdeddffd Z dejdede eje ffdZ de eje fdejfdZxZS)HQQQuantizedCacheProcessorz Quantized cache processor that uses `HQQ` as a backend to perform quantization. Current implementation supports `int2`, `int4`, `int8` dtypes. rrrrrrprrrrrrrrrr6r"Nc Tt |||||||||| |dk7rtd||jdvrtd|j|jdvrtd|j|j dvrtd|j t |_y) z*Initialize the HQQ quantization processor.rzCHQQQuantizedCacheProcessor only supports `quanto` backend, but got rzM`nbits` for `HQQ` backend has to be one of [`1`, `2`, `3`, `4`, `8`] but got )rr zA`axis_key` for `HQQ` backend has to be one of [`0`, `1`] but got zC`axis_value` for `HQQ` backend has to be one of [`0`, `1`] but got N)rrrrrr HQQQuantizer quantizer) rrrrrrrrrr6rs rrz#HQQQuantizedCacheProcessor.__init__s  7E8ZXegm  h bcjbklm m ::_ ,_`d`j`j_kl  == &`aeanan`opq q ??& (bcgcrcrbstu u%rrrc|jj|||j|j|j|j \}}|j|d<|jj |||j|dj|j|d<|dj|j|d<||fS)z"Quantize tensor using HQQ backend.)rr6rr group_sizer)metar6r5zero)r>quantizer6rrrrr8)rrrr7rAs rrz$HQQQuantizedCacheProcessor._quantizes// ;;,,**(( 0 !% 2 2_ G$t{{CW ((8W F|w~~6V }rqtensor_and_metacH|\}}|jj||}|S)z$Dequantize tensor using HQQ backend.)r>r9)rrD quant_tensorrArs rrz&HQQQuantizedCacheProcessor._dequantizes'- d**<> r)r;r<r=r>r@r#rCrErurrArDrBrrrrs@rr;r;s "%*]]&&& &  &  &&&{{&& &B CE%,,PTBT._wrapped_updates    +'+';';'F'Fj, <( $J &(j, S_%`" ]    +)-)=)=)I)Ik=)\* &KM))rr$) functoolswrapsr@rArEr rBrCrrD)rGrKs` rapply_processorsrNs__R 26 *LL*ll* * tCH~. * u||U\\) * **0 rc6eZdZdZd dZdZdZdZdZdZ y) KeyValuesWrapperzHelper class for Cache that simulates layer-indexed key/value lists from a layered cache. This allows for BC access and writing, e.g., cache.key_cache[idx] = ... Deprecated in favor of Cache.layers[idx].keys/values. TODO: remove in v4.56.0c ||_||_yr$)r cache_type)rrrRs rrzKeyValuesWrapper.__init__s $rct|tr1|j|Dcgc]}t||jc}St|j||jScc}wr$)rslicergetattrrR)ridxrjs r __getitem__zKeyValuesWrapper.__getitem__sQ c5 !AESAQRGE4??3R Rt{{3'99SsA%ct|tr9t|j||D]\}}t ||j |yt |j||j |yr$)rrTziprsetattrrR)rrVvaluerjvals r __setitem__zKeyValuesWrapper.__setitem__sZ c5 !!$++c"2E: 5 st4 5 DKK$doou =rc,t|jSr$)rrrs r__len__zKeyValuesWrapper.__len__s4;;rc#^K|jD]}t||jywr$)rrUrRrrjs r__iter__zKeyValuesWrapper.__iter__s*[[ 2E%1 1 2s+-c,t|jSr$)r$rrs r__bool__zKeyValuesWrapper.__bool__sDKK  rN)r) r;r<r=r>rrWr]r_rbrdr%rrrPrPs&U%: > 2!rrPc,eZdZdZ d+deeeeeefdee deee ee fdee dee dee je dfd ee jd eee e jfd ee fd Zd e dee j&e j&ffdZdZdZdZd e ddfdZe d,de j&de j&d e deee efdee j&e j&ff dZd-d e de fdZde j&d e dee e ffdZedefdZ edefdZ!d.d e de fdZ"dZ#de jHfd Z%d!e fd"Z&d#e fd$Z'd%e j&fd&Z(ede fd'Z)ede fd(Z*ede+fd)Z,edee+fd*Z-y)/ra Base container for per-layer key/value caches. A `Cache` behaves like a list of `CacheLayerMixin` objects, one per model layer. Sub-classes such as `DynamicCache`, `StaticCache`, or `SlidingWindowCache` simply pre-select which `CacheLayerMixin` class to use and may attach a `CacheProcessor` (off-loading, quantization). Example ------- ```python from transformers import AutoModelForCausalLM, AutoTokenizer, DynamicCache model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf") tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf") inputs = tok("Hello", return_tensors="pt") cache = DynamicCache() outputs = model(**inputs, past_key_values=cache, use_cache=True) ``` Parameters: layer_classes (`type[CacheLayerMixin]` or `list[type[CacheLayerMixin]]`): A list of `CacheLayerMixin` classes to instantiate for the cache. If only a `CacheLayerMixin` class is provided, then it is used for all layers. config (`PretrainedConfig`, *optional*): Model configuration used to infer number of layers, head sizes, default device/dtype, etc. cache_processor (`CacheProcessor` or `str`, *optional*): Cache processor to apply (e.g., "offloaded", "quanto_quantized", "hqq_quantized") or a CacheProcessor class. max_batch_size (`int`, *optional*): Maximum batch size for static caches. max_cache_len (`int`, *optional*): Maximum sequence length. For hybrid caches, SlidingWindowLayers are clamped to `min(sliding_window, max_cache_len)`, StaticLayers use full `max_cache_len`. device (`torch.device`, *optional*): Device for cache tensors. dtype (`torch.dtype`, *optional*): Data type for cache tensors. layer_device_map (`dict[int, Union[str, torch.device]]`, *optional*): Per-layer device mapping. tp_size (`int`, *optional*): Tensor parallel size to adjust the number of key/value heads. Additional keyword arguments are forwarded to the chosen layers constructor(s) and CacheProcessors. See the documentation of the relevant `CacheLayerMixin` class and `CacheProcessor` class for more details. N layer_classesconfigrJryrqr6rulayer_device_maptp_sizec  Tg|_||_t|tr t|n|} | j |||||| t | | \} } t|fi| |_t|dd|_ |j|jdz | | |fi| |_ yd|_ y)N)ryrqr6rurhrirr ) rrfrrCPROCESSOR_CLASS_MAPr'parse_processor_args"parse_layer_args_from_model_configrrUrappend_new_layersrJ) rrfrgrJryrqr6rurhrirprocessor_classprocessor_kwargss rrzCache.__init__8s.0 *BL_^aBb-o>hw )'-  $8#P &!CF!Uf!U!(1Da!H t559:L[LgtH7GHmqrrr"c|t|jkr2|j|j|j|jfSt dt|jd|z Support for backwards-compatible `past_key_value` indexing, e.g. `past_key_value[0][0].shape[2]` to get the sequence length. zCache only has z. layers, attempted to access layer with index )rrrrKeyErrorr"s rrWzCache.__getitem__Ysh s4;;' ';;y).. I0F0M0MM M!#dkk"2!33abkalm rc#Ktt|D]6}|j|j|j|jf8ywz Support for backwards-compatible `past_key_value` iteration, e.g. `for x in past_key_value:` to iterate over keys and values N)rangerrrrr"s rrbzCache.__iter__esK s4y) OI;;y).. I0F0M0MN N OsAAc^t|dd#t|ddt|jSyt|ddduxrTt|jdk(xr:t |jdt xr|jdj du}|st|jSdS) Support for backwards-compatible `past_key_value` length, e.g. `len(past_key_value)`. This value corresponds to the number of layers in the model. rN key_cacherr )rUrryrrrHr)r dynamic_emptys rr_z Cache.__len__ms 44 ( 0t[$/;4>>** D(D ) 5 ,DKK A% ,4;;q><8 , A##t+  (5s4;;;!;rcN|jjd|jdS)Nz(layers=))rr;rrs r__repr__zCache.__repr__s$..))*(4;;-qAArct|j|kr|jj}|jj dd*|j dt|j|d<t |jtr"|jt|jn |j}|di|}|jj|t|j|kryy)a Appends layers to the cache until the layer `layer_idx` is reached. Used for preallocation in static caches and on the fly in dynamic caches. Args: layer_idx (`int`): The index of the layer to append. rhNr6r%) rrrcopyrrrrflistr)rrrnew_layer_class new_layers rrnzCache.append_new_layerss$++)+++002F%%))*.sAE5''As)allrrs rr?zCache.is_compileablesAT[[AAArcV|jDcgc]}t|ddc}Scc}w)z9Return whether the layers of the cache are sliding windowrlF)rrUras rrlzCache.is_slidings'BFM|U3MMMs&)NNNNNNNNr$rN)r).r;r<r=r>r rtyperr rrCrrEr@r6rurBrrDrArWrbr_r}rnrNrr'r+r/propertyrPryrr-r2rFr9rYr^rbryrqr$r?rlr%rrrr s)\.2FJ(,'+15'+>B!%rT$"78$:OOPr)*r"%T.-A(A"BC r ! r  } rellC-.r $r#4U\\(9#:;r#rB S U5<<3M-N O<&B*3*4*( 26 ULLUllU U tCH~. U u||U\\) * UU6EEE$U\\$c$eTWY\T\o$5+557-77>> from transformers import AutoTokenizer, AutoModelForCausalLM, DynamicCache >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> past_key_values = DynamicCache() >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation DynamicCache() ``` ddp_cache_datact||dti||:|D]4\}}|jj tj ||6yyNrf)rrrHrrrk)rrrrrr rs rrzDynamicCache.__init__*s\ dE|EfE  %,: X( L ""<#<#r rrDr@rArrrm FloatTensorrrrs@rrrs6 Xxu||U\\?Y9Z0['\ XuU\\5<<-G'H#'M!N eEK|]}t|t ywr$)rrHrs rrz"_get_cache_dict..SsMu:e\22MszFThis pytree flattening function should only be applied to DynamicCachez[DynamicCache + torch.export is tested on torch 2.6.0+ and may not work on earlier versions.)ryr)rrrr rrrr)rrjs r_get_cache_dictrRs M M Mgh h1   m  38,,Y%**BX%**Y6;ll_UellF^ELL_  Y_sB&#B&B+B+contextctjjj||}t }|j dg}|j dg}t tt|t|D]?}|t|kr||nd}|t|kr||nd}|j|||A|S)Nryr) r@utils_pytree_dict_unflattenrrrvrrr') rr dictionaryrkey_list value_listrVr r[s r_unflatten_dynamic_cacher`s[[((88I >>+r2^^M26 S]C O<= *C#&X#6(3-DC'*S_'rqs!ekk11??P]@^_rrcftjjjt |Sr$)r@rr_dict_flatten_with_keysrrs rrrts#5;;3F3F3^3^ M *4 r)serialized_type_nameflatten_with_keys_fnchtjjjt ||Sr$)r@fxr_dict_flatten_specr)rspecs rrrzs$%((*:*:*M*Mo^cNdfj*krc$eZdZdZdfd ZxZS)OffloadedCachea A drop-in replacement for DynamicCache that conserves accelerator(GPU, XPU) memory at the expense of more CPU memory. Useful for generating from models with very long context. In addition to the default accelerator stream, where all forward() computations happen, this class uses another stream, the prefetch stream, which it creates itself. Since scheduling of operations on separate streams happens independently, this class uses the prefetch stream to asynchronously prefetch the KV cache of layer k+1 when layer k is executing. The movement of the layer k-1 cache to the CPU is handled by the default stream as a simple way to ensure the eviction is scheduled after all computations on that cache are finished. c.t|ty)N)rJrrrrs rrzOffloadedCache.__init__s )@Arr:r;r<r=r>rrrs@rrr~s BBrrc"eZdZdZfdZxZS) StaticCachea Static Cache class to be used with `torch.compile(model)` and `torch.export()`. See `Cache` for details on common methods that are implemented by all cache classes. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, StaticCache >>> model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf") >>> inputs = tokenizer(text="My name is Llama", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = StaticCache(config=model.config, max_batch_size=1, max_cache_len=max_generated_length, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation StaticCache() ``` c0t||dti|yr)rrrors rrzStaticCache.__init__s TD{DVDrrrs@rrrs2EErrc$eZdZdZdfd ZxZS)OffloadedStaticCacheaG A drop-in replacement for StaticCache that conserves accelerator memory by offloading cache tensors to CPU when not actively being used. This cache maintains the compilation-friendly properties of StaticCache while enabling much longer sequences by offloading inactive layers to CPU memory. See `Cache` for details on common methods that are implemented by all cache classes. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, OffloadedStaticCache >>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2") >>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> inputs = tokenizer(text="My name is GPT2", return_tensors="pt") >>> # Prepare a cache class with offloading >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = OffloadedStaticCache( ... config=model.config, ... max_batch_size=1, ... max_cache_len=max_generated_length, ... device=model.device, ... dtype=model.dtype ... ) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache with offloaded layers OffloadedStaticCache() ``` c0t||dti|yNrJrrs rrzOffloadedStaticCache.__init__ $R0GR6Rrr:rrs@rrrsBSSrrc"eZdZdZfdZxZS)SlidingWindowCachea Sliding Window Cache class to be used with `torch.compile` for models like Mistral that support sliding window attention. Every time when we try to update the cache, we compute the `indices` based on `cache_position >= self.sliding_window - 1`, if true(which means the cache can not hold all the old key value states and new states together because of the sliding window constraint), we need to do a cycle shift based on `indices` to replace the oldest states by the new key value states passed in. The `to_shift` is only true once we are above sliding_window. Thus with `sliding_window==64`: indices = (slicing + to_shift[-1].sum()-1) % self.sliding_window tensor([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 0]) We overwrite the cache using these, then we always write at cache_position (clamped to `sliding_window`) See `Cache` for details on common methods that are implemented by all cache classes. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, SlidingWindowCache >>> model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3") >>> tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3") >>> inputs = tokenizer(text="My name is Mistral", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = SlidingWindowCache(config=model.config, max_batch_size=1, max_cache_len=max_generated_length, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation SlidingWindowCache() ``` c0t||dti|yr)rrrrs rrzSlidingWindowCache.__init__s DK'9KFKrrrs@rrrs$LLLrrc(eZdZdZdeffd ZxZS) HybridCachea Hybrid Cache class to be used with `torch.compile` for models that alternate between a local sliding window attention and global attention in every other layer (originally implemented for Gemma2). Under the hood, Hybrid Cache leverages ["SlidingWindowCache"] for sliding window attention and ["StaticCache"] for global attention. For more information, see the documentation of those layer types. See `Cache` for details on common methods that are implemented by all cache classes. Example: ```python >>> from transformers import AutoTokenizer, AutoModelForCausalLM, HybridCache >>> model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b") >>> tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b") >>> inputs = tokenizer(text="My name is Gemma", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> # Leave empty space for 10 new tokens, which can be used when calling forward iteratively 10 times to generate >>> max_generated_length = inputs.input_ids.shape[1] + 10 >>> past_key_values = HybridCache(config=model.config, max_batch_size=1, max_cache_len=max_generated_length, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation HybridCache() ``` rgct|dr!|jDcgc] }t| }}ntg|jz}t ||||d|ycc}w)N layer_types)rgrf)hasattrrLAYER_CLASS_MAProrrr)rrgrr layer_typerfrs rrzHybridCache.__init__s[ 6= )KQK]K]^Z_Z8^M^)MF,D,DDM dmUfU _sA)r;r<r=r>rrrrs@rrrs8V/VVrrc eZdZy)HybridChunkedCacheN)r;r<r=r%rrrr&srrc$eZdZdZdfd ZxZS)OffloadedHybridCachea A drop-in replacement for HybridChunkedCache that conserves accelerator memory by offloading cache tensors to CPU when not actively being used. This cache maintains the compilation-friendly properties of HybridChunkedCache while enabling much longer sequences by offloading inactive layers to CPU memory. See `Cache` for details on common methods that are implemented by all cache classes. c0t||dti|yrrrs rrzOffloadedHybridCache.__init__4rrr:rrs@rrr)sSSrrc$eZdZdZdfd ZxZS)QuantizedCachea A quantizer cache similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://arxiv.org/abs/2402.02750). It allows the model to generate longer sequence length without allocating too much memory for Key and Value cache by applying quantization. The cache has two types of storage, one for original precision and one for the quantized cache. A `residual length` is set as a maximum capacity for the original precision cache. When the length goes beyond maximum capacity, the original precision cache is discarded and moved into the quantized cache. The quantization is done per-channel with a set `q_group_size` for both Keys and Values, in contrast to what was described in the paper. It stores Keys and Values a list of quantized tensors (tuples in case we need to store metadata), one for each layer. Additionally, it stores the Key and Value in original precision states as a list of tensors, one for each layer. The size of each tensor is `[batch_size, num_heads, seq_len - residual_length, head_dim]`. See `Cache` for details on common methods that are implemented by all cache classes. c v|dk(rt}n|dk(rt}ntd|dt|dd|i|y)NrhqqzUnknown quantization backend ``rJr%)r&r;rrr)rrr processorrs rrzQuantizedCache.__init__HsG h 5I  2I=gYaHI I ==f=rr:rrs@rrr8s >>rrceZdZdZddZy)QuantoQuantizedCachea A quantizer cache similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://huggingface.co/papers/2402.02750). It allows the model to generate longer sequence length without allocating too much memory for Key and Value cache by applying quantization. The cache has two types of storage, one for original precision and one for the quantized cache. A `residual length` is set as a maximum capacity for the original precision cache. When the length goes beyond maximum capacity, the original precision cache is discarded and moved into the quantized cache. The quantization is done per-channel with a set `q_group_size` for both Keys and Values, in contrast to what was described in the paper. It stores Keys and Values a list of quantized tensors (tuples in case we need to store metadata), one for each layer. Additionally, it stores the Key and Value in original precision states as a list of tensors, one for each layer. The size of each tensor is `[batch_size, num_heads, seq_len - residual_length, head_dim]` Uses `quanto` as a backend to perform quantization. Current implementation supports `int2` and `int4` dtypes only. See `Cache` for details on common methods that are implemented by all cache classes. Example: ```python >>> # Run pip install quanto first if you don't have it yet >>> from transformers import AutoTokenizer, AutoModelForCausalLM, QuantoQuantizedCache, QuantizedCacheConfig >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> cache_config = QuantizedCacheConfig(nbits=4) >>> past_key_values = QuantoQuantizedCache(cache_config=cache_config) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation QuantoQuantizedCache() ``` Nc <tj|fdti|yr)rrr&rrs rrzQuantoQuantizedCache.__init__xsd\4Q\U[\rr:r;r<r=r>rr%rrrrSs"H]rrceZdZdZdddZy)HQQQuantizedCachea A quantizer cache similar to what is described in the [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache paper](https://arxiv.org/abs/2402.02750). It allows the model to generate longer sequence length without allocating too much memory for Key and Value cache by applying quantization. The cache has two types of storage, one for original precision and one for the quantized cache. A `residual length` is set as a maximum capacity for the original precision cache. When the length goes beyond maximum capacity, the original precision cache is discarded and moved into the quantized cache. The quantization is done per-channel with a set `q_group_size` for both Keys and Values, in contrast to what was described in the paper. It stores Keys and Values a list of quantized tensors (tuples in case we need to store metadata), one for each layer. Additionally, it stores the Key and Value in original precision states as a list of tensors, one for each layer. The size of each tensor is `[batch_size, num_heads, seq_len - residual_length, head_dim]` Uses `HQQ` as a backend to perform quantization. Current implementation supports `int2`, `int4`, `int8` dtypes. See `Cache` for details on common methods that are implemented by all cache classes. Example: ```python >>> # Run pip install hqq first if you don't have it yet >>> from transformers import AutoTokenizer, AutoModelForCausalLM, HQQQuantizedCache, QuantizedCacheConfig >>> model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2-0.5B-Instruct") >>> inputs = tokenizer(text="My name is Qwen2", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> cache_config = QuantizedCacheConfig(nbits=4, axis_key=1, axis_value=1) >>> past_key_values = HQQQuantizedCache(cache_config=cache_config) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation HQQQuantizedCache() ``` Nc J|dk(sJtj|fdti|y)NHQQrJ)rrr;)rrrs rrzHQQQuantizedCache.__init__s*%dY4NYRXYr)rr:rr%rrrr|s"HZrrceZdZdZdZdedeffd ZdZdede e je je je jffd Z d Z de e e jfd Zed e e e j e j fd fddfdZd"deedefdZdZde j*fdZdefdZdefdZdededdfdZdefdZde jfdZdefdZd e jdede eeffd!ZxZS)#EncoderDecoderCachea[ Base, abstract class for all encoder-decoder caches. Can be used to hold combinations of self-attention and cross-attention caches. See `Cache` for details on common methods that are implemented by all cache classes. Example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM, DynamicCache, EncoderDecoderCache >>> model = AutoModelForCausalLM.from_pretrained("openai/whisper-small") >>> processor = AutoProcessor.from_pretrained("openai/whisper-small") >>> inputs = processor(audio=YOUR-AUDIO, return_tensors="pt") >>> # Prepare cache classes for encoder and decoder and pass it to model's forward >>> self_attention_cache = DynamicCache() >>> cross_attention_cache = DynamicCache() >>> past_key_values = EncoderDecoderCache(self_attention_cache, cross_attention_cache) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values # access cache filled with key/values from generation EncoderDecoderCache() ``` Nself_attention_cachecross_attention_cachect|t||_||_t |jdd|_i|_tt|D],}t|j|dkD|j|<.y)N)rfr?Fr) rrrHrrrUr? is_updatedrvrr$r+)rrrrrs rrzEncoderDecoderCache.__init__s |4$8!%:"%d&?&?AQSXYs#89: cI)-.C.R.RS\.]`a.a)bDOOI & crc#VKtt|D]}|jj|j|jj|j |j j|j|j j|j fywru)rvrrrrrrr"s rrbzEncoderDecoderCache.__iter__s s4y) I))00;@@))00;BB**11)<AA**11)<CC   sB'B)rr"cf|t|kr|jj|j|jj|j|j j|j|j j|jfSt dt|d|rr)rrrrrrrsr"s rrWzEncoderDecoderCache.__getitem__s s4y ))00;@@))00;BB**11)<AA**11)<CC  _SYK7efoepqr rrc,t|jS)rx)rrrs rr_zEncoderDecoderCache.__len__s 4,,--rc d}t|jdkDrOt|jj |jj D]\}}|||zfz }|S|jj }|S)z[Converts the `EncoderDecoderCache` instance into its equivalent in the legacy cache format.r%r)rrrYrr)rr self_attn cross_attns rrz#EncoderDecoderCache.to_legacy_caches t)) *Q .),))99;T=W=W=g=g=i* :% :Z!7 99  :  44DDFLrr.cR|tt}|tt|D]s}||dd\}}|jj |||t||dkDs=||dd\}}|j j |||d|j |<u|S)zUConverts a cache in the legacy cache format into an equivalent `EncoderDecoderCache`.)rrNrT)rrvrrr'rrrs rrz%EncoderDecoderCache.from_legacy_caches !-"..   &"3#78 7 +:9+Ebq+I( L**11*lIVy12Q6/>y/I!"/M,J //66zRRrct|jdr|jjt|jdr|jjnnt|jdsXt|jdsBt d|jj d|jj d|j D]}d|j |<y)Nr2zNeither self nor cross-attention cache have valid `.reset()` methods. `.reset()` should only be called on compatible cache classes, such as `StaticCache` or `SlidingWindowCache`. Got " for the self attention cache and  for the cross attention cache.F)rrr2rr__str__rr"s rr2zEncoderDecoderCache.resets 4,,g 6  % % + + - 4--w 7  & & , , .22G>rr*c:|jj||Sr$)rr/)rr*rs rr/z"EncoderDecoderCache.get_mask_sizes[s((77 RRrr) r;r<r=r>r?rrrbrErDr@rArWr_rrmrrr r+r2rFr9rCrrYr r^rbr-r/rrs@rrrs8NcUc5c  sS sU5<<u||]b]i]i3i-j s. uU\\':!; #E%*;*;U=N=N*N$OQT$TU $S SSVS /;e&6&6; #737 3 C D_ DsD AELLA ?S?SU\\ScSeTWY\T\oSrrrorc2 ttj|jjdd}|Dcic] }||vs|||}}|j Dcic] \}}||vs ||}}}||fS#t $ri|fcYSwxYwcc}wcc}}w)a Parse processor arguments from kwargs based on the processor class init signature. Args: processor_class: The processor class to inspect, or None kwargs: Dictionary of keyword arguments Returns: tuple: (processor_kwargs, remaining_kwargs) rN)rinspect signaturer parameters Exceptionitems)rorparamskrpvremaining_kwargss rrlrl_sg''(@(@ALLMabQ/5DV 6!9 DD)/UA1DT;T1UU - -- 6zEUs(5A; BB" B/B; B  B rgrrrqr6rurhriryc b|iSt|dd4d|jvr&d|jvrt|dd td|xs |j}t|dd |jn|j |j z}t|dd |j n |j} |$|d kDr| |zd k7rtd | d |d | |z} ||n|||tj|nd|||| t|ddd} | jD cic] \} } |  | |  c} } Scc} } w)a5 Parse layer arguments from model configuration for cache initialization. Args: config (`Optional[PretrainedConfig]`): Model configuration containing shape/device info. batch_size (`Optional[int]`): Batch size for cache initialization. max_cache_len (`Optional[int]`): Maximum sequence length for cache. device (`Union[torch.device, str, None]`): Device for cache tensors. dtype (`Optional[torch.dtype]`): Data type for cache tensors. layer_device_map: Per-layer device mapping. tp_size (`Optional[int]`): Tensor parallel size to adjust number of key/value heads. max_batch_size (`Optional[int]`): Maximum batch size for cache initialization. Returns: `dict`: Dictionary containing parsed layer arguments for cache initialization. Nrsliding_attentionfull_attentionrvzSetting up a hybrid or sliding window KVCache requires the model config supporting sliding window attention, please check if there is a `sliding_window` field in the model config and it's not set to None.rtnum_key_value_headsr rzNumber of key value heads z+ must be divisible by tensor parallel size r)rrrqr6rurhrtrsrv) rUrrmax_position_embeddingsrt hidden_sizenum_attention_headsrr@r6r) rgrrrqr6rurhriryrtrs layer_argsrrs rrmrmts6~ FM4 0 <#v'9'99 F$6$66v/6> 7 &G)G)G vz40< OO##v'A'AA v4d;C  & &++   7Q;7"a' 0 ;fgnfoopq ' !I,:,F.J*.4.@ell6*d 0 "%f.>E  ",!1!1!3EAq}1EEEs  D+"D+)rrchunked_attentionr) offloadedquanto_quantized hqq_quantizedrkceZdZdZddZy) SinkCachea Is its now a `custom_generate` repository on the Hub: https://huggingface.co/transformers-community/sink_cache. See [these docs](https://huggingface.co/docs/transformers/generation_strategies#custom-decoding-methods) for general `custom_generate`usage. Nc td)Nz`SinkCache` has been moved as a `custom_generate` repository on the Hub: https://huggingface.co/transformers-community/sink_cache. See the repository for usage examples.rrs rrzSinkCache.__init__s! o  rr:rr%rrr'r's  rr'ceZdZUdZded<dZedZdee e jffdZ de e effd Zd Zd Zd Zd Zy) CacheConfigzT Base class for cache configs. Deprecated in favor of a simpler dictionary. Ncache_implementationc.tjdyNZCacheConfig is deprecated and will be removed in v4.55.0 in favor of a simpler dictionary.)rrrs r __post_init__zCacheConfig.__post_init__s h rc tjd|di|}g}|jD]0\}}t||st ||||j |2|D]}|j |d|S)ar Constructs a CacheConfig instance from a dictionary of parameters. Args: config_dict (dict[str, Any]): Dictionary containing configuration parameters. **kwargs: Additional keyword arguments to override dictionary values. Returns: CacheConfig: Instance of CacheConfig constructed from the dictionary. r.Nr%)rrrrrZrr)ri config_dictrrg to_remover r[s r from_dictzCacheConfig.from_dicts  h #{#  ,,. &JCvs#U+  % & "C JJsD ! " rjson_file_pathct|dd5}|j}tj|dddz}|j |dddy#1swYyxYw) a Save this instance to a JSON file. Args: json_file_path (`str` or `os.PathLike`): Path to the JSON file in which this configuration instance's parameters will be saved. use_diff (`bool`, *optional*, defaults to `True`): If set to `True`, only the difference between the config instance and the default `QuantizationConfig()` is serialized to JSON file. wzutf-8)encodingrT)indent sort_keys N)opento_dictjsondumpswrite)rr4writerr1 json_strings r to_json_filezCacheConfig.to_json_filesU.# 8 &F,,.K**[dKdRK LL %  & & &s =AAr"c@tj|jS)z Serializes this instance to a Python dictionary. Returns: `dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance. )rdeepcopy__dict__rs rr<zCacheConfig.to_dicts }}T]]++rc#Ktj|jjD] \}}||f yw)zTallows `dict(obj)` for situations where obj may be a dict or QuantizationConfigMixinN)rrDrEr)rattrr[s rrbzCacheConfig.__iter__s9==7==? KD%+  s=?cT|jjd|jS)N )rr;to_json_stringrs rr}zCacheConfig.__repr__$s(..))*!D,?,?,A+BCCrcJtj|jddzS)z Serializes this instance to a JSON formatted string. Returns: str: JSON formatted string representing the configuration instance. r)r8r:)r=r>rErs rrJzCacheConfig.to_json_string's zz$--2T99rc g}|jD]0\}}t||st||||j|2|jDcic] \}}||vs ||}}}|Scc}}w)a Updates attributes of this class instance with attributes from `kwargs` if they match existing attributes, returning all the unused kwargs. Args: kwargs (`dict[str, Any]`): Dictionary of attributes to tentatively update this class. Returns: `dict[str, Any]`: Dictionary containing all the key-value pairs that were not used to update the instance. )rrrZr)rrr2r r[ unused_kwargss rr'zCacheConfig.update0s}  ,,. &JCtS!c5)  % & 7=lln] US\H\e] ]^s  A0&A0)r;r<r=r>__annotations__r/rmr3r rCosPathLikerBrBrr<rbr}rJr'r%rrr*r*sk 0&5bkk1A+B&$,c3h, D:rr*ceZdZdZddddddej dfded eed eed eed eed eedeejdeefdZ dZ y)QuantizedCacheConfiga4 Configuration class for quantized cache settings. Deprecated in favor of a simpler dictionary. Attributes: backend (`str`, *optional*, defaults to `"quanto"`): Backend to use when performing quantization, Can be one of [`quanto`, `HQQ`] nbits (`Optional[int]`, *optional*, defaults to 4): Number of bits, can be 2 or 4 for the `quanto` backend and one of [1, 2, 3, 4, 8] for the `HQQ` backend. Defaults to 2. axis_key (`int`, *optional*, defaults to 0): Axis over which to perform grouping for the key tensors. Can be [0, -1] for `quanto` backend and [0, 1] for `HQQ` backend. axis_value (`int`, *optional*, defaults to 0): Axis over which to perform grouping for the value tensors. Can be [0, -1] for `quanto` backend and [0, 1] for `HQQ` backend. q_group_size (`Optional[int]`, *optional*, defaults to 64): Size of the quantization group, should be a divisor of the model's hidden dimension. Defaults to 64. residual_length (`Optional[int]`, *optional*, defaults to 128): Length of the residual cache which will always be stored in original precision. Defaults to 128. compute_dtype (`torch.dtype`, *optional*, defaults to `torch.float16`): The default dtype used for computations in the model. Keys and Values will be cast to this dtype after dequantization. device (`str`, *optional*, defaults to `"cpu"`): Device on which to perform computations, should be same as the model's device. rrrrrrprrrrrrrr6c tjd||_||_||_||_||_||_||_||_ yr-) rrrrrrrrrr6) rrrrrrrrr6s rrzQuantizedCacheConfig.__init__asR  h     $(.* rcd}|jdvr't|jdd|j|jdkr't|jdd|j|jdkr't|jd d|j|j d vr't|jd d |j |j d vr't|jd d|j yrrrs rrzQuantizedCacheConfig.validatexrrN) r;r<r=r>r@r#rCr rErurrr%rrrRrRGs4  "#$%&(),/4}} %}3-  SM  sm "#  , .2rrRcReZdZdZdZd dedefdZdZdZded d fd Z ded d fd Z y )StaticCacheConfigz8 Configuration class for static cache settings. staticrrrqcXtjd||_||_||_yr-)rrrrrqr6)rrrrqr6s rrzStaticCacheConfig.__init__s+ h %* rct|j|kDry|jd}|j|r |jn |j }|j| }|j ||j|jf}|j ||j|jf}|j|r|n|}tj||j||} tj||j||} tjj| tjj| |jj| |j j| |j"|j|sg|_g|_t'dD]} tj||j|j} tj||j|j} tjj| tjj| |j"j| |j$j| yyy)zIOverridden to use the correct device if offloaded layer (and pin memory).Nr )rur6 pin_memoryrrx)rryrPrlr6rryrqrtrvr@rz_dtyper{r|rrdevice_key_cachedevice_value_cacherv)rrrrr6rZglobal_cache_shapesliding_cache_shape cache_shapenew_layer_key_cachenew_layer_value_cache_device_layer_key_cachedevice_layer_value_caches rinitialise_cache_layerz(StaticCacheConfig.initialise_cache_layers  t~~  * (..q1&*ooi&@""dFYFY33 "113FHZHZ\`\i\ij#224GI\I\^b^k^kl.2__Y-G)M_ #kk+T[[QWdno % Kt{{SYfp q ))*=> ))*?@ 12  56  (1K$&D !&(D #1X I).[ \f\m\m)n&+0;;{$++^h^o^o+p( 112EF 112GH%%,,-CD''../GH  I 2L (rc$|jBtjj|jj |j|j |j}|j|j}|j|||dddd|f<||dddd|f<|j|j|j|dddd|f<|j|j|j|dddd|f<||fSr$)_prefetch_streamr@rdefault_streamr6 wait_streamr\active_device_layerr]_prefetch_next_layerr8rryr)rr*rrr k_outv_outrqs r_static_updatez StaticCacheConfig._static_updates  , JJ % %j&7&7 8 D DTEZEZ [%%d&>&>?''(@(@A !!),'1aN"#&2aN"#;E--H[H[:\y!!Q"67r+rErrfrorlrrr%rrrVrVsW$3sIB,8c8d8(ICIDIrrVnamecx|dk(r!tjdGdd}|Stdtd|)N MambaCachezImporting `MambaCache` from `transformers.cache_utils` is deprecated and will be removed in a future version. Please import it from `transformers` or `transformers.models.mamba.cache_mamba` instead.c eZdZdZdZej dfdedejde eje dffdZ ded ejd ejd ejfd Zded ejfdZdZy)__getattr__..MambaCachea4 Importing `MambaCache` from `transformers.cache_utils` is deprecated and will be removed in a future version. Please import it from `transformers` or `transformers.models.mamba.cache_mamba` instead. Cache for mamba model which does not have attention mechanism and key value states. Arguments: config (`PretrainedConfig): The configuration file defining the shape-related attributes required to initialize the static cache. max_batch_size (`int`): The maximum batch size with which the model will be used. Note that a new instance must be instantiated if a smaller batch size is used. dtype (`torch.dtype`, *optional*, defaults to `torch.float16`): The default `dtype` to use when initializing the layer. device (`torch.device` or `str`, *optional*): The device on which the cache should be initialized. Should be the same as the layer. Example: ```python >>> from transformers import AutoTokenizer, MambaForCausalLM, MambaCache >>> model = MambaForCausalLM.from_pretrained("state-spaces/mamba-130m-hf") >>> tokenizer = AutoTokenizer.from_pretrained("state-spaces/mamba-130m-hf") >>> inputs = tokenizer(text="My name is Mamba", return_tensors="pt") >>> # Prepare a cache class and pass it to model's forward >>> past_key_values = MambaCache(config=model.config, max_batch_size=1, device=model.device, dtype=model.dtype) >>> outputs = model(**inputs, past_key_values=past_key_values, use_cache=True) >>> outputs.past_key_values MambaCache() ``` TNryrur6c||_||_|j|_|j|_|j |_g|_g|_|tj|nd}t|jD]}tj|j|j|j ||j}tj|j|j|j||j}tjj|tjj||jj!||jj!|y)Nr6ru)ryr[intermediate_size state_sizessm_state_size conv_kernelconv_kernel_size conv_states ssm_statesr@r6rvrrzr{r|r)rrgryrur6rc conv_state ssm_states rrz(__getattr__..MambaCache.__init__B s/'5## )/)A)A&&,&7&7#(.(:(:%79 68171Cf-v7786A/4{{++..--%"kk 0J/4kk++..++%"kk /IMM55jAMM55i@$$++J7OO**95'6rrnew_conv_stater*r"c"|j|j|jk7r5|j|j|j|j|<|j|}|jd|jdz }|j dd}|j|j|j |dddd|f<|j|j|j|xx|z cc<|j|S)Nrr rR)shiftsdimsr|)rr6r8rrrollrur1)rrrr*rs rupdate_conv_statez1__getattr__..MambaCache.update_conv_stateg s ##I.559N9NN262B2B92M2P2PQ_QfQf2gD$$Y/!--i8 !/!5!5a9N9NQR9R!S'__BR_@ 3A3D3DJL]L]eoeueu3D3v 1a/0  +113  +z9+'' 22r new_ssm_statec|j|j|j|j|<|j|Sr$)r8rr6)rrrs rupdate_ssm_statez0__getattr__..MambaCache.update_ssm_statex s9-:-=-=dooi>X>_>_-` *y11rctt|jD]<}|j|j|j|j>yr$)rvrrr1rr"s rr2z%__getattr__..MambaCache.reset| sM!&s4+;+;'r?r@r#rErur r6rCrrArFrrr2r%rrrxrz s D"N&+]]9= # 6!$# 6{{ # 6 ellC56 # 6J 3!$ 36;ll 3TYTdTd 3 3" 2# 2ell 2 7rzmodule z has no attribute )rrAttributeErrorr;)rvrxs r __getattr__r sL | | d 7d 7L 78,.@I JJr)NNNNNNN)XrrLimportlib.metadatar,rr=rOabcrrcollections.abcr dataclassesrtypingrrr r r@ packagingr transformers.pytorch_utilsr configuration_utilsrrrrrrhqq.core.quantizerr= get_loggerr;rrrHrorrrrrr&r;rDrArNrPrrrrContextrregister_pytree_noder<rregister_pytree_flatten_specrrrrrrrrrrrrrBrlrEr6rCrurmrrNrkr'r*rRrVrr%rrrsi  #$!11 I1dd;   H %%Kc%KPk?k\D$/D$N^$^$BS$,S$l<*<*~fonfoRv6nv6r;$$;;$|;!8;|eELL%,,6778 c5u||3445>!!@NND=5=DU#  |   $$,,  KK,,_ , 7 78,:O:O9PQ - HH11k B\B$E%E<#S;#SL(L(LV#V%#VN+* S- S>\>6&]>&]R'Z'ZTvS%vSr.(48H3I*J.TX.]bcgimcm]n..!%#'-1#':>!$(IF % &IF IFC=IF %,,T) * IF EKK IF tC$567 IFc]IFSMIF IFZ"+,7c4 1223 )//:T#t$4556     ee eP b;b bJ bI bI bINnKcnKcnKr