rhVddlmZmZmZmZddlZddlmcmZ ddlmZddl m Z ddl m Z mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&e#r ddl'm(Z(ddl)m*Z*e$jVe,Z-GddedZ.Gddej^Z0Gddej^Z1Gddej^Z2Gddej^Z3Gdd ej^Z4d!Z5d?d"Z6d#ejnd$e8d%ejnfd&Z9 d@d'ej^d(ejnd)ejnd*ejnd+eejnd,e:d-e:fd.Z;Gd/d0ej^Z<Gd1d2eZ=e"Gd3d4eZ>Gd5d6ej^Z?e"Gd7d8e>Z@ dAd9eejneAejndfd:ee8d+eejnd%eejne8ffd;ZBGd<d=e>eZCgd>ZDy)B)CallableOptional TypedDictUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringis_torch_flex_attn_availablelogging)GraniteMoeSharedConfig) BlockMask)make_flex_block_causal_maskceZdZUdZej ed<ej ed<eed<eed<ejed<y)GraniteFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__int IntTensor/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/granitemoeshared/modeling_granitemoeshared.pyrr2s7"###### __r/rF)totalc`eZdZdZdeffd ZdejdejfdZxZ S)GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configc`t||j|_|j|_t |j |_tj|j|jdzd|_ tj|j|jd|_ y)NFbias) super__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr4 __class__s r0r:zGraniteMoeSharedMLP.__init__Ts  ,,!:: !2!23IIdoot7G7G!7KRWXYYt'7'7uUr/ hidden_statesreturnc|j|}|jdd}|j|d|dz}|j|}|S)Nr6dimrr)rAchunkr?rB)rDrFchunked_hidden_statess r0forwardzGraniteMoeSharedMLP.forward]s^))-8 - 3 3A2 3 >(=a(@ADYZ[D\\ **=9 r/) r%r&r'r(rr:r)TensorrN __classcell__rEs@r0r3r3Ks2V5VU\\ellr/r3c,eZdZdfd ZdZdZxZS)GraniteMoeSharedRMSNormct|tjt j ||_||_y)zF GraniteMoeSharedRMSNorm is equivalent to T5LayerNorm N)r9r:r Parameterr)onesweightvariance_epsilon)rDr;epsrEs r0r:z GraniteMoeSharedRMSNorm.__init__fs1 ll5::k#:; #r/c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)Nr6rIT)keepdim) dtypetor)float32powmeanrsqrtrXrW)rDrF input_dtypevariances r0rNzGraniteMoeSharedRMSNorm.forwardnsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r/c^t|jjd|jS)Nz, eps=)tuplerWshaperXrDs r0 extra_reprz"GraniteMoeSharedRMSNorm.extra_reprus* ))*+6$2G2G1HIIr/)gư>)r%r&r'r:rNrhrPrQs@r0rSrSes$;Jr/rSc6eZdZdedededdffd ZdZxZS)GraniteMoeSharedParallelExperts num_expertsr< output_sizerGNct|tjt j ||||_||_||_||_ y)a Initialize the GraniteMoeSharedParallelExperts module. The experts weights are stored in [num_experts, output_size, input_size] format. Such that it's compatible with many MoE libraries, such as [Megablock](https://github.com/databricks/megablocks) and [ScatterMoE](https://github.com/shawntan/scattermoe), as well as the [MoE kernel](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/fused_moe.py) used in vllm. Args: num_experts (int): Number of experts. input_size (int): Size of the input. output_size (int): Size of the output. N) r9r:rrUr)emptyrWrkr<rl)rDrkr<rlrEs r0r:z(GraniteMoeSharedParallelExperts.__init__zsD" ll5;;{K#TU &$&r/c|j|d}g}t|jD]7}|jt j |||j |9tj|d}|S)a Forward pass of the GraniteMoeSharedParallelExperts module. Args: inputs (Tensor): Input tensor. expert_size: Expert size information. Returns: Tensor: Output tensor. rrJ) splitrangerkappendFlinearrWr)cat)rDinputs expert_size input_list output_listiresultss r0rNz'GraniteMoeSharedParallelExperts.forwardsq\\+1\5  t''( HA   qxx 1 t{{1~F G H))KQ/r/r%r&r'r,r:rNrPrQs@r0rjrjys)'C'S's't'.r/rjc2eZdZdededeffd ZdZxZS)GraniteMoeSharedTopKGatingr<rktop_kct|||_||_||_t j ||d|_y)a Initialize the top-k gating mechanism. Args: input_size (`int`): Size of the input. num_experts (`int`): Number of experts. top_k (`int`): Number of top experts to select. Fr7N)r9r:rkr<rrr@layer)rDr<rkrrEs r0r:z#GraniteMoeSharedTopKGating.__init__s: &$ YYz;UC r/c|j|j}|j|jd\}}t j |dj |}t j|jd|jg|j|j}|jd|d}|jjd}|j}|j!} | j#d\} } | j%|jd} |j!}|| } | | | ||fS)NrrJrr\devicetrunc) rounding_mode)rfloattopkrr)softmaxtype_aszerossizerkr\rscatterlongsumtolistflattensortdiv)rDrFlogits top_k_logits top_k_indices top_k_gatesrgatesrw top_k_experts_index_sorted_experts batch_index batch_gatess r0rNz"GraniteMoeSharedTopKGating.forwards.M*002&,kk$**!k&D# mmmLa8@@O    a $"2"2 3;;L;LU`UgUg  a2jjl&&q) "((* &--/ "/"4"4Q"7 *..tzz.Q "))+ !"67 #[+{FRRr/r|rQs@r0r~r~s'D3DSDD&Sr/r~c.eZdZdZdeffd ZdZxZS)GraniteMoeSharedMoEz A Sparsely gated mixture of experts layer with 1-layer Feed-Forward networks as experts. Args: config: Configuration object with model hyperparameters. r4ct||j|_|j|_t |j |_t|j|j|jdz|_ t|j|j|j|_ t|j|j|j|_y)Nr6)r<rkr)r9r:r;r<intermediate_sizer r>r?rjnum_local_expertsrArBr~num_experts_per_tokrouterrCs r0r:zGraniteMoeSharedMoE.__init__s  ,,!33 !2!23;  $ $doot7G7G!7K =  $ $d&6&6 100,,  r/c|j\}}}|jd|}|j|\}}}}} ||} |j| |} | j dd} |j | d| dz} |j | |} | |dddfz} tj||z|jf| j| j}|jd|| }|j|||j}|| fS)a Forward pass of the mixture of experts layer. Args: layer_input (Tensor): Input tensor. Returns: Tensor: Output tensor. Tensor: Router logits. rIr6rJrrNr)rreshaperrArLr?rBr)rr<r\r index_addview)rD layer_inputbszlengthemb_sizerrrrw router_logits expert_inputsrFrMexpert_outputsr layer_outputs r0rNzGraniteMoeSharedMoE.forwards!, 0 0 2VX!))"h7 BF++kBZ?; [-#K0 ))-E - 3 3A2 3 >(=a(@ADYZ[D\\ ++M;G'+ag*>> S6\4??;>CWCW`n`u`uvq+~F #((fdooF ]**r/)r%r&r'r(rr:rNrPrQs@r0rrs 5 &+r/rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..NrIr6rJ)rfr)ru)xx1x2s r0 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r/c|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkcossin position_ids unsqueeze_dimq_embedk_embeds r0apply_rotary_pos_embrsY( -- &C -- &C3w;q>C/0G3w;q>C/0G G r/rFn_reprGc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rfexpandr)rFrbatchnum_key_value_headsslenhead_dims r0 repeat_kvr4so 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr/modulequerykeyvalueattention_maskscalingdropoutc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr6rrI)rKr\)ptrainingr)rnum_key_value_groupsr)matmul transposerfr functionalrr^r]r\rr contiguous) rrrrrrrkwargs key_states value_states attn_weights causal_mask attn_outputs r0eager_attention_forwardr@s3 ; ;Q5RR S0)r/rc|eZdZdedeffd Z ddejdeejdeejdee dee d ee d eejd ee d ee ejejfd e ede ejee ejejfffdZxZS)GraniteMoeSharedDecoderLayerr4rct||j|_t|||_|j dkDrt ||_t|j|j|_ t|j|j|_ |j|_ |jdk(rd|_yt||_y)N)r4rrrY)r9r:r;r self_attnrrblock_sparse_moerS rms_norm_epsinput_layernormpost_attention_layernormresidual_multiplierr=r3 shared_mlprs r0r:z%GraniteMoeSharedDecoderLayer.__init__s !--2&IV  # #a '$7$?D !6v7I7IvObObc(?@R@RX^XkXk(l%#)#=#= "("A"AQ"F$L_`fLgr/rFrrroutput_attentionsrroutput_router_logitsrrrGc j|} |j|}|jd|||||||| d| \}} | ||jzz}|} |j|}|j |\} }|j | }n| |j |z}| ||jzz}|f}|r|| fz }|r||fz }|S)aC Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. )rFrrrr rrrr.)rrr r rr )rDrFrrrr rrr rrresidualself_attn_weightsmoe_hidden_statesroutputss r0rNz$GraniteMoeSharedDecoderLayer.forwardsL!,,]; ,:4>> , ')%)/) 3 ,  , ( (!=43K3K#KK !55mD +/+@+@+O(= ?? "-M- 0NNM =43K3K#KK  "  )+ +G   ' 'Gr/)NNNFFNFN)r%r&r'rr,r:r)rOrr*r rrerr FloatTensorrNrPrQs@r0rrs, h5 h# h 2637*.,1$)59/4KOM||M!.Mu//0 M ! M $D> MD>M!!1!12M'tnM&eELL%,,,F&GHM45M u  (51B1BEDUDU1U+V"WW XMr/rcJeZdZUeed<dZdZdgZdgZdZ dZ dZ fdZ xZ S)GraniteMoeSharedPreTrainedModelr4modelTrpast_key_valuesFct||t|tr<|jj j d|jjyy)Nr)r`std) r9 _init_weights isinstancerjrWdatanormal_r4initializer_range)rDrrEs r0rz-GraniteMoeSharedPreTrainedModel._init_weightssG f% f= > MM   & &CT[[5R5R & S ?r/)r%r&r'rr+base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraphrrPrQs@r0rrsD ""&*#78#4"5N"TTr/rc^eZdZddeffd Zej edZxZ S)GraniteMoeSharedRotaryEmbeddingr4ct|t|drUt|jt r;|jj d|jj d|_nd|_|j|_ |j|_ ||_ t|j|_ |j|j|\}|_|jd|d|j |_y)N rope_scaling rope_typetypedefaultinv_freqF) persistent)r9r:hasattrrr)dictgetr*max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr4r rope_init_fnattention_scalingregister_bufferr-original_inv_freq)rDr4rr-rEs r0r:z(GraniteMoeSharedRotaryEmbedding.__init__$s  6> *z&:M:Mt/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.r) r%r&r'rr:r)no_gradrrNrPrQs@r0r'r'#s4/5/"U]]_ < ,>FDWDWX &+#$*$?$?!!--33((DNN: '-'E'E$ ++'-'E'E$EIEaEaekEk9&Aqu ! osF input_idsrrr inputs_embedsrr output_hidden_statesr  return_dictrrGc ^||n|jj}||n|jj}||n|jj}| | n|jj} |du|duz r t d|j r%|jr|rtjdd}||j|}||jz}t|tdtfs t d|r | t}| F||j!nd} t#j$| | |j&dz|j(} || j+d}|j-||| ||}|}d}|j.|j/||}|rdnd}|rdnd}| rdnd}|j0D]7}|r||fz }|||||||| | | }|d}|r ||dfz }| s/||d fz }9|j3|}|r||fz }| st5d ||||fDSt7||||| S) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzBThe `past_key_values` should be either a `Cache` object or `None`.rrrr.)rrrr rrr rrIc3&K|] }|| ywrr.).0vs r0 z0GraniteMoeSharedModel.forward..sbcbos)last_hidden_staterrF attentionsr)r4r rZruse_return_dictrrRrrrrMrSrr+r r get_seq_lengthr)arangerfrr_update_causal_maskrVrPrQrer)rDrXrrrrYrr rZr r[rrpast_seen_tokensrrFrall_hidden_statesall_self_attnsall_router_logits decoder_layer layer_outputss r0rNzGraniteMoeSharedModel.forward`s 2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?ab b 0*nO  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L.. M>?L] & " ?? &"&//-"N #7BD0d"6BD![[ :M#!m%55!)*)."3#-%9$7 M*!,M =#3"55#!mB&7%99!- :0 -0  -!1 1 )?T]n%>>*'7 MM  ""&,,Q/ !+??AMnell;$$R(%7!; PP +')#))!, Q  KK , , 6*%%**.DD%  E*..I0CCKQZ[Kr/rurvr\rwc ||jdk(r|}|Stj|j}tj||f|||j }|dk7rtj |d}|tj||j |jddkDz}|ddddddfj|ddd}||j}|jd} |ddddddd| f|ddddddfj|j z} | dk(} |ddddddd| fj| ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer\rr)diagonalr]rIr)rKr)rrfullrtriurfrrclonerfr] masked_fill) rrurvr\rrwrrr mask_length padding_masks r0rzKGraniteMoeSharedModel._prepare_4d_causal_attention_mask_with_cache_positions<  %.*<*<*>!*C(K*' E*..I** -0Ye\j\q\qK!##jjqA 5<< n>S>STWeWmWmnprsWtt tK%dD!Q&67>>z1bRTUK))//1 ,2226 *1aL[L+@ANSTVZ\`bcScDdDgDg&&E   ,q0 5@Aq,;,AV5W5c5c )6 Aq!\k\12r/) NNNNNNNNNNN)F)r%r&r'rr:rrr)r*rOrr listrrrerrNrg staticmethodr,r\rrPrQs@r0rFrFEs52151537KO59$(,0/3/3&*59l E,,-l !.l u//0 l "%tE4E4E/F(F"GH l   1 12 l D>l $D>l 'tnl 'tnl d^l !!1!12l  u-- .l l h#( BellK78BllB B  B  BH4 444{{ 4  4  44r/rF gate_logitsrkc|t|tsyt|trC|dj}tj|Dcgc]}|j |c}d}tj jjd}tj||d\}} tj jj| |} |>tj| jd} tj|d} n1|j\} }|jd| |zz}|dddddddfj|| |||fjd||j }tj | j|zdtj |dz } |ddddddfj|| ||jdfjd|jdj |}tj ||zdtj |dz } |jdt#|jj$z}tj | dd|||jdzf| j'dz}||zScc}w)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. NrrJrIr)rrerr)rur]rrrrone_hotr`rrfrrrr,indexr)rrkrrcompute_device layer_gateconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expertrwrurOexpert_attention_mask router_per_expert_attention_maskrank overall_losss r0load_balancing_loss_funcrKs:*[%"@+u%$Q..#(99^i-jPZjmmN.K-jpq#r hh))112JPR1SO**_eDA((%%--.> LK!JJ{'8'8':B"'O!C&4&:&:# O4::1=*B^_ 4AtT1 2 V& OUKX Y WR , R  "IIk&7&7&9>> from transformers import AutoTokenizer, GraniteMoeSharedForCausalLM >>> model = GraniteMoeSharedForCausalLM.from_pretrained("ibm/PowerMoE-3b") >>> tokenizer = AutoTokenizer.from_pretrained("ibm/PowerMoE-3b") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rXrrrrYrr rZr r[rrrKrIr)lossaux_lossrrrFrcrr.)r4r r rZrdrrr,slicerlogits_scalingr loss_functionrKrrrkrrr]rrrrFrc)rDrXrrrrYrrr rZr r[rrrrrF slice_indicesrrroutputs r0rNz#GraniteMoeSharedForCausalLM.forwards*P2C1N-TXT_T_TqTq$8$D $++JjJj %9$D $++JjJj &1%SV8W~ot4]k mA}a,?@A$++444  \\^F%4%% ;;11 D /)4%%'"+  (( H !11HKK 4LLLY,F#"v-'+'7D7V# CV C(#33!//))!//  r/) NNNNNNNNNNNNr)r%r&r'_tied_weights_keysrr:rrrrr)r*rOrr rrrr,rerrNrPrQs@r0rrs*+ 5 151537KO59-1$(,0/3/3&*5934k E,,-k !.k u//0 k "%tE4E4E/F(F"GH k   1 12 k ))*k D>k $D>k 'tnk 'tnk d^k !!1!12k c5<</0k u// 0!k k r/r)rrFr)Nr)r)Nr6N)Etypingrrrrr)torch.nn.functionalrrrs activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersrmodeling_outputsrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrconfiguration_granitemoesharedr!torch.nn.attention.flex_attentionrintegrations.flex_attentionr get_loggerr%rrModuler3rSrjr~rrrrOr,rrrrrrr'rFrerr__all__r.r/r0rsV,87 !.)>9jjKF&JJB !;J   H %)52"))4JbiiJ(*bii*Z-S-S`9+"))9+x(6 UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%:S) S)l[#=[|ToTT"?R&#R&U\\* R&  5<<  R&jB "A?B J fr/