rh1dZddlZddlZddlZddlmZ ddlZdZ ddZ dZ Gdd ZGd d eZdd Z dd ZGddeZdZdZddZ ddZy#e $r e dYSwxYw)zO This contrib module contains a few routines useful to do clustering variants. N) ThreadPoolz2scipy not accessible, Python k-means will not workcyN)argkwargss k/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/faiss/contrib/clustering.py print_nopr sc |jd}|jdd}|rtnt}|d|jd|d||dt j ||f|dd |} | j || jg} || j} |d tj} | j|\} }tj|| }|d tj| z d dt|dt||j}~ |s*tj |dz|z|z}|dd|ddz }natj"|}||z|dz}|ddxxx|ddzccct%||k(sJ|dt|dt|d}g}tj} t'|D]}t)||}|dtj| z d d|d|d|d dd|||z}|||}tj*|||k(sJt j ||fi|} ||}| j || j-| j|j-| j~ |}|d tj| z d dtj.|| fS)a= perform 2-level clustering on a training set xt nc1 and nc2 are the number of clusters at each level, the final number of clusters is nc2. Additional arguments are passed to the Kmeans object. Rebalance allocates the number of sub-clusters depending on the number of first-level assignment. verboseFz2-level clustering of z nb 1st level clusters = z total zperform coarse trainingi)nitermax_points_per_centroidzassigning the training set minlengthzdone in z.2fz s. Sizes of clusters -Nznb 2nd-level centroids r[z s] training sub-cluster /z nc2= Tendflushz s)shapegetprintr faissKmeanstrainiteration_stats centroidstimeassignnpbincountminmaxargsortarangecumsumsumrangeintallappendvstack)xtnc1nc2 rebalanceclustering_niterargsdrlogkmr" centroids1t0_assign1bcoccall_nc2bc_sumi0c2c1i1subsetxtsubs r two_level_clusteringrKs  Ahhy%(G% C  *CC5PSuUV!"  3 & $   B HHRL))*OEJ$% B2JAw W ,B(499;#C((>s2wiqR RSA  YYsQw # % ,QR&2cr7"23,&*, ws|# 7|s""" %c'l^1S\N CD B B BCj '"+ a b %%>rd!C5cURT U[]eij "R&[2bvvgfo+,,, \\!S )D )6  r112 ",,   (499;#C(+, 99R=/ ))r c tj|}t|tjrt |j j D]?}|j j|}|j||j|}At|j|fi|d|_ yt|tjsJ|jtjk(sJt!t#j$|j&}t)d|t+|||j&fi|\}}|j,j||j,j/||j|y)zJ Applies 2-level clustering to an index_ivf embedded in an index. TNz REBALANCE=)rdowncast_index isinstanceIndexPreTransformr.chainsizeatr!applytrain_ivf_index_with_2levelindex is_trainedIndexIVF metric_type METRIC_L2r/r&sqrtnlistrrK quantizeradd)rUr3r8ivtr4r#r>s r rTrT_s&   'E%001u{{'')* A"B HHRL"B  $EKK.handle_query_blocksa!c'?1q3w1q3w  XXhnnQ&7&;&;A&>?N%a!c'2Nq"c" (A+1q3w'!!a#g. FB Avq q F{!$x!|t !$xt  (r ) rr&emptyfillinfonesr/r-listmapr.r) rrrrrrntrrpoolrrrs `````` @@@r sparse_assign_to_dense_blocksrs !B !B 9%AFF266N 3 A!G==#((. Qw"'R3Y S#U1b#%6 78 a4K"~ #U1b#%67 a4Kr c*eZdZdZdZdZdZddZy)DatasetAssignSparserbc|jtjjk(sJ||_t j |jdjd|_ y)Nrr ) __class__scipysparse csr_matrixrir&rrr- squared_normsrjs r rlzDatasetAssignSparse.__init__sG{{ell55555XXaggajnnQ&78r cbtj|j|jSr)r&rritodenservs r rxzDatasetAssignSparse.get_subsets"xxw//122r cFt|j||jS)N)r)rrirr{s r r|z"DatasetAssignSparse.perform_searchs , FFI(:(:< ??r c|tj}|j\}}d}t|}tj|dk(d}t |dk(ry|rddl}|j|d} ntj|d} | dddxxdz cc<| dddxxdzcc<t |dkDr|jddz } d| | dk<| | jz} | dkDj} t| |j} |j|| | } t|d| | D]:\}}||}|| z||<|| z ||<||dz||<||xx||zcc<|dz }<|| d}t |dkDr|S)z/ reassign centroids when some of them collapse NrrgP?r float)rQp)r&randomrrwhererr ones_likeastyper-r(rQchoicezip)hassignr#rsrr9nsplitis_torch empty_centsrfacprobasnnznreplacecjscicjcs r reassign_centroidsr)s z YY ??DAq Fi(H((7a<(+K ;1ooil+ll9Q<(!H H1II k Q (1,vz&**,z sK,,-iiFi3+ix0#6 FB" AGIbMGIbM!"+*GBK BK72; &K aKF "(), ) k Q , Mr c |j|j}}|rtnt} | d|||||fztj j |} tdtj} | j||d} |j| } t| }g}| dd}g}t|D]}tj}| ddd |j| \}}}| d dd |tj|z z }|j}|r|j}|j|t j || }|j#d dj%d}d||dk(<|r.ddl}|j)|j+|j,}||z } t/|| | }|tj| z |t1|||d}| d||d|d||d|fz|j||h| d||rddl}|j3| |t j2|| |r| |fS| S)a0Pure python kmeans implementation. Follows the Faiss C++ version quite closely, but takes a DatasetAssign instead of a training data matrix. Also redo is not implemented. For the torch implementation, the centroids are tensors (possibly on GPU), but the indices remain numpy on CPU. zAClustering %d points in %dD to %d clusters, %d iterations seed %dz preproc...F)rQreplacez doner assigningrTrzcompute centroidsrrr reN)objr$ time_searchrrzM Iteration %d (%.2f s, search %.2f s): objective=%g imbalance=%.3f nsplit=%dr$rrzstoring centroids in)rprtrr r&r RandomStater$rrxrr.rr-itemr1r'reshaperr from_numpytodevicerrsave)rdatarseed checkpointr return_statsrr9r:rr=permr#rr" t_search_totrr^t0sr%rsumserrrrrrss r kmeansrZsa ::<qA% C $()1a'= >?   t $B , B 99QQ9 .D%Ii(HOML C 5\1/iik KT...34 T6 c)) eeg ((*C 3++f2oob!$++I6C1H  ""3'**4;;7C3J #GY;YY[2%' 0F ;   5ai=!1,-9  q!  ! & 3 9j1 I.c1/f/))r )T)NN)NN@rNr)riNTF)rnumpyr&rr$multiprocessing.poolr scipy.sparser ImportErrorrr rKrTrarrrrrrrrrr r r s  +@ D*NF&&D,},($HL-`&-&DJ @-bCGRQ @ >?@sA A.-A.