rh PddlZddlddlZddlZdZdZedfdZ ddZ ddZ e Z ddZ d ZeZdd Zd Zd ZeZdd ZeZddZGddZd dZGddZedfdZd!dZGddZdZeZdZe Z!d"dZ y)#N)*ctj|d}|j\}}tj||fd}tj||fd}t j }t ||_t ||_||_ ||_ |j|j|t ||j||fS)zPreturn k smallest values (and their indices) of the lines of a float32 arrayfloat32dtypeint64)npascontiguousarrayshapezerosfaissfloat_maxheap_array_tswig_ptridsvalnhkheapifyaddnreorderarrayrmnIDhas g/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/faiss/extra_wrappers.pykminr  i 8E ;;DAq !Qw'A !Qy)A  $ $ &B a[BF a[BF BE BDJJLGGAxJJL a4Kctj|d}|j\}}tj||fd}tj||fd}t j }t ||_t ||_||_ ||_ |j|j|t ||j||fS)zOreturn k largest values (and their indices) of the lines of a float32 arrayrrr)r r r r r float_minheap_array_trrrrrrrrrs rkmaxr$+r r!c tj|d}tj|d}|j\}}|j\}}||k(sJtj||fd}|tk(r-t ||t ||t |t ||S|tk(r||jz|dd|St||t ||t |||t ||S)zJcompute the whole pairwise distance matrix between two sets of vectorsrrN) r r r empty METRIC_L2pairwise_L2sqrrMETRIC_INNER_PRODUCTTpairwise_extra_distances) xqxbmetric metric_argnqdnbd2diss rpairwise_distancesr5=s  b 2B  b 2B HHEB XXFB 7N7 ((B89 -C  r8B<  SM  J ' 'bddA J ! r8B<  J SM   Jr!cttj|d}tt||j||SNrr)r r& float_randrsizerseedress rrandr=Vs+ ((1I &Cx}chh- Jr!ctj|d}|"tt||j||St t||j|||SNrr)r r& int64_randrr9int64_rand_max)rr;vmaxr<s rrandintrC\sO ((1G $C |8C=#((D1 J x}chhd; Jr!cttj|d}tt||j||Sr7)r r& float_randnrr9r:s rrandnrFhs+ ((1I &C sxx. Jr!c|jd}|jdk(rt|jt |S|j \}}t j|d}t||t |t ||S)z> compute a checksum for quick-and-dirty comparisons of arrays uint8uint64r) viewndim bvec_checksumr9rr r r bvecs_checksum)arr1css rchecksumrQnsg wAvv{QVVXa[11 77DAq !8 $B1a!hrl3 Ir!cftj||fd}t||t|||Sr7)r r&rand_smooth_vectors_cr)rr1r;r<s rrand_smooth_vectorsrTzs- ((Aq6 +C!Q t4 Jr!c Ztj|d}tj|d}|jd}|jd|k(sJ|jd|jd}}d}t|D]+}|t |t |||t ||z }-|S)z< size of intersection between each line of two result tablesrrrrI)r r r rangeranklist_intersection_sizer)I1I2rk1k2ninteris reval_intersectionr^s  b 0B  b 0B  A 88A;!   XXa["((1+B F 1X6, AXbe_6 66 Mr!cdt|jd|jdt|y)NrIr)fvec_renorm_L2r rxs r normalize_L2rcs"1771:qwwqz8A;7r!c tj|d}|t|jdz}tj|dzd}tj|j d}t |j tj|jd|tj|tj||||fS)aPerform a bucket sort on a table of integers. Parameters ---------- tab : array_like elements to sort, max value nbucket - 1 nbucket : integer number of buckets, None if unknown nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) perm : array_like perm[lims[i] : lims[i + 1]] contains the indices of bucket #i (size tab.size) rrrIrJ) r r intmaxr&r9 bucket_sort_cr rrK)tabnbucketntlimsperms r bucket_sortrms&  s' 2Ccggi!m$ 88GaKw /D 88CHHG ,D %..(!34%u~~d';  :r!c @|jdk(s|jdk(sJ|j\}}|t|jdz}t j |dzd}t ||tj||tj|||S)aPerform a bucket sort on a matrix, recording the original row of each element. Parameters ---------- tab : array_like array of size (N, ncol) that contains the bucket ids, maximum value nbucket - 1. On output, it the elements are shuffled such that the flat array tab.ravel()[lims[i] : lims[i + 1]] contains the row numbers of each bucket entry. nbucket : integer number of buckets (the maximum value in tab should be nbucket - 1) nt : integer number of threads to use (0 = use unthreaded codepath) Returns ------- lims : array_like cumulative sum of bucket sizes (size vmax + 1) int32rrIr) rr rerfr r&matrix_bucket_sort_inplace_cr r)rhrirjnrowncolrks rmatrix_bucket_sort_inplacerss, 99 399#77 7JD$cggi!m$ 88GaKw /D  dENN3'%  Kr!c*eZdZdZddZdZdZdZy) ResultHeapz_Accumulate query results from a sliced dataset. The final result will be in self.D, self.I.cztj||fd|_tj||fd|_||c|_|_|r t }n t}||_||_t|j|_ t|j|_ |j||_ y)z nq: number of query vectors, k: number of results per query keep_max: keep the top-k maximum values instead of the minima rrrN)r r rrr0rr#rrrrrrheaps)selfr0rkeep_maxrws r__init__zResultHeap.__init__s 2q'12q'3a )+E)+ETVV$ TVV$   r!c&|j\}}tj|d}tj|d}|j||fk(sJ||jk(sJ|jj |t |t ||y)z Add results for all heaps D, I should be of size (nh, nres) D, I do not need to be in a particular order (heap or sorted) rrrN)r r r r0rw addn_with_idsr)rxrrr0kds r add_resultzResultHeap.add_results B  ) 4  ' 2ww2r("""TWW}}    QK r!c |j\}}|t|k(sJ|jdk(r|j|jk(s!|jdk(r|j|fk(sJtj|d}tj|d}tj|d}|jdk(rdn|}|j j |t||t|t||y)z Add results for a subset of heaps. D, I should hold resutls for all the subset as a special case, if I is 1D, then all ids are assumed to be the same rIrrrrN)r lenrLr r rwaddn_query_subset_with_idsr)rxsubsetrrnsubsetr} id_strides radd_result_subsetzResultHeap.add_result_subsets gg #f+%%% FFaKAGGqww. FFaKAGGv-  .  ) 4  ' 2%%fG<1A"  -- Xf%  Xa[) r!c8|jjyN)rwr)rxs rfinalizezResultHeap.finalizes r!NF)__name__ __module__ __qualname____doc__rzr~rrr!rrurus(  *r!ruc l|j|jk(sJ|j\}}}tj||f|j}tj||f|j}|rtnt }||||t |t |t |t |||fS)z Merge a set of sorted knn-results obtained from different shards in a dataset Dall and Iall are of size (nshard, nq, k) each D[i, j] should be sorted returns D, I of size (nq, k) as the merged result set r)r r r&rmerge_knn_results_CMaxmerge_knn_results_CMinr) DallIallrynshardrrDnewInewfuncs rmerge_knn_resultsrs :: ## #::LFAq 88QF$** -D 88QF$** -D%- !3ID 1f :r!ceZdZdZdZdZy)MapInt64ToInt64c,ttj||_|d|jzk(sJd||_tj |dfd|_tj|jt|j y)Nrzneed power of 2 capacityrr) rer log2 log2_capacitycapacityr&rhr hashtable_int64_to_int64_initr)rxrs rrzzMapInt64ToInt64.__init__3ss !231 2 222N4NN2  88XqM9 ++D,>,>@RSr!c |j\}|j|fk(sJtj|jt |j |t |t |yr)r r hashtable_int64_to_int64_addrrrh)rxkeysvalsrs raddzMapInt64ToInt64.add:sR ZZzzaT!!! **    2 x~x~ /r!c |j\}tj|fd}tj|j t |j|t |t ||Sr?)r r r&r hashtable_int64_to_int64_lookuprrrh)rxrrrs rlookupzMapInt64ToInt64.lookupAsW ZZxxG, --    2 x~x~ / r!N)rrrrzrrrr!rrr1sT/r!rctj|d}tj|d}|j\}}|j\}}||k(sJtj||fd} tj||fd} |tk(r:t t |t |||||t | t | | | fS|tk(r:tt |t |||||t | t | | | fStt |t |||||||t | t |  | | fS)a Compute the k nearest neighbors of a vector without constructing an index Parameters ---------- xq : array_like Query vectors, shape (nq, d) where the dimension d is that same as xb `dtype` must be float32. xb : array_like Database vectors, shape (nb, d) where dimension d is the same as xq `dtype` must be float32. k : int Number of nearest neighbors. metric : MetricType, optional distance measure to use (either METRIC_L2 or METRIC_INNER_PRODUCT) Returns ------- D : array_like Distances of the nearest neighbors, shape (nq, k) I : array_like Labels of the nearest neighbors, shape (nq, k) rrr) r r r r&r' knn_L2sqrrr)knn_inner_productknn_extra_metrics) r,r-rr.r/r0r1r2r3rrs rknnrMs.2  b 2B  b 2B HHEB XXFB 7N7 "a(A "a *A  RL(2, r2q(1+x{ a4K ' ' RL(2, r2q(1+x{  a4K  RL(2, r2vz1 QK! a4Kr!c |j\}}|j\}}||k(sJtj||fd}tj||fd} |dk(rtj} || _|| _tj| | _tj|| _ tj| tj|tj|||d|| fS|dk(rltjtj|tj|||||tj|tj| || fSt)a Compute the k nearest neighbors of a set of vectors without constructing an index. Parameters ---------- xq : array_like Query vectors, shape (nq, d) where d is the number of bits / 8 `dtype` must be uint8. xb : array_like Database vectors, shape (nb, d) where d is the number of bits / 8 `dtype` must be uint8. k : int Number of nearest neighbors. variant : string Function variant to use, either "mc" (counter) or "hc" (heap) Returns ------- D : array_like Distances of the nearest neighbors, shape (nq, k) I : array_like Labels of the nearest neighbors, shape (nq, k) rorrhcrImc) r r r&r int_maxheap_array_trrrrrhammings_knn_hchammings_knn_mcNotImplementedError) r,r-rvariantr0r1r2r3rrheaps r knn_hammingrs-2 HHEB XXFB 7N7 "a(A "a(A$((*>>!$>>!$  %..$ennR&8" q  a4K D  NN2 r 2BAq NN1 u~~a0 a4K"!r!c2eZdZdZdZdZddZd dZdZy) KmeansaObject that performs k-means clustering and manages the centroids. The `Kmeans` class is essentially a wrapper around the C++ `Clustering` object. Parameters ---------- d : int dimension of the vectors to cluster k : int number of clusters gpu: bool or int, optional False: don't use GPU True: use all GPUs number: use this many GPUs progressive_dim_steps: use a progressive dimension clustering (with that number of steps) Subsequent parameters are fields of the Clustring object. The most important are: niter: int, optional clustering iterations nredo: int, optional redo clustering this many times and keep best verbose: bool, optional spherical: bool, optional do we want normalized centroids? int_centroids: bool, optional round centroids coordinates to integer seed: int, optional seed for the random number generator c t||_|j|d|_d|vrt|_nt |_|j D]S\}}|dk(r|dk(s|dk(r t}||_'t|j|t|j||U|jy)zd: input dimension, k: nb of centroids. Additional parameters are passed on the ClusteringParameters object, including niter=25, verbose=False, spherical = False Fprogressive_dim_stepsgpuTN) r1resetr"ProgressiveDimClusteringParameterscpClusteringParametersitems get_num_gpusgetattrsetattr set_index)rxr1rkwargsvs rrzzKmeans.__init__s  1  "f ,8:DG*,DGLLN 'DAqEz9R$A#A& ' r!c|j}|jjtk(ru|jjrt ||_nt||_|jr1tj|j |j|_yy|jrt|j}n t}||_ y)N)ngpu)r1r __class__r spherical IndexFlatIPindex IndexFlatL2rr index_cpu_to_all_gpusGpuProgressiveDimIndexFactoryProgressiveDimIndexFactoryfac)rxr1rs rrzKmeans.set_indexs FF 77   4 4ww  (^ (^ xx"88$((S xx3B02DHr!NcR|t||_d|_d|_d|_y)zg prepare k-means object to perform a new clustering, possibly with another number of centroids N)rer centroidsobjiteration_stats)rxrs rrz Kmeans.resets* =VDF#r!c tj|d}|j\}}||jk(sJ|jj t k(rt||j|j}|D|j\}}||k(sJtj|j|j|j||j|ng|J|J|jjrJt!||j|j}|j|t#||j$tj&|j} | j)|j||_ |j*} t-| j/D cgc]} | j1| } } tj2| D cgc]} | j4c} |_dj7} | D cgc]} | Dcic]}|t9| |c}c}} |_|j4j.dkDr|j4dSdScc} wcc} wcc}wcc}} w)a  Perform k-means clustering. On output of the function call: - the centroids are in the centroids field of size (`k`, `d`). - the objective value at each iteration is in the array obj (size `niter`) - detailed optimization statistics are in the array iteration_stats. Parameters ---------- x : array_like Training vectors, shape (n, d), `dtype` must be float32 and n should be larger than the number of clusters `k`. weights : array_like weight associated to each vector, shape `n` init_centroids : array_like initial set of centroids, shape (n, d) Returns ------- final_obj: float final optimization objective rrz,obj time time_search imbalance_factor nsplitrrr)r r r r1rrr Clusteringrr copy_array_to_vectorravelrtrainrrProgressiveDimClusteringrrvector_float_to_arrayreshaperrVr9atrrsplitr)rxrbweightsinit_centroidsrr1clusncr3rstatsr]st stat_fieldsfields rrz Kmeans.train s4  ) 4ww1DFF{{ 77   4 4a1D)'--BQww**>+?+?+A4>>R JJq$**g .? "?!) ))ww(( ((+Atvvtww?D JJq(1+txx 0//? "**46615$$&+EJJL&9:!::88e4RVV45DJJL  5@ @5UGB& & @  $xx}}q0txx|9c9;4 A s$(II# I-I( &I-(I-cRtj|d}|jJd|jj |jj |j|jj |d\}}|j|jfS)Nrrzshould train before assigningrI)r r rrrrsearchr)rxrbrrs rassignz Kmeans.assignHs  ) 4~~)J+JJ)  t~~&zz  A&1wwy!'')##r!r)NN) rrrrrzrrrrrr!rrrs"@. $::x$r!rcJt|tjjSr) isinstance collectionsabcSequenceras r is_sequencerUs a11 22r!c |j\}}tj|d}t|rtj|d}|j|fk(sJt |j dzdz}tj ||fd}t||t|t|t|||S||zdzdz}tj ||fd}t|||t|t|||S)a> Pack a set integers (i, j) where i=0:n and j=0:M into n bitstrings. Output is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. If nbit is an integer: all entries takes nbit bits. If nbit is an array: entry (i, j) takes nbit[j] bits. rorrH) r r r rresumr&pack_bitstrings_cr)rOnbitrM code_sizebs rpack_bitstringsrZs 77DAq Qg.A4##D8zzaT!!!aA-. HHa^7 3 q(4.(1+x{I G HX\a' HHa^7 3!Qhqk8A; J Hr!c |j\}}|tj|d}t|}t |j dzdz}||k\sJtj ||fd}t||t|t||t||S|}||zdzdz}||k\sJtj ||fd}t|||t||t||S)a Unpack a set integers (i, j) where i=0:n and j=0:M from n bitstrings (encoded as uint8s). Input is an uint8 array of size (n, code_size), where code_size is such that at most 7 bits per code are wasted. Two forms: - when called with (array, M, nbit): there are M entries of size nbit per row - when called with (array, nbits): element (i, j) is encoded in nbits[j] bits rorrr) r r r rrerr&unpack_bitstrings_cr)r M_or_nbitsrrrr min_code_sizerOs runpack_bitstringsrus77LAy |##Jg> ITXXZ!^12 M))) HHaV7 + q(4. QKHQK 1 H TA!+ M))) HHaV7 + q$ Y  = Hr!)90)rN)i)Nrr)rr)"numpyr faiss.loaderr collections.abcrrr$r'r5r=rClrandrFrQrTrSr^rcrmrgrsrprurrrrrrrrrrrr!rrs $$'0A2   , 8 > : N==@,8$3l0pS$S$t3$ 2( r!