rh R dZddlZddlmZmZddlZddlmZm Z m Z ddl m Z ddl mZddlmZmZdd lmZmZmZmZmZdd lmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*dd l+m,Z,dd l-m.Z.ddl/m0Z0m1Z1ddl2m3Z3m4Z4m5Z5dZ6dZ7dZ8e*ddgddgdgddgddddddZ9e*dgdgddgddge)hddgdddddddZ:e*ddgddgddgddgdgddddd d!d"Z;e*dgdgddge)d#d$hdgddgd%ddddd&d'Z<e*ddgddgddgee=ddge)hd(dgddge(edd)he)d*hgd+ddd)d,dd*d-d.Z>e*dgdgddgd/ddd0d1Z?e*ddgddgdgddgdddddd2Z@e*ddgddgddgee=ddge)hd(dgddge(ed3d4hd5e)d*hgd+ddd)d,dd*d-d6ZAe*ddgddge'ed3dd78gddgee=ddge)hd(dgddge(ed3d4hd5e)d*hgd9ddd)d,dd*d-d:ZB d^d;ZCd<ZDd=ZEe*ddgddge'ed3dd78gddgee=ddge)hd(dgeFeGeHgddge(ed3d4hd5e)d*hgd> dd4dd)dd?dd*d@dAZIe*ddgddgddgddgde&e)dBhge(ed4ejheKgdCddddBejdDdEZLe*ddgddgddgee=ddge)hd(dgddge(ed3d4hd5e)d*hgd+ddd)d,dd*d-dFZMe*ddgddgddgee=ddge)hd(dgddge(ed3d4hd5e)d*hgd+ddd)d,dd*d-dGZNe*dgdgddgdgdHddd dIdJZOe*ddgddgddgddgddge'edddK8gdge(ed3d4hd5e)d*hgdLdddddd d*dMdNZPe*ddgddgddgd/ddd0dOZQe*dgdgdgddgddgdPdddddQdRZRe*dgdgddgddgdSddddTdUZSdVZTe*dgdgddgee=ddgddgde)dWhgdXdddddWdYdZZUe*dgdgddgddgd[dddd\d]ZVy)_a Metrics to assess performance on classification task given class prediction. Functions named as ``*_score`` return a scalar value to maximize: the higher the better. Function named as ``*_error`` or ``*_loss`` return a scalar value to minimize: the lower the better. N)IntegralReal) coo_matrix csr_matrixissparse)xlogy)UndefinedMetricWarning)LabelBinarizer LabelEncoder)assert_all_finite check_arraycheck_consistent_length check_scalar column_or_1d) _average _bincount_count_nonzero_find_matching_floating_dtype_is_numpy_namespace_max_precision_float_dtype _searchsorted_tolist_union1d get_namespaceget_namespace_and_devicexpx)HiddenIntervalOptions StrOptionsvalidate_params) attach_unique) _nanaverage)type_of_target unique_labels)_check_pos_label_consistency_check_sample_weight _num_samplesct|tr|dk(rtjdSt|tt fr|dvrtj|Stj S)Nwarn)r) isinstancestrnpfloat64intfloatnan zero_divisions r/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sklearn/metrics/_classification.py_check_zero_divisionr8<sN-%-6*Azz# MC< 0]f5Lzz-((vv c t||\}}t||t|d}t|d}||h}|ddhk(rdh}t|dkDrt dj |||j }|dvrt d j ||d vrLt||\}}t|}t|}|dk(r" t|||}|jddkDrd}|jdr#t|rt|}t|}d}|||fS#t$r6}td |j|d |j|d |d}~wwxYw)a5Check that y_true and y_pred belong to the same classification task. This converts multiclass or binary types to a common shape, and raises a ValueError for a mix of multilabel and multiclass targets, a mix of multilabel formats, for the presence of continuous-valued or multioutput targets, or for targets of different lengths. Column vectors are squeezed to 1d, while multilabel formats are returned as CSR sparse label indicators. Parameters ---------- y_true : array-like y_pred : array-like Returns ------- type_true : one of {'multilabel-indicator', 'multiclass', 'binary'} The type of the true target data, as output by ``utils.multiclass.type_of_target``. y_true : array or indicator matrix y_pred : array or indicator matrix y_true input_namey_predbinary multiclassr-z@Classification metrics can't handle a mix of {0} and {1} targetsr?r@multilabel-indicator{0} is not supportedr?r@zCLabels in y_true and y_pred should be of the same type. Got y_true=z and y_pred=z[. Make sure that the predictions provided by the classifier coincides with the true labels.Nrr multilabelrB)rrr%len ValueErrorformatpoprr TypeErroruniqueshape startswithrr) r;r>xp_ type_true type_predy_type unique_valueses r7_check_targetsrUEs6 && )EBFF+v(;Iv(;I #F (L)) 6{Q N U U9   ZZ\FEE/66v>?? ))ff-Af%f% X   ( < ""1%)% & r "  'F'F' 66 !!3   ""$))F"3!45 ii/01''   s D$$ E#-1EE#ct|dtjtjtjg}|j dkDrt d|j |jdkrt d|jt|||t}||j|}tj|j|k(s9tjd|d |jjd t tj"||js:t%|t%|z }t d |d |d |j|}t'|jdk(rM|'t dj)|jdt dj)|j|j+|}|j,ddk(rtj.d|z |d}|j0dk(r|ddtj2f}|j,ddk(rtj.d|z |d}tj4|j6j8}|j;d}tj<|dtj>|stjdt t|}t'|j|j,dk7rf|@t dj)|j,d|j,d|jt dj)|j||fS)aConvert y_true and y_prob to shape (n_samples, n_classes) 1. Verify that y_true, y_prob, and sample_weights have the same first dim 2. Ensure 2 or more classes in y_true i.e. valid classification task. The classes are provided by the labels argument, or inferred using y_true. When inferring y_true is assumed binary if it has shape (n_samples, ). 3. Validate y_true, and y_prob have the same number of classes. Convert to shape (n_samples, n_classes) Parameters ---------- y_true : array-like or label indicator matrix Ground truth (correct) labels for n_samples samples. y_prob : array-like of float, shape=(n_samples, n_classes) or (n_samples,) Predicted probabilities, as returned by a classifier's predict_proba method. If `y_prob.shape = (n_samples,)` the probabilities provided are assumed to be that of the positive class. The labels in `y_prob` are assumed to be ordered lexicographically, as done by :class:`preprocessing.LabelBinarizer`. sample_weight : array-like of shape (n_samples,), default=None Sample weights. labels : array-like, default=None If not provided, labels will be inferred from y_true. If `labels` is `None` and `y_prob` has shape `(n_samples,)` the labels are assumed to be binary and are inferred from `y_true`. Returns ------- transformed_labels : array of shape (n_samples, n_classes) y_prob : array of shape (n_samples, n_classes) F ensure_2ddtyper-'y_prob contains values greater than 1: rz%y_prob contains values lower than 0: NzLabels passed were zZ. But this function assumes labels are ordered lexicographically. Pass the ordered labels=zC and ensure that the columns of y_prob correspond to this ordering.zy_true contains values z$ not belonging to the passed labels .zy_true contains only one label ({0}). Please provide the list of all expected class labels explicitly through the labels argument.z?The labels array needs to contain at least two labels, got {0}.axis)rtolzEThe y_prob values do not sum to one. Make sure to pass probabilities.zy_true and y_prob contain different number of classes: {0} vs {1}. Please provide the true labels explicitly through the labels argument. Classes found in y_true: {2}z^The number of classes in labels is different from that in y_prob. Classes found in labels: {0}) rr0r1float32float16maxrGminrr fitallclasses_warningsr+tolist UserWarningisinsetrFrH transformrLappendndimnewaxisfinforYepssumallclosesqrt) r;y_prob sample_weightlabelslbundeclared_labelstransformed_labelsrp y_prob_sums r7-_validate_multiclass_probabilistic_predictionr{s$N% BJJ 'KFzz|aB6::<.QRR zz|a@OPPFFM:  B  VVF^vvbkkV+, MM%fX.++-;;+=+=+?*@AEE  wwvv&**, #F c&k 9 )*;)<=((.xq2  VVF^ 2;;1 >##)6"++a.#9  ##)6"++#6  f-"a'YY " "$6Q  {{a2:: & ||A!1v:vA6 ((6<< $ $C#J ;;z12773< 8 S  %%78 2;;6<<?* >%f&,,Q/a"++  $fR[[1  v %%r9z array-likez sparse matrixboolean)r;r> normalizeruT)prefer_skip_nested_validationr}ruc(t|||\}}}t||\}}t||\}}}t||||j dr)t ||z ||d}|j |dk(|} n||k(} tt| ||S)aAccuracy classification score. In multilabel classification, this function computes subset accuracy: the set of labels predicted for a sample must *exactly* match the corresponding set of labels in y_true. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) labels. y_pred : 1d array-like, or label indicator array / sparse matrix Predicted labels, as returned by a classifier. normalize : bool, default=True If ``False``, return the number of correctly classified samples. Otherwise, return the fraction of correctly classified samples. sample_weight : array-like of shape (n_samples,), default=None Sample weights. Returns ------- score : float or int If ``normalize == True``, return the fraction of correctly classified samples (float), else returns the number of correctly classified samples (int). The best performance is 1 with ``normalize == True`` and the number of samples with ``normalize == False``. See Also -------- balanced_accuracy_score : Compute the balanced accuracy to deal with imbalanced datasets. jaccard_score : Compute the Jaccard similarity coefficient score. hamming_loss : Compute the average Hamming loss or Hamming distance between two sets of samples. zero_one_loss : Compute the Zero-one classification loss. By default, the function will return the percentage of imperfectly predicted subsets. Examples -------- >>> from sklearn.metrics import accuracy_score >>> y_pred = [0, 2, 1, 3] >>> y_true = [0, 1, 2, 3] >>> accuracy_score(y_true, y_pred) 0.5 >>> accuracy_score(y_true, y_pred, normalize=False) 2.0 In the multilabel case with binary label indicators: >>> import numpy as np >>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2))) 0.5 rEr-)rNdevicer]rrweightsr}) rr#rUrrMrasarrayr3r) r;r>r}rurNrOrrRdiffering_labelsscores r7accuracy_scorersJ-VV]KMB6"662NFF+FF;FFFFFM: &)&6/bVWX +q0 @&  %)L MMr9>rdpredtrue)r;r>rvrur})rvrur}c  t||\}}t||\}}}|dvrtd|z| t||}nt j |}|j }|dk(r td|j dk(rt j||ftStt j||dk(r td|3t j|jdtj}nt j |}t||||j }|jj dvxrUt j"|t j$|k(xr(|j'dk\xr|j'dk\ }|rt)|D cic]\}} | | } }} t j*|Dcgc]}| j-||d zc}}t j*|Dcgc]}| j-||d zc}}t j.||k||k} t j"| s|| }|| }|| }|jj dvrtj} ntj0} t3|||ff||f| j5} t j6d 5|d k(r| | j9d dz } n4|dk(r| | j9ddz } n|dk(r| | j9z } t j:| } ddd| jdk(rt=j>dt@| Scc} }wcc}wcc}w#1swYDxYw)a Compute confusion matrix to evaluate the accuracy of a classification. By definition a confusion matrix :math:`C` is such that :math:`C_{i, j}` is equal to the number of observations known to be in group :math:`i` and predicted to be in group :math:`j`. Thus in binary classification, the count of true negatives is :math:`C_{0,0}`, false negatives is :math:`C_{1,0}`, true positives is :math:`C_{1,1}` and false positives is :math:`C_{0,1}`. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array-like of shape (n_samples,) Ground truth (correct) target values. y_pred : array-like of shape (n_samples,) Estimated targets as returned by a classifier. labels : array-like of shape (n_classes), default=None List of labels to index the matrix. This may be used to reorder or select a subset of labels. If ``None`` is given, those that appear at least once in ``y_true`` or ``y_pred`` are used in sorted order. sample_weight : array-like of shape (n_samples,), default=None Sample weights. .. versionadded:: 0.18 normalize : {'true', 'pred', 'all'}, default=None Normalizes confusion matrix over the true (rows), predicted (columns) conditions or all the population. If None, confusion matrix will not be normalized. Returns ------- C : ndarray of shape (n_classes, n_classes) Confusion matrix whose i-th row and j-th column entry indicates the number of samples with true label being i-th class and predicted label being j-th class. See Also -------- ConfusionMatrixDisplay.from_estimator : Plot the confusion matrix given an estimator, the data, and the label. ConfusionMatrixDisplay.from_predictions : Plot the confusion matrix given the true and predicted labels. ConfusionMatrixDisplay : Confusion Matrix visualization. References ---------- .. [1] `Wikipedia entry for the Confusion matrix `_ (Wikipedia and other references may use a different convention for axes). Examples -------- >>> from sklearn.metrics import confusion_matrix >>> y_true = [2, 0, 2, 2, 0, 1] >>> y_pred = [0, 0, 2, 2, 0, 2] >>> confusion_matrix(y_true, y_pred) array([[2, 0, 0], [0, 0, 1], [1, 0, 2]]) >>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"] >>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"] >>> confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"]) array([[2, 0, 0], [0, 0, 1], [1, 0, 2]]) In the binary case, we can extract true positives, etc. as follows: >>> tn, fp, fn, tp = confusion_matrix([0, 1, 0, 1], [1, 1, 1, 0]).ravel().tolist() >>> (tn, fp, fn, tp) (0, 2, 1, 1) rD%s is not supportedNrz,'labels' should contains at least one label.rY.At least one label specified must be in y_true>biur-)rLrYignore)rdrT)r]keepdimsrrd)r-r-zA single label was found in 'y_true' and 'y_pred'. For the confusion matrix to have the correct shape, use the 'labels' parameter to pass all known labels.)!r#rUrGr&r0rsizezerosr2rF intersect1donesrLint64rrYkindrdarangerb enumeratearrayget logical_andr1rtoarrayerrstaterq nan_to_numrfr+rh)r;r>rvrur}rRn_labelsneed_index_conversionxy label_to_indindrYcms r7confusion_matrixrss:~#662NFF+FF;FFF --.788 ~vv.F#;; q=KL L [[A 88Xx0< < / 0A 5MN N Qrxx@  =1 FFM:{{H  _,  FF6RYYx00 1  JJLA   JJLA   )26):;A1; ;fM<++Ax!|<MNfM<++Ax!|<MN ..(*FX,= >C 66#;%c* ?2   ()" gi   "  bff!df33B & bff!df33B % bffhB ]]2  xx6 $    IW<MM,s. M4M:M?A&NN )r;r>rurv samplewiseFrurvrc t||\}}t||\}}}t||\}}}| t||}t ||||dvrt d|zt ||} || }d} nN|j||}|jd} |j|tj| |d|gd }|jd k(r]|r t d t} | j|| j|}| j|}| j } ||k(} || }||| }nd}|jdrt#|||jd| }n"|j%|jdx}x}}|jdrt#|||jd| }|jdrt#|||jd| }t'| |d| | }|j)||d }|j)|d }|j)|d }nF|rd nd}|j| jk7s!|j+|j-|| r|j/||j/| kDr.t d|j/||j/| fz|j1|dkrt d|j1|z| |dd|d| f}|dd|d| f}t3|s t3|r|j5|}n|j5||}t7|||||}t7|||||}t7|||||}||z }||z }|} |Q|rO|j| } |j|}|j|}||jd z| z |z |z }nP||j9|| z |z |z }n3|r|jd | z |z |z }n|jd| z |z |z }|j;|j=|||| gj>dS)au Compute a confusion matrix for each class or sample. .. versionadded:: 0.21 Compute class-wise (default) or sample-wise (samplewise=True) multilabel confusion matrix to evaluate the accuracy of a classification, and output confusion matrices for each class or sample. In multilabel confusion matrix :math:`MCM`, the count of true negatives is :math:`MCM_{:,0,0}`, false negatives is :math:`MCM_{:,1,0}`, true positives is :math:`MCM_{:,1,1}` and false positives is :math:`MCM_{:,0,1}`. Multiclass data will be treated as if binarized under a one-vs-rest transformation. Returned confusion matrices will be in the order of sorted unique labels in the union of (y_true, y_pred). Read more in the :ref:`User Guide `. Parameters ---------- y_true : {array-like, sparse matrix} of shape (n_samples, n_outputs) or (n_samples,) Ground truth (correct) target values. y_pred : {array-like, sparse matrix} of shape (n_samples, n_outputs) or (n_samples,) Estimated targets as returned by a classifier. sample_weight : array-like of shape (n_samples,), default=None Sample weights. labels : array-like of shape (n_classes,), default=None A list of classes or column indices to select some (or to force inclusion of classes absent from the data). samplewise : bool, default=False In the multilabel case, this calculates a confusion matrix per sample. Returns ------- multi_confusion : ndarray of shape (n_outputs, 2, 2) A 2x2 confusion matrix corresponding to each output in the input. When calculating class-wise multi_confusion (default), then n_outputs = n_labels; when calculating sample-wise multi_confusion (samplewise=True), n_outputs = n_samples. If ``labels`` is defined, the results will be returned in the order specified in ``labels``, otherwise the results will be returned in sorted order by default. See Also -------- confusion_matrix : Compute confusion matrix to evaluate the accuracy of a classifier. Notes ----- The `multilabel_confusion_matrix` calculates class-wise or sample-wise multilabel confusion matrices, and in multiclass tasks, labels are binarized under a one-vs-rest way; while :func:`~sklearn.metrics.confusion_matrix` calculates one confusion matrix for confusion between every two classes. Examples -------- Multilabel-indicator case: >>> import numpy as np >>> from sklearn.metrics import multilabel_confusion_matrix >>> y_true = np.array([[1, 0, 1], ... [0, 1, 0]]) >>> y_pred = np.array([[1, 0, 0], ... [0, 1, 1]]) >>> multilabel_confusion_matrix(y_true, y_pred) array([[[1, 0], [0, 1]], [[1, 0], [0, 1]], [[0, 1], [1, 0]]]) Multiclass case: >>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"] >>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"] >>> multilabel_confusion_matrix(y_true, y_pred, ... labels=["ant", "bird", "cat"]) array([[[3, 1], [0, 2]], [[5, 0], [1, 0]], [[2, 1], [1, 2]]]) NrrArrT) assume_uniquerNr\r-zJSamplewise metrics are not available outside of multilabel classification.)r minlengthrNrNzGAll labels must be in [0, n labels) for multilabel targets. Got %d > %dzFAll labels must be in [0, n labels) for multilabel targets. Got %d < 0)r]rurNr)rr r ) r#rrUrrrGr&rrLconcatr setdiff1drmr rcrkrerrrtakeany not_equalrarbrmultiplyrrqreshapestackT)r;r>rurvrrNrOdevice_rRpresent_labelsrle sorted_labelstptp_binstp_bins_weightstp_sumtrue_sumpred_sumindicessum_axis true_and_predfpfntns r7multilabel_confusion_matrixr"s\#662NFF-ff=NB7+FF;FFF $]7C FFM: EE.788"662N ~F73<<? S]]>6RTU V  {{a -  ^ vf%f% v *  $+B/O"O == FLLOPRF ,.88FLLO+D DH Dx& <<?  aRH <<?  aRH  vix/@RHq1778W175778W175#1 <<>// /266 LL 04 vvf~~ 66 "%'VVF^RVVN5K$LM vvf~! !#%66&>2  Avix001FAvix001F F x/"OOF3MKK7M '  " '  " '   F B F B B Z ZZ^ ZZ^ ZZ^ V\\!_ ,r 1B 6 ;  " VVM "R '" ,r 1  \\!_r !B & + \\!_r !B & + ::bhhBB/022J ??r9linear quadratic)y1y2rvrru)rvrruc t||||}|jd}t j |d} t j |d} t j | | t j | z } |3t j||gt} d| jdd|dz<not j||gt} | t j|z } |d k(r#t j| | jz } n| | jz d z} t j | |zt j | | zz } td| z S#t$r!}dt|vrd}t||d}~wwxYw) a Compute Cohen's kappa: a statistic that measures inter-annotator agreement. This function computes Cohen's kappa [1]_, a score that expresses the level of agreement between two annotators on a classification problem. It is defined as .. math:: \kappa = (p_o - p_e) / (1 - p_e) where :math:`p_o` is the empirical probability of agreement on the label assigned to any sample (the observed agreement ratio), and :math:`p_e` is the expected agreement when both annotators assign labels randomly. :math:`p_e` is estimated using a per-annotator empirical prior over the class labels [2]_. Read more in the :ref:`User Guide `. Parameters ---------- y1 : array-like of shape (n_samples,) Labels assigned by the first annotator. y2 : array-like of shape (n_samples,) Labels assigned by the second annotator. The kappa statistic is symmetric, so swapping ``y1`` and ``y2`` doesn't change the value. labels : array-like of shape (n_classes,), default=None List of labels to index the matrix. This may be used to select a subset of labels. If `None`, all labels that appear at least once in ``y1`` or ``y2`` are used. Note that at least one label in `labels` must be present in `y1`, even though this function is otherwise agnostic to the order of `y1` and `y2`. weights : {'linear', 'quadratic'}, default=None Weighting type to calculate the score. `None` means not weighted; "linear" means linear weighting; "quadratic" means quadratic weighting. sample_weight : array-like of shape (n_samples,), default=None Sample weights. Returns ------- kappa : float The kappa statistic, which is a number between -1 and 1. The maximum value means complete agreement; zero or lower means chance agreement. References ---------- .. [1] :doi:`J. Cohen (1960). "A coefficient of agreement for nominal scales". Educational and Psychological Measurement 20(1):37-46. <10.1177/001316446002000104>` .. [2] `R. Artstein and M. Poesio (2008). "Inter-coder agreement for computational linguistics". Computational Linguistics 34(4):555-596 `_. .. [3] `Wikipedia entry for the Cohen's kappa `_. Examples -------- >>> from sklearn.metrics import cohen_kappa_score >>> y1 = ["negative", "positive", "negative", "neutral", "positive"] >>> y2 = ["negative", "positive", "negative", "neutral", "negative"] >>> cohen_kappa_score(y1, y2) 0.6875 rvrurzAt least one label in `labels` must be present in `y1` (even though `cohen_kappa_score` is otherwise agnostic to the order of `y1` and `y2`).Nrr\r-rrr )rrGr/rLr0rqouterrr2flatrrabsrr3)rrrvrru confusionrTmsg n_classessum0sum1expectedw_matks r7cohen_kappa_scorers\X $RF-X "I 66)! $D 66)! $Dxxd#bffTl2HI.c:'( #i!m#$)Y/s; 9%% h FF5577?+EUWW_*E uy !BFF58+;$<macromicror?samplesweightedr-r+r;r>rv pos_labelaveragerur6r?)rvrrrur6c &t|||||}|dk(}t|||||}|ddddf} |ddddf|ddddfz|ddddfz} t||\} } } |dk(rH| j| j | d| } | j| j | d| } t | | d d |d | }||S|dk(r)|ddddf|ddddfz}| j |sd}n |dk(r||}nd}tt||| S)aRJaccard similarity coefficient score. The Jaccard index [1], or Jaccard similarity coefficient, defined as the size of the intersection divided by the size of the union of two label sets, is used to compare set of predicted labels for a sample to the corresponding set of labels in ``y_true``. Support beyond term:`binary` targets is achieved by treating :term:`multiclass` and :term:`multilabel` data as a collection of binary problems, one for each label. For the :term:`binary` case, setting `average='binary'` will return the Jaccard similarity coefficient for `pos_label`. If `average` is not `'binary'`, `pos_label` is ignored and scores for both classes are computed, then averaged or both returned (when `average=None`). Similarly, for :term:`multiclass` and :term:`multilabel` targets, scores for all `labels` are either returned or averaged depending on the `average` parameter. Use `labels` specify the set of labels to calculate the score for. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) labels. y_pred : 1d array-like, or label indicator array / sparse matrix Predicted labels, as returned by a classifier. labels : array-like of shape (n_classes,), default=None The set of labels to include when `average != 'binary'`, and their order if `average is None`. Labels present in the data can be excluded, for example in multiclass classification to exclude a "negative class". Labels not present in the data can be included and will be "assigned" 0 samples. For multilabel targets, labels are column indices. By default, all labels in `y_true` and `y_pred` are used in sorted order. pos_label : int, float, bool or str, default=1 The class to report if `average='binary'` and the data is binary, otherwise this parameter is ignored. For multiclass or multilabel targets, set `labels=[pos_label]` and `average != 'binary'` to report metrics for one label only. average : {'micro', 'macro', 'samples', 'weighted', 'binary'} or None, default='binary' If ``None``, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data: ``'binary'``: Only report results for the class specified by ``pos_label``. This is applicable only if targets (``y_{true,pred}``) are binary. ``'micro'``: Calculate metrics globally by counting the total true positives, false negatives and false positives. ``'macro'``: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account. ``'weighted'``: Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label). This alters 'macro' to account for label imbalance. ``'samples'``: Calculate metrics for each instance, and find their average (only meaningful for multilabel classification). sample_weight : array-like of shape (n_samples,), default=None Sample weights. zero_division : "warn", {0.0, 1.0}, default="warn" Sets the value to return when there is a zero division, i.e. when there there are no negative values in predictions and labels. If set to "warn", this acts like 0, but a warning is also raised. .. versionadded:: 0.24 Returns ------- score : float or ndarray of shape (n_unique_labels,), dtype=np.float64 The Jaccard score. When `average` is not `None`, a single scalar is returned. See Also -------- accuracy_score : Function for calculating the accuracy score. f1_score : Function for calculating the F1 score. multilabel_confusion_matrix : Function for computing a confusion matrix for each class or sample. Notes ----- :func:`jaccard_score` may be a poor metric if there are no positives for some samples or classes. Jaccard is undefined if there are no true or predicted labels, and our implementation will return a score of 0 with a warning. References ---------- .. [1] `Wikipedia entry for the Jaccard index `_. Examples -------- >>> import numpy as np >>> from sklearn.metrics import jaccard_score >>> y_true = np.array([[0, 1, 1], ... [1, 1, 0]]) >>> y_pred = np.array([[1, 1, 1], ... [1, 0, 0]]) In the binary case: >>> jaccard_score(y_true[0], y_pred[0]) 0.6666 In the 2D comparison case (e.g. image similarity): >>> jaccard_score(y_true, y_pred, average="micro") 0.6 In the multilabel case: >>> jaccard_score(y_true, y_pred, average='samples') 0.5833 >>> jaccard_score(y_true, y_pred, average='macro') 0.6666 >>> jaccard_score(y_true, y_pred, average=None) array([0.5, 0.5, 1. ]) In the multiclass case: >>> y_pred = [0, 2, 1, 2] >>> y_true = [0, 1, 2, 2] >>> jaccard_score(y_true, y_pred, average=None) array([1. , 0. , 0.33]) rrNr-rrT)rrjaccardztrue or predicted)rr5r)rrN) _check_set_wise_labelsrrrrq _prf_dividerr3r)r;r>rvrrrur6rMCM numerator denominatorrNrOrrrs r7 jaccard_scorers\B$FFGVY OFI%J %#  CAq!G IaAg,Q1W-Aq!G ruruc,t||\}}t||\}}}t||||dvrtd|zt }|j t j||g|j|}|j|}t|||}|jdt j}|jdt j}t j|t j}|j} || zt j||z } | dzt j||z } | dzt j||z } | | z} | dk(ry t| t j| z S) a Compute the Matthews correlation coefficient (MCC). The Matthews correlation coefficient is used in machine learning as a measure of the quality of binary and multiclass classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient. [source: Wikipedia] Binary and multiclass labels are supported. Only in the binary case does this relate to information about true and false positives and negatives. See references below. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array-like of shape (n_samples,) Ground truth (correct) target values. y_pred : array-like of shape (n_samples,) Estimated targets as returned by a classifier. sample_weight : array-like of shape (n_samples,), default=None Sample weights. .. versionadded:: 0.18 Returns ------- mcc : float The Matthews correlation coefficient (+1 represents a perfect prediction, 0 an average random prediction and -1 and inverse prediction). References ---------- .. [1] :doi:`Baldi, Brunak, Chauvin, Andersen and Nielsen, (2000). Assessing the accuracy of prediction algorithms for classification: an overview. <10.1093/bioinformatics/16.5.412>` .. [2] `Wikipedia entry for the Matthews Correlation Coefficient (phi coefficient) `_. .. [3] `Gorodkin, (2004). Comparing two K-category assignments by a K-category correlation coefficient `_. .. [4] `Jurman, Riccadonna, Furlanello, (2012). A Comparison of MCC and CEN Error Measures in MultiClass Prediction `_. Examples -------- >>> from sklearn.metrics import matthews_corrcoef >>> y_true = [+1, +1, +1, -1] >>> y_pred = [+1, -1, +1, +1] >>> matthews_corrcoef(y_true, y_pred) -0.33 >r?r@rrr-)r]rYrrr r,)r#rUrrGr rcr0hstackrkrrqr1tracedotr3rs)r;r>rurRrwCt_sump_sum n_correct n_samplescov_ytypcov_ypypcov_ytyt cov_ypyp_ytyts r7matthews_corrcoefrNs^N#662NFF+FF;FFFFFM: --.788 BFF299ff% &' \\& !F \\& !F}EA EEq E +E EEq E +E"**-I I9$rvveU';;H!|bffUE22H!|bffUE22Hx'MX 6677r9ct||\}}t||||}|rd|z S||j|}||z St|}||z S)aZero-one classification loss. If normalize is ``True``, return the fraction of misclassifications (float), else it returns the number of misclassifications (int). The best performance is 0. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) labels. y_pred : 1d array-like, or label indicator array / sparse matrix Predicted labels, as returned by a classifier. normalize : bool, default=True If ``False``, return the number of misclassifications. Otherwise, return the fraction of misclassifications. sample_weight : array-like of shape (n_samples,), default=None Sample weights. Returns ------- loss : float or int, If ``normalize == True``, return the fraction of misclassifications (float), else it returns the number of misclassifications (int). See Also -------- accuracy_score : Compute the accuracy score. By default, the function will return the fraction of correct predictions divided by the total number of predictions. hamming_loss : Compute the average Hamming loss or Hamming distance between two sets of samples. jaccard_score : Compute the Jaccard similarity coefficient score. Notes ----- In multilabel classification, the zero_one_loss function corresponds to the subset zero-one loss: for each sample, the entire set of labels must be correctly predicted, otherwise the loss for that sample is equal to one. Examples -------- >>> from sklearn.metrics import zero_one_loss >>> y_pred = [1, 2, 3, 4] >>> y_true = [2, 2, 3, 4] >>> zero_one_loss(y_true, y_pred) 0.25 >>> zero_one_loss(y_true, y_pred, normalize=False) 1.0 In the multilabel case with binary label indicators: >>> import numpy as np >>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2))) 0.5 rr-)rrrqr))r;r>r}rurNrOrrs r7 zero_one_lossrsoL && )EB )= E5y  $}-I5  %V,I5  r9r,?r4c (t||d|||||S)aCompute the F1 score, also known as balanced F-score or F-measure. The F1 score can be interpreted as a harmonic mean of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal. The formula for the F1 score is: .. math:: \text{F1} = \frac{2 * \text{TP}}{2 * \text{TP} + \text{FP} + \text{FN}} Where :math:`\text{TP}` is the number of true positives, :math:`\text{FN}` is the number of false negatives, and :math:`\text{FP}` is the number of false positives. F1 is by default calculated as 0.0 when there are no true positives, false negatives, or false positives. Support beyond :term:`binary` targets is achieved by treating :term:`multiclass` and :term:`multilabel` data as a collection of binary problems, one for each label. For the :term:`binary` case, setting `average='binary'` will return F1 score for `pos_label`. If `average` is not `'binary'`, `pos_label` is ignored and F1 score for both classes are computed, then averaged or both returned (when `average=None`). Similarly, for :term:`multiclass` and :term:`multilabel` targets, F1 score for all `labels` are either returned or averaged depending on the `average` parameter. Use `labels` specify the set of labels to calculate F1 score for. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) target values. y_pred : 1d array-like, or label indicator array / sparse matrix Estimated targets as returned by a classifier. labels : array-like, default=None The set of labels to include when `average != 'binary'`, and their order if `average is None`. Labels present in the data can be excluded, for example in multiclass classification to exclude a "negative class". Labels not present in the data can be included and will be "assigned" 0 samples. For multilabel targets, labels are column indices. By default, all labels in `y_true` and `y_pred` are used in sorted order. .. versionchanged:: 0.17 Parameter `labels` improved for multiclass problem. pos_label : int, float, bool or str, default=1 The class to report if `average='binary'` and the data is binary, otherwise this parameter is ignored. For multiclass or multilabel targets, set `labels=[pos_label]` and `average != 'binary'` to report metrics for one label only. average : {'micro', 'macro', 'samples', 'weighted', 'binary'} or None, default='binary' This parameter is required for multiclass/multilabel targets. If ``None``, the metrics for each class are returned. Otherwise, this determines the type of averaging performed on the data: ``'binary'``: Only report results for the class specified by ``pos_label``. This is applicable only if targets (``y_{true,pred}``) are binary. ``'micro'``: Calculate metrics globally by counting the total true positives, false negatives and false positives. ``'macro'``: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account. ``'weighted'``: Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters 'macro' to account for label imbalance; it can result in an F-score that is not between precision and recall. ``'samples'``: Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from :func:`accuracy_score`). sample_weight : array-like of shape (n_samples,), default=None Sample weights. zero_division : {"warn", 0.0, 1.0, np.nan}, default="warn" Sets the value to return when there is a zero division, i.e. when all predictions and labels are negative. Notes: - If set to "warn", this acts like 0, but a warning is also raised. - If set to `np.nan`, such values will be excluded from the average. .. versionadded:: 1.3 `np.nan` option was added. Returns ------- f1_score : float or array of float, shape = [n_unique_labels] F1 score of the positive class in binary classification or weighted average of the F1 scores of each class for the multiclass task. See Also -------- fbeta_score : Compute the F-beta score. precision_recall_fscore_support : Compute the precision, recall, F-score, and support. jaccard_score : Compute the Jaccard similarity coefficient score. multilabel_confusion_matrix : Compute a confusion matrix for each class or sample. Notes ----- When ``true positive + false positive + false negative == 0`` (i.e. a class is completely absent from both ``y_true`` or ``y_pred``), f-score is undefined. In such cases, by default f-score will be set to 0.0, and ``UndefinedMetricWarning`` will be raised. This behavior can be modified by setting the ``zero_division`` parameter. References ---------- .. [1] `Wikipedia entry for the F1-score `_. Examples -------- >>> import numpy as np >>> from sklearn.metrics import f1_score >>> y_true = [0, 1, 2, 0, 1, 2] >>> y_pred = [0, 2, 1, 0, 0, 1] >>> f1_score(y_true, y_pred, average='macro') 0.267 >>> f1_score(y_true, y_pred, average='micro') 0.33 >>> f1_score(y_true, y_pred, average='weighted') 0.267 >>> f1_score(y_true, y_pred, average=None) array([0.8, 0. , 0. ]) >>> # binary classification >>> y_true_empty = [0, 0, 0, 0, 0, 0] >>> y_pred_empty = [0, 0, 0, 0, 0, 0] >>> f1_score(y_true_empty, y_pred_empty) 0.0... >>> f1_score(y_true_empty, y_pred_empty, zero_division=1.0) 1.0... >>> f1_score(y_true_empty, y_pred_empty, zero_division=np.nan) nan... >>> # multilabel classification >>> y_true = [[0, 0, 0], [1, 1, 1], [0, 1, 1]] >>> y_pred = [[0, 0, 0], [1, 1, 1], [1, 1, 0]] >>> f1_score(y_true, y_pred, average=None) array([0.66666667, 1. , 0.66666667]) r-)betarvrrrur6) fbeta_scorers r7f1_scorers*h  ##  r9both)closed)r;r>rrvrrrur6c 8t||||||d|| \}}} }| S)aCompute the F-beta score. The F-beta score is the weighted harmonic mean of precision and recall, reaching its optimal value at 1 and its worst value at 0. The `beta` parameter represents the ratio of recall importance to precision importance. `beta > 1` gives more weight to recall, while `beta < 1` favors precision. For example, `beta = 2` makes recall twice as important as precision, while `beta = 0.5` does the opposite. Asymptotically, `beta -> +inf` considers only recall, and `beta -> 0` only precision. The formula for F-beta score is: .. math:: F_\beta = \frac{(1 + \beta^2) \text{tp}} {(1 + \beta^2) \text{tp} + \text{fp} + \beta^2 \text{fn}} Where :math:`\text{tp}` is the number of true positives, :math:`\text{fp}` is the number of false positives, and :math:`\text{fn}` is the number of false negatives. Support beyond term:`binary` targets is achieved by treating :term:`multiclass` and :term:`multilabel` data as a collection of binary problems, one for each label. For the :term:`binary` case, setting `average='binary'` will return F-beta score for `pos_label`. If `average` is not `'binary'`, `pos_label` is ignored and F-beta score for both classes are computed, then averaged or both returned (when `average=None`). Similarly, for :term:`multiclass` and :term:`multilabel` targets, F-beta score for all `labels` are either returned or averaged depending on the `average` parameter. Use `labels` specify the set of labels to calculate F-beta score for. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) target values. y_pred : 1d array-like, or label indicator array / sparse matrix Estimated targets as returned by a classifier. beta : float Determines the weight of recall in the combined score. labels : array-like, default=None The set of labels to include when `average != 'binary'`, and their order if `average is None`. Labels present in the data can be excluded, for example in multiclass classification to exclude a "negative class". Labels not present in the data can be included and will be "assigned" 0 samples. For multilabel targets, labels are column indices. By default, all labels in `y_true` and `y_pred` are used in sorted order. .. versionchanged:: 0.17 Parameter `labels` improved for multiclass problem. pos_label : int, float, bool or str, default=1 The class to report if `average='binary'` and the data is binary, otherwise this parameter is ignored. For multiclass or multilabel targets, set `labels=[pos_label]` and `average != 'binary'` to report metrics for one label only. average : {'micro', 'macro', 'samples', 'weighted', 'binary'} or None, default='binary' This parameter is required for multiclass/multilabel targets. If ``None``, the metrics for each class are returned. Otherwise, this determines the type of averaging performed on the data: ``'binary'``: Only report results for the class specified by ``pos_label``. This is applicable only if targets (``y_{true,pred}``) are binary. ``'micro'``: Calculate metrics globally by counting the total true positives, false negatives and false positives. ``'macro'``: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account. ``'weighted'``: Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters 'macro' to account for label imbalance; it can result in an F-score that is not between precision and recall. ``'samples'``: Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from :func:`accuracy_score`). sample_weight : array-like of shape (n_samples,), default=None Sample weights. zero_division : {"warn", 0.0, 1.0, np.nan}, default="warn" Sets the value to return when there is a zero division, i.e. when all predictions and labels are negative. Notes: - If set to "warn", this acts like 0, but a warning is also raised. - If set to `np.nan`, such values will be excluded from the average. .. versionadded:: 1.3 `np.nan` option was added. Returns ------- fbeta_score : float (if average is not None) or array of float, shape = [n_unique_labels] F-beta score of the positive class in binary classification or weighted average of the F-beta score of each class for the multiclass task. See Also -------- precision_recall_fscore_support : Compute the precision, recall, F-score, and support. multilabel_confusion_matrix : Compute a confusion matrix for each class or sample. Notes ----- When ``true positive + false positive + false negative == 0``, f-score returns 0.0 and raises ``UndefinedMetricWarning``. This behavior can be modified by setting ``zero_division``. F-beta score is not implemented as a named scorer that can be passed to the `scoring` parameter of cross-validation tools directly: it requires to be wrapped with :func:`make_scorer` so as to specify the value of `beta`. See examples for details. References ---------- .. [1] R. Baeza-Yates and B. Ribeiro-Neto (2011). Modern Information Retrieval. Addison Wesley, pp. 327-328. .. [2] `Wikipedia entry for the F1-score `_. Examples -------- >>> import numpy as np >>> from sklearn.metrics import fbeta_score >>> y_true = [0, 1, 2, 0, 1, 2] >>> y_pred = [0, 2, 1, 0, 0, 1] >>> fbeta_score(y_true, y_pred, average='macro', beta=0.5) 0.238 >>> fbeta_score(y_true, y_pred, average='micro', beta=0.5) 0.33 >>> fbeta_score(y_true, y_pred, average='weighted', beta=0.5) 0.238 >>> fbeta_score(y_true, y_pred, average=None, beta=0.5) array([0.71, 0. , 0. ]) >>> y_pred_empty = [0, 0, 0, 0, 0, 0] >>> fbeta_score( ... y_true, ... y_pred_empty, ... average="macro", ... zero_division=np.nan, ... beta=0.5, ... ) 0.128 In order to use :func:`fbeta_scorer` as a scorer, a callable scorer objects needs to be created first with :func:`make_scorer`, passing the value for the `beta` parameter. >>> from sklearn.metrics import fbeta_score, make_scorer >>> ftwo_scorer = make_scorer(fbeta_score, beta=2) >>> from sklearn.model_selection import GridSearchCV >>> from sklearn.svm import LinearSVC >>> grid = GridSearchCV( ... LinearSVC(dual="auto"), ... param_grid={'C': [1, 10]}, ... scoring=ftwo_scorer, ... cv=5 ... ) )f-scorerrvrrwarn_forrur6precision_recall_fscore_support) r;r>rrvrrrur6rOfs r7rrs9\1 ## JAq!Q Hr9cdt||\}}t|||} |dk(} |j|d| }d|| <|j|| |z } |j| s| St |} | | | <|dk7s||vr| S||vr,t |||j d| jd| S) aSPerforms division and handles divide-by-zero. On zero-division, sets the corresponding result elements equal to 0, 1 or np.nan (according to ``zero_division``). Plus, if ``zero_division != "warn"`` raises a warning. The metric, modifier and average arguments are used only for determining an appropriate warning. rrT)copyrYr-rr+z is)rrrrr8 _warn_prf capitalizerL) rrmetricmodifierrrr6rNrO dtype_floatmaskresultzero_division_values r7rrs )[ 1EB/ ;2NK ! D**[t;*GKK ZZ Z 5 CF 66$< /}=&F4L &"8 '8(9(9(;'rrvraverage_optionsrRrs r7rrs& FOo%'X*=4s?7KKLL"662NFF+FF;FFF]66:;N( X .~&!+$$YK0,,:+;= [F" M#?3O%&&y1@FY  ) # ' #    Mr9) r;r>rrvrrrrur6) precisionrecallrrc t|t|||||}|dk(} t||||| } | ddddf} | | ddddfz} | | ddddfz} t||\}}}|dk(rc|j |j | d} |j |j | d} |j |j | d} |dz}t | | d d |||}t | | d d |||}tj|r|}nd|dk(r|}n\t|| }||j| |z|j| |z}t d|z|j| |z|dd|||}|dk(r| }n |dk(r|}nd}|Dtt||}tt||}tt||}d} |||| fS)afCompute precision, recall, F-measure and support for each class. The precision is the ratio ``tp / (tp + fp)`` where ``tp`` is the number of true positives and ``fp`` the number of false positives. The precision is intuitively the ability of the classifier not to label a negative sample as positive. The recall is the ratio ``tp / (tp + fn)`` where ``tp`` is the number of true positives and ``fn`` the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples. The F-beta score can be interpreted as a weighted harmonic mean of the precision and recall, where an F-beta score reaches its best value at 1 and worst score at 0. The F-beta score weights recall more than precision by a factor of ``beta``. ``beta == 1.0`` means recall and precision are equally important. The support is the number of occurrences of each class in ``y_true``. Support beyond term:`binary` targets is achieved by treating :term:`multiclass` and :term:`multilabel` data as a collection of binary problems, one for each label. For the :term:`binary` case, setting `average='binary'` will return metrics for `pos_label`. If `average` is not `'binary'`, `pos_label` is ignored and metrics for both classes are computed, then averaged or both returned (when `average=None`). Similarly, for :term:`multiclass` and :term:`multilabel` targets, metrics for all `labels` are either returned or averaged depending on the `average` parameter. Use `labels` specify the set of labels to calculate metrics for. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) target values. y_pred : 1d array-like, or label indicator array / sparse matrix Estimated targets as returned by a classifier. beta : float, default=1.0 The strength of recall versus precision in the F-score. labels : array-like, default=None The set of labels to include when `average != 'binary'`, and their order if `average is None`. Labels present in the data can be excluded, for example in multiclass classification to exclude a "negative class". Labels not present in the data can be included and will be "assigned" 0 samples. For multilabel targets, labels are column indices. By default, all labels in `y_true` and `y_pred` are used in sorted order. .. versionchanged:: 0.17 Parameter `labels` improved for multiclass problem. pos_label : int, float, bool or str, default=1 The class to report if `average='binary'` and the data is binary, otherwise this parameter is ignored. For multiclass or multilabel targets, set `labels=[pos_label]` and `average != 'binary'` to report metrics for one label only. average : {'micro', 'macro', 'samples', 'weighted', 'binary'} or None, default='binary' This parameter is required for multiclass/multilabel targets. If ``None``, the metrics for each class are returned. Otherwise, this determines the type of averaging performed on the data: ``'binary'``: Only report results for the class specified by ``pos_label``. This is applicable only if targets (``y_{true,pred}``) are binary. ``'micro'``: Calculate metrics globally by counting the total true positives, false negatives and false positives. ``'macro'``: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account. ``'weighted'``: Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters 'macro' to account for label imbalance; it can result in an F-score that is not between precision and recall. ``'samples'``: Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from :func:`accuracy_score`). warn_for : list, tuple or set, for internal use This determines which warnings will be made in the case that this function is being used to return only one of its metrics. sample_weight : array-like of shape (n_samples,), default=None Sample weights. zero_division : {"warn", 0.0, 1.0, np.nan}, default="warn" Sets the value to return when there is a zero division: - recall: when there are no positive labels - precision: when there are no positive predictions - f-score: both Notes: - If set to "warn", this acts like 0, but a warning is also raised. - If set to `np.nan`, such values will be excluded from the average. .. versionadded:: 1.3 `np.nan` option was added. Returns ------- precision : float (if average is not None) or array of float, shape = [n_unique_labels] Precision score. recall : float (if average is not None) or array of float, shape = [n_unique_labels] Recall score. fbeta_score : float (if average is not None) or array of float, shape = [n_unique_labels] F-beta score. support : None (if average is not None) or array of int, shape = [n_unique_labels] The number of occurrences of each label in ``y_true``. Notes ----- When ``true positive + false positive == 0``, precision is undefined. When ``true positive + false negative == 0``, recall is undefined. When ``true positive + false negative + false positive == 0``, f-score is undefined. In such cases, by default the metric will be set to 0, and ``UndefinedMetricWarning`` will be raised. This behavior can be modified with ``zero_division``. References ---------- .. [1] `Wikipedia entry for the Precision and recall `_. .. [2] `Wikipedia entry for the F1-score `_. .. [3] `Discriminative Methods for Multi-labeled Classification Advances in Knowledge Discovery and Data Mining (2004), pp. 22-30 by Shantanu Godbole, Sunita Sarawagi `_. Examples -------- >>> import numpy as np >>> from sklearn.metrics import precision_recall_fscore_support >>> y_true = np.array(['cat', 'dog', 'pig', 'cat', 'dog', 'pig']) >>> y_pred = np.array(['cat', 'pig', 'dog', 'cat', 'cat', 'dog']) >>> precision_recall_fscore_support(y_true, y_pred, average='macro') (0.222, 0.333, 0.267, None) >>> precision_recall_fscore_support(y_true, y_pred, average='micro') (0.33, 0.33, 0.33, None) >>> precision_recall_fscore_support(y_true, y_pred, average='weighted') (0.222, 0.333, 0.267, None) It is possible to compute per-label precisions, recalls, F1-scores and supports instead of averaging: >>> precision_recall_fscore_support(y_true, y_pred, average=None, ... labels=['pig', 'dog', 'cat']) (array([0. , 0. , 0.66]), array([0., 0., 1.]), array([0. , 0. , 0.8]), array([2, 2, 2])) rrNr-rr)r-r r" predictedr#r)rNrrztrue nor predictedrr) r8rrrrrqrr0isposinfrastyper3r$)r;r>rrvrrrrur6rrrrrrNrOrbeta2r"r#f_scoremax_float_typedenomrs r7r r sR' #FFGVY OFI%J %#  CAq\FAq!G $HAq!G $H-ff=NB7'BFF6ND1::bffX.5::bffX.5 !GE+{GX}I(FGX}F {{4 4r'J (N;;bii n?   Y"))FN; ;      * I +iAB {67;< GW=> fgx //r9 deprecated)r;r>rvru raise_warningreplace_undefined_by)rvrur.r/ct||\}}t||\}}}|dk7rtd|d}|dk7rtj|t nd}|dk(rddd}t |trhd|d }d |vrSd |vrO |jd d } t| d tdd|jd d } t| dtdddn t|t||||} | j\} }}}||z}| |z}||z}||z}||z}| |z}|dk(r>d}tj|tdtj }tj }|dk(rZ|r-|dk(rd}nd}d} tj||ztdt |t"rtj$|r|}n }n||z }| dk(rK|rd}tj|tdt |t"rtj$|r|}n }n||z }t#|t#|fS#t$r} t|| d } ~ wwxYw)aCompute binary classification positive and negative likelihood ratios. The positive likelihood ratio is `LR+ = sensitivity / (1 - specificity)` where the sensitivity or recall is the ratio `tp / (tp + fn)` and the specificity is `tn / (tn + fp)`. The negative likelihood ratio is `LR- = (1 - sensitivity) / specificity`. Here `tp` is the number of true positives, `fp` the number of false positives, `tn` is the number of true negatives and `fn` the number of false negatives. Both class likelihood ratios can be used to obtain post-test probabilities given a pre-test probability. `LR+` ranges from 1.0 to infinity. A `LR+` of 1.0 indicates that the probability of predicting the positive class is the same for samples belonging to either class; therefore, the test is useless. The greater `LR+` is, the more a positive prediction is likely to be a true positive when compared with the pre-test probability. A value of `LR+` lower than 1.0 is invalid as it would indicate that the odds of a sample being a true positive decrease with respect to the pre-test odds. `LR-` ranges from 0.0 to 1.0. The closer it is to 0.0, the lower the probability of a given sample to be a false negative. A `LR-` of 1.0 means the test is useless because the odds of having the condition did not change after the test. A value of `LR-` greater than 1.0 invalidates the classifier as it indicates an increase in the odds of a sample belonging to the positive class after being classified as negative. This is the case when the classifier systematically predicts the opposite of the true label. A typical application in medicine is to identify the positive/negative class to the presence/absence of a disease, respectively; the classifier being a diagnostic test; the pre-test probability of an individual having the disease can be the prevalence of such disease (proportion of a particular population found to be affected by a medical condition); and the post-test probabilities would be the probability that the condition is truly present given a positive test result. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) target values. y_pred : 1d array-like, or label indicator array / sparse matrix Estimated targets as returned by a classifier. labels : array-like, default=None List of labels to index the matrix. This may be used to select the positive and negative classes with the ordering `labels=[negative_class, positive_class]`. If `None` is given, those that appear at least once in `y_true` or `y_pred` are used in sorted order. sample_weight : array-like of shape (n_samples,), default=None Sample weights. raise_warning : bool, default=True Whether or not a case-specific warning message is raised when there is division by zero. .. deprecated:: 1.7 `raise_warning` was deprecated in version 1.7 and will be removed in 1.9, when an :class:`~sklearn.exceptions.UndefinedMetricWarning` will always raise in case of a division by zero. replace_undefined_by : np.nan, 1.0, or dict, default=np.nan Sets the return values for LR+ and LR- when there is a division by zero. Can take the following values: - `np.nan` to return `np.nan` for both `LR+` and `LR-` - `1.0` to return the worst possible scores: `{"LR+": 1.0, "LR-": 1.0}` - a dict in the format `{"LR+": value_1, "LR-": value_2}` where the values can be non-negative floats, `np.inf` or `np.nan` in the range of the likelihood ratios. For example, `{"LR+": 1.0, "LR-": 1.0}` can be used for returning the worst scores, indicating a useless model, and `{"LR+": np.inf, "LR-": 0.0}` can be used for returning the best scores, indicating a useful model. If a division by zero occurs, only the affected metric is replaced with the set value; the other metric is calculated as usual. .. versionadded:: 1.7 Returns ------- (positive_likelihood_ratio, negative_likelihood_ratio) : tuple A tuple of two floats, the first containing the positive likelihood ratio (LR+) and the second the negative likelihood ratio (LR-). Warns ----- Raises :class:`~sklearn.exceptions.UndefinedMetricWarning` when `y_true` and `y_pred` lead to the following conditions: - The number of false positives is 0 and `raise_warning` is set to `True` (default): positive likelihood ratio is undefined. - The number of true negatives is 0 and `raise_warning` is set to `True` (default): negative likelihood ratio is undefined. - The sum of true positives and false negatives is 0 (no samples of the positive class are present in `y_true`): both likelihood ratios are undefined. For the first two cases, an undefined metric can be defined by setting the `replace_undefined_by` param. References ---------- .. [1] `Wikipedia entry for the Likelihood ratios in diagnostic testing `_. Examples -------- >>> import numpy as np >>> from sklearn.metrics import class_likelihood_ratios >>> class_likelihood_ratios([0, 1, 0, 1, 0], [1, 1, 0, 0, 0]) (1.5, 0.75) >>> y_true = np.array(["non-cat", "cat", "non-cat", "cat", "non-cat"]) >>> y_pred = np.array(["cat", "cat", "non-cat", "non-cat", "non-cat"]) >>> class_likelihood_ratios(y_true, y_pred) (1.33, 0.66) >>> y_true = np.array(["non-zebra", "zebra", "non-zebra", "zebra", "non-zebra"]) >>> y_pred = np.array(["zebra", "zebra", "non-zebra", "non-zebra", "non-zebra"]) >>> class_likelihood_ratios(y_true, y_pred) (1.5, 0.75) To avoid ambiguities, use the notation `labels=[negative_class, positive_class]` >>> y_true = np.array(["non-cat", "cat", "non-cat", "cat", "non-cat"]) >>> y_pred = np.array(["cat", "cat", "non-cat", "non-cat", "non-cat"]) >>> class_likelihood_ratios(y_true, y_pred, labels=["non-cat", "cat"]) (1.5, 0.75) r?z[class_likelihood_ratios only supports binary classification problems, got targets of type: z`raise_warning` was deprecated in version 1.7 and will be removed in 1.9. An `UndefinedMetricWarning` will always be raised in case of a division by zero and the value set with the `replace_undefined_by` param will be returned.r-Tr)LR+LR-zThe dictionary passed as `replace_undefined_by` needs to be in the form `{'LR+': `value_1`, 'LR-': `value_2`}` where the value for `LR+` ranges from `1.0` to `np.inf` or is `np.nan` and the value for `LR-` ranges from `0.0` to `1.0` or is `np.nan`; got `z`.r1r2Npositive_likelihood_ratioleft) target_typemin_valinclude_boundariesnegative_likelihood_ratior,r)r5r6max_valr7rurvraANo samples of the positive class are present in `y_true`. `positive_likelihood_ratio` and `negative_likelihood_ratio` are both set to `np.nan`. Use the `replace_undefined_by` param to control this behavior. To suppress this warning or turn it into an error, see Python's `warnings` module and `warnings.catch_warnings()`.r rzTNo samples were predicted for the positive class and `positive_likelihood_ratio` is z/`positive_likelihood_ratio` is ill-defined and z9set to `np.nan`. Use the `replace_undefined_by` param to z`negative_likelihood_ratio` is ill-defined and set to `np.nan`. Use the `replace_undefined_by` param to control this behavior. To suppress this warning or turn it into an error, see Python's `warnings` module and `warnings.catch_warnings()`.)r#rUrGrfr+ FutureWarningr.dictrrr Exceptionrravelr r0r4r3isnan)r;r>rvrur.r/rRmsg_deprecated_paramrdesired_lr_posdesired_lr_negrTrrrrr support_pos support_negpos_num pos_denomneg_num neg_denomr3r8 msg_beginningmsg_ends r7class_likelihood_ratiosrKsx#662NFF+FF;FFF  ..4X 7   T  $ *M: s"'*37&- 34H2H L ) )9M0M -!5!9!9%!F"/!%'- "6!9!9%!F"/!%'- S/ ! #  BXXZNBBr'Kr'K;G[ I;G[ Ia A   c1a@$&FF!$&FF! Qw Qw6 !R QG X M MM-'13IVW X *E 2rxx@T7U(< % )7 %$+i$7! Qw E  MM#5! D *E 2rxx@T7U(< % )7 %$+i$7! * +U3L-M MM[ - o1, -sA H-- I6 IIc 6t|||||d||\}}}}|S)aNCompute the precision. The precision is the ratio ``tp / (tp + fp)`` where ``tp`` is the number of true positives and ``fp`` the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative. The best value is 1 and the worst value is 0. Support beyond term:`binary` targets is achieved by treating :term:`multiclass` and :term:`multilabel` data as a collection of binary problems, one for each label. For the :term:`binary` case, setting `average='binary'` will return precision for `pos_label`. If `average` is not `'binary'`, `pos_label` is ignored and precision for both classes are computed, then averaged or both returned (when `average=None`). Similarly, for :term:`multiclass` and :term:`multilabel` targets, precision for all `labels` are either returned or averaged depending on the `average` parameter. Use `labels` specify the set of labels to calculate precision for. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) target values. y_pred : 1d array-like, or label indicator array / sparse matrix Estimated targets as returned by a classifier. labels : array-like, default=None The set of labels to include when `average != 'binary'`, and their order if `average is None`. Labels present in the data can be excluded, for example in multiclass classification to exclude a "negative class". Labels not present in the data can be included and will be "assigned" 0 samples. For multilabel targets, labels are column indices. By default, all labels in `y_true` and `y_pred` are used in sorted order. .. versionchanged:: 0.17 Parameter `labels` improved for multiclass problem. pos_label : int, float, bool or str, default=1 The class to report if `average='binary'` and the data is binary, otherwise this parameter is ignored. For multiclass or multilabel targets, set `labels=[pos_label]` and `average != 'binary'` to report metrics for one label only. average : {'micro', 'macro', 'samples', 'weighted', 'binary'} or None, default='binary' This parameter is required for multiclass/multilabel targets. If ``None``, the metrics for each class are returned. Otherwise, this determines the type of averaging performed on the data: ``'binary'``: Only report results for the class specified by ``pos_label``. This is applicable only if targets (``y_{true,pred}``) are binary. ``'micro'``: Calculate metrics globally by counting the total true positives, false negatives and false positives. ``'macro'``: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account. ``'weighted'``: Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters 'macro' to account for label imbalance; it can result in an F-score that is not between precision and recall. ``'samples'``: Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from :func:`accuracy_score`). sample_weight : array-like of shape (n_samples,), default=None Sample weights. zero_division : {"warn", 0.0, 1.0, np.nan}, default="warn" Sets the value to return when there is a zero division. Notes: - If set to "warn", this acts like 0, but a warning is also raised. - If set to `np.nan`, such values will be excluded from the average. .. versionadded:: 1.3 `np.nan` option was added. Returns ------- precision : float (if average is not None) or array of float of shape (n_unique_labels,) Precision of the positive class in binary classification or weighted average of the precision of each class for the multiclass task. See Also -------- precision_recall_fscore_support : Compute precision, recall, F-measure and support for each class. recall_score : Compute the ratio ``tp / (tp + fn)`` where ``tp`` is the number of true positives and ``fn`` the number of false negatives. PrecisionRecallDisplay.from_estimator : Plot precision-recall curve given an estimator and some data. PrecisionRecallDisplay.from_predictions : Plot precision-recall curve given binary class predictions. multilabel_confusion_matrix : Compute a confusion matrix for each class or sample. Notes ----- When ``true positive + false positive == 0``, precision returns 0 and raises ``UndefinedMetricWarning``. This behavior can be modified with ``zero_division``. Examples -------- >>> import numpy as np >>> from sklearn.metrics import precision_score >>> y_true = [0, 1, 2, 0, 1, 2] >>> y_pred = [0, 2, 1, 0, 0, 1] >>> precision_score(y_true, y_pred, average='macro') 0.22 >>> precision_score(y_true, y_pred, average='micro') 0.33 >>> precision_score(y_true, y_pred, average='weighted') 0.22 >>> precision_score(y_true, y_pred, average=None) array([0.66, 0. , 0. ]) >>> y_pred = [0, 0, 0, 0, 0, 0] >>> precision_score(y_true, y_pred, average=None) array([0.33, 0. , 0. ]) >>> precision_score(y_true, y_pred, average=None, zero_division=1) array([0.33, 1. , 1. ]) >>> precision_score(y_true, y_pred, average=None, zero_division=np.nan) array([0.33, nan, nan]) >>> # multilabel classification >>> y_true = [[0, 0, 0], [1, 1, 1], [0, 1, 1]] >>> y_pred = [[0, 0, 0], [1, 1, 1], [1, 1, 0]] >>> precision_score(y_true, y_pred, average=None) array([0.5, 1. , 1. ]) )r"rvrrrrur6r) r;r>rvrrrur6prOs r7precision_scorerO4 s6P1## JAq!Q Hr9c 6t|||||d||\}}}}|S)aKCompute the recall. The recall is the ratio ``tp / (tp + fn)`` where ``tp`` is the number of true positives and ``fn`` the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples. The best value is 1 and the worst value is 0. Support beyond term:`binary` targets is achieved by treating :term:`multiclass` and :term:`multilabel` data as a collection of binary problems, one for each label. For the :term:`binary` case, setting `average='binary'` will return recall for `pos_label`. If `average` is not `'binary'`, `pos_label` is ignored and recall for both classes are computed then averaged or both returned (when `average=None`). Similarly, for :term:`multiclass` and :term:`multilabel` targets, recall for all `labels` are either returned or averaged depending on the `average` parameter. Use `labels` specify the set of labels to calculate recall for. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) target values. y_pred : 1d array-like, or label indicator array / sparse matrix Estimated targets as returned by a classifier. labels : array-like, default=None The set of labels to include when `average != 'binary'`, and their order if `average is None`. Labels present in the data can be excluded, for example in multiclass classification to exclude a "negative class". Labels not present in the data can be included and will be "assigned" 0 samples. For multilabel targets, labels are column indices. By default, all labels in `y_true` and `y_pred` are used in sorted order. .. versionchanged:: 0.17 Parameter `labels` improved for multiclass problem. pos_label : int, float, bool or str, default=1 The class to report if `average='binary'` and the data is binary, otherwise this parameter is ignored. For multiclass or multilabel targets, set `labels=[pos_label]` and `average != 'binary'` to report metrics for one label only. average : {'micro', 'macro', 'samples', 'weighted', 'binary'} or None, default='binary' This parameter is required for multiclass/multilabel targets. If ``None``, the metrics for each class are returned. Otherwise, this determines the type of averaging performed on the data: ``'binary'``: Only report results for the class specified by ``pos_label``. This is applicable only if targets (``y_{true,pred}``) are binary. ``'micro'``: Calculate metrics globally by counting the total true positives, false negatives and false positives. ``'macro'``: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account. ``'weighted'``: Calculate metrics for each label, and find their average weighted by support (the number of true instances for each label). This alters 'macro' to account for label imbalance; it can result in an F-score that is not between precision and recall. Weighted recall is equal to accuracy. ``'samples'``: Calculate metrics for each instance, and find their average (only meaningful for multilabel classification where this differs from :func:`accuracy_score`). sample_weight : array-like of shape (n_samples,), default=None Sample weights. zero_division : {"warn", 0.0, 1.0, np.nan}, default="warn" Sets the value to return when there is a zero division. Notes: - If set to "warn", this acts like 0, but a warning is also raised. - If set to `np.nan`, such values will be excluded from the average. .. versionadded:: 1.3 `np.nan` option was added. Returns ------- recall : float (if average is not None) or array of float of shape (n_unique_labels,) Recall of the positive class in binary classification or weighted average of the recall of each class for the multiclass task. See Also -------- precision_recall_fscore_support : Compute precision, recall, F-measure and support for each class. precision_score : Compute the ratio ``tp / (tp + fp)`` where ``tp`` is the number of true positives and ``fp`` the number of false positives. balanced_accuracy_score : Compute balanced accuracy to deal with imbalanced datasets. multilabel_confusion_matrix : Compute a confusion matrix for each class or sample. PrecisionRecallDisplay.from_estimator : Plot precision-recall curve given an estimator and some data. PrecisionRecallDisplay.from_predictions : Plot precision-recall curve given binary class predictions. Notes ----- When ``true positive + false negative == 0``, recall returns 0 and raises ``UndefinedMetricWarning``. This behavior can be modified with ``zero_division``. Examples -------- >>> import numpy as np >>> from sklearn.metrics import recall_score >>> y_true = [0, 1, 2, 0, 1, 2] >>> y_pred = [0, 2, 1, 0, 0, 1] >>> recall_score(y_true, y_pred, average='macro') 0.33 >>> recall_score(y_true, y_pred, average='micro') 0.33 >>> recall_score(y_true, y_pred, average='weighted') 0.33 >>> recall_score(y_true, y_pred, average=None) array([1., 0., 0.]) >>> y_true = [0, 0, 0, 0, 0, 0] >>> recall_score(y_true, y_pred, average=None) array([0.5, 0. , 0. ]) >>> recall_score(y_true, y_pred, average=None, zero_division=1) array([0.5, 1. , 1. ]) >>> recall_score(y_true, y_pred, average=None, zero_division=np.nan) array([0.5, nan, nan]) >>> # multilabel classification >>> y_true = [[0, 0, 0], [1, 1, 1], [0, 1, 1]] >>> y_pred = [[0, 0, 0], [1, 1, 1], [1, 1, 0]] >>> recall_score(y_true, y_pred, average=None) array([1. , 1. , 0.5]) )r#rMr) r;r>rvrrrur6rOrs r7 recall_scorerR s6R1## JAq!Q Hr9)r;r>ruadjusted)rurSct|||}tjdd5tj||j dz }dddtj tj r.tjd|tj |}tj|}|rt|}d|z }||z}|d|z z}t|S#1swYxYw)a Compute the balanced accuracy. The balanced accuracy in binary and multiclass classification problems to deal with imbalanced datasets. It is defined as the average of recall obtained on each class. The best value is 1 and the worst value is 0 when ``adjusted=False``. Read more in the :ref:`User Guide `. .. versionadded:: 0.20 Parameters ---------- y_true : array-like of shape (n_samples,) Ground truth (correct) target values. y_pred : array-like of shape (n_samples,) Estimated targets as returned by a classifier. sample_weight : array-like of shape (n_samples,), default=None Sample weights. adjusted : bool, default=False When true, the result is adjusted for chance, so that random performance would score 0, while keeping perfect performance at a score of 1. Returns ------- balanced_accuracy : float Balanced accuracy score. See Also -------- average_precision_score : Compute average precision (AP) from prediction scores. precision_score : Compute the precision score. recall_score : Compute the recall score. roc_auc_score : Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. Notes ----- Some literature promotes alternative definitions of balanced accuracy. Our definition is equivalent to :func:`accuracy_score` with class-balanced sample weights, and shares desirable properties with the binary case. See the :ref:`User Guide `. References ---------- .. [1] Brodersen, K.H.; Ong, C.S.; Stephan, K.E.; Buhmann, J.M. (2010). The balanced accuracy and its posterior distribution. Proceedings of the 20th International Conference on Pattern Recognition, 3121-24. .. [2] John. D. Kelleher, Brian Mac Namee, Aoife D'Arcy, (2015). `Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies `_. Examples -------- >>> from sklearn.metrics import balanced_accuracy_score >>> y_true = [0, 1, 0, 0, 1, 0] >>> y_pred = [0, 1, 0, 0, 0, 1] >>> balanced_accuracy_score(y_true, y_pred) 0.625 rr)divideinvalidr-r\Nz%y_pred contains classes not in y_true) rr0rdiagrqrr?rfr+meanrFr3) r;r>rurSr per_classrrchances r7balanced_accuracy_scorer[ s\ }EA Hh 7/GGAJA. / vvbhhy!" =>rxx 223 GGI E N Y  V <//s )C,,C5r4)r;r>rv target_namesrudigits output_dictr6)rvr\rur]r^r6c Ht||\}}t||\}}}|t||}d} ntj|}d} |dk(xs|dk(xr&| xs!t |t t||k\} |~t |t |k7rg| r8tjdjt |t |n-tdjt |t |||D cgc]} d| z }} gd } t|||d|| \} }}}t|| |||}|jd rd }nd }|r^|Dcic] }|d|dd }}|jD]4\}}tt| |Dcgc] }t!|c}||<6nzd}t#d|D}t#|t ||}ddt | zz}|jdg| d|i}|dz }d}|D]}||j|||dz }|dz }|D]}|jdr| rd}n|dz}t|||||| \} }!}"}#| |!|"tj$|g}$|r0tt| |$Dcgc] }t!|c}|<|dk(r"d}%|%j|ddg|$dd|dz }j|g|$|dz }|rdj'vr |dd|d<|SScc} wcc}wcc}wcc}w) avBuild a text report showing the main classification metrics. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) target values. y_pred : 1d array-like, or label indicator array / sparse matrix Estimated targets as returned by a classifier. labels : array-like of shape (n_labels,), default=None Optional list of label indices to include in the report. target_names : array-like of shape (n_labels,), default=None Optional display names matching the labels (same order). sample_weight : array-like of shape (n_samples,), default=None Sample weights. digits : int, default=2 Number of digits for formatting output floating point values. When ``output_dict`` is ``True``, this will be ignored and the returned values will not be rounded. output_dict : bool, default=False If True, return output as dict. .. versionadded:: 0.20 zero_division : {"warn", 0.0, 1.0, np.nan}, default="warn" Sets the value to return when there is a zero division. If set to "warn", this acts as 0, but warnings are also raised. .. versionadded:: 1.3 `np.nan` option was added. Returns ------- report : str or dict Text summary of the precision, recall, F1 score for each class. Dictionary returned if output_dict is True. Dictionary has the following structure:: {'label 1': {'precision':0.5, 'recall':1.0, 'f1-score':0.67, 'support':1}, 'label 2': { ... }, ... } The reported averages include macro average (averaging the unweighted mean per label), weighted average (averaging the support-weighted mean per label), and sample average (only for multilabel classification). Micro average (averaging the total true positives, false negatives and false positives) is only shown for multi-label or multi-class with a subset of classes, because it corresponds to accuracy otherwise and would be the same for all metrics. See also :func:`precision_recall_fscore_support` for more details on averages. Note that in binary classification, recall of the positive class is also known as "sensitivity"; recall of the negative class is "specificity". See Also -------- precision_recall_fscore_support: Compute precision, recall, F-measure and support for each class. confusion_matrix: Compute confusion matrix to evaluate the accuracy of a classification. multilabel_confusion_matrix: Compute a confusion matrix for each class or sample. Examples -------- >>> from sklearn.metrics import classification_report >>> y_true = [0, 1, 2, 2, 2] >>> y_pred = [0, 0, 2, 2, 1] >>> target_names = ['class 0', 'class 1', 'class 2'] >>> print(classification_report(y_true, y_pred, target_names=target_names)) precision recall f1-score support class 0 0.50 1.00 0.67 1 class 1 0.00 0.00 0.00 1 class 2 1.00 0.67 0.80 3 accuracy 0.60 5 macro avg 0.50 0.56 0.49 5 weighted avg 0.70 0.60 0.61 5 >>> y_pred = [1, 1, 0] >>> y_true = [1, 1, 1] >>> print(classification_report(y_true, y_pred, labels=[1, 2, 3])) precision recall f1-score support 1 1.00 0.67 0.80 3 2 0.00 0.00 0.00 0 3 0.00 0.00 0.00 0 micro avg 1.00 0.67 0.80 3 macro avg 0.33 0.22 0.27 3 weighted avg 1.00 0.67 0.80 3 NFTr@r?z:labels size, {0}, does not match size of target_names, {1}zeNumber of classes, {0}, does not match size of target_names, {1}. Try specifying the labels parameterz%s)r"r#zf1-scoresupport)rvrrur6rE)rrrr)rrrrr-z weighted avgc32K|]}t|yw)N)rF).0cns r7 z(classification_report.. s8RR8sz {:>{width}s} z {:>9}widthz zD{:>{width}s} {:>9.{digits}f} {:>9.{digits}f} {:>9.{digits}f} {:>9} )rfr] raccuracyz avgzB{:>{width}s} {:>9.{digits}} {:>9.{digits}} {:>9.{digits}f} {:>9} r r")r#rUr&r0rrjrFrfr+rHrGr ziprMitemsr<r3rarqkeys)&r;r>rvr\rur]r^r6rR labels_givenmicro_is_accuracylheadersrNrQf1srowsr!r report_dictscoresrlongest_last_line_heading name_widthrfhead_fmtreportrow_fmtrowr line_headingavg_pavg_ravg_f1rOavgrow_fmt_accuracys& r7classification_reportr sN#662NFF+FF;FFF ~vv. F#  </E6X3E OS[C ff0M,NNCK3|3D$D  MMLSSK\!2  "F3v;L0AB  *01Qq1 1G"XG %<< ;g;U;&G GC ngnncvF FF G$#Y   g &+<%L"V+L#B  '' # ufaeVRVVAY/ (,SS:Q58:Q-R(SK %z)!! 1*11 "b+.qr7:?.'..XX5QWXX?YB ))+ +&1*&=k&JK # U2(>3M@;Rs LL L Lct||\}}t||\}}}t|||t|||\}}}|d}n |j ||}t ||}|j dr@t||z |||}t||jd|jdz|zz S|dvrtt ||k7|d Std j|) a Compute the average Hamming loss. The Hamming loss is the fraction of labels that are incorrectly predicted. Read more in the :ref:`User Guide `. Parameters ---------- y_true : 1d array-like, or label indicator array / sparse matrix Ground truth (correct) labels. y_pred : 1d array-like, or label indicator array / sparse matrix Predicted labels, as returned by a classifier. sample_weight : array-like of shape (n_samples,), default=None Sample weights. .. versionadded:: 0.18 Returns ------- loss : float or int Return the average Hamming loss between element of ``y_true`` and ``y_pred``. See Also -------- accuracy_score : Compute the accuracy score. By default, the function will return the fraction of correct predictions divided by the total number of predictions. jaccard_score : Compute the Jaccard similarity coefficient score. zero_one_loss : Compute the Zero-one classification loss. By default, the function will return the percentage of imperfectly predicted subsets. Notes ----- In multiclass classification, the Hamming loss corresponds to the Hamming distance between ``y_true`` and ``y_pred`` which is equivalent to the subset ``zero_one_loss`` function, when `normalize` parameter is set to True. In multilabel classification, the Hamming loss is different from the subset zero-one loss. The zero-one loss considers the entire set of labels for a given sample incorrect if it does not entirely match the true set of labels. Hamming loss is more forgiving in that it penalizes only the individual labels. The Hamming loss is upperbounded by the subset zero-one loss, when `normalize` parameter is set to True. It is always between 0 and 1, lower being better. References ---------- .. [1] Grigorios Tsoumakas, Ioannis Katakis. Multi-Label Classification: An Overview. International Journal of Data Warehousing & Mining, 3(3), 1-13, July-September 2007. .. [2] `Wikipedia entry on the Hamming distance `_. Examples -------- >>> from sklearn.metrics import hamming_loss >>> y_pred = [1, 2, 3, 4] >>> y_true = [2, 2, 3, 4] >>> hamming_loss(y_true, y_pred) 0.25 In the multilabel case with binary label indicators: >>> import numpy as np >>> hamming_loss(np.array([[0, 1], [1, 1]]), np.zeros((2, 2))) 0.75 rrrrE)rNrrurr-rDTrrC) r#rUrrrrrMrr3rLrGrH) r;r>rurRrNrOrweight_average n_differencess r7 hamming_lossr sf#662NFF+FF;FFFFFM:,VV]KMB6 = @ !-B7 && VO6 ]# LLOfll1o - >   + +Xf. QUVWW/66v>??r9r;r>r}rurv)r}rurvct||||\}}tj|jj}tj ||d|z }t ||jd }tt|||S)aF Log loss, aka logistic loss or cross-entropy loss. This is the loss function used in (multinomial) logistic regression and extensions of it such as neural networks, defined as the negative log-likelihood of a logistic model that returns ``y_pred`` probabilities for its training data ``y_true``. The log loss is only defined for two or more labels. For a single sample with true label :math:`y \in \{0,1\}` and a probability estimate :math:`p = \operatorname{Pr}(y = 1)`, the log loss is: .. math:: L_{\log}(y, p) = -(y \log (p) + (1 - y) \log (1 - p)) Read more in the :ref:`User Guide `. Parameters ---------- y_true : array-like or label indicator matrix Ground truth (correct) labels for n_samples samples. y_pred : array-like of float, shape = (n_samples, n_classes) or (n_samples,) Predicted probabilities, as returned by a classifier's predict_proba method. If ``y_pred.shape = (n_samples,)`` the probabilities provided are assumed to be that of the positive class. The labels in ``y_pred`` are assumed to be ordered alphabetically, as done by :class:`~sklearn.preprocessing.LabelBinarizer`. `y_pred` values are clipped to `[eps, 1-eps]` where `eps` is the machine precision for `y_pred`'s dtype. normalize : bool, default=True If true, return the mean loss per sample. Otherwise, return the sum of the per-sample losses. sample_weight : array-like of shape (n_samples,), default=None Sample weights. labels : array-like, default=None If not provided, labels will be inferred from y_true. If ``labels`` is ``None`` and ``y_pred`` has shape (n_samples,) the labels are assumed to be binary and are inferred from ``y_true``. .. versionadded:: 0.18 Returns ------- loss : float Log loss, aka logistic loss or cross-entropy loss. Notes ----- The logarithm used is the natural logarithm (base-e). References ---------- C.M. Bishop (2006). Pattern Recognition and Machine Learning. Springer, p. 209. Examples -------- >>> from sklearn.metrics import log_loss >>> log_loss(["spam", "ham", "ham", "spam"], ... [[.1, .9], [.9, .1], [.8, .2], [.35, .65]]) 0.21616 r-r\r) r{r0rorYrpcliprrqr3r)r;r>r}rurvryrplosss r7log_lossrZ s\"O v" ((6<< $ $C WWVS!c' *F $f - 1 1q 1 9 9D $ K LLr9)r; pred_decisionrvrurct|||t|d}t|}tj||n|}|j dkDrU|j dkr5td|jdd|j d |j|j |jdk7rB| td td |jdd|j d |j||}t}|j||j|}tj|t }d|tj|jd|f<||}|tj||j!|jdd dz}nJt|}tj"|}t%d }|j'|dddf} ||z}d|z } tj*| dd| t-tj.| |S#t($r t)dwxYw)a Average hinge loss (non-regularized). In binary class case, assuming labels in y_true are encoded with +1 and -1, when a prediction mistake is made, ``margin = y_true * pred_decision`` is always negative (since the signs disagree), implying ``1 - margin`` is always greater than 1. The cumulated hinge loss is therefore an upper bound of the number of mistakes made by the classifier. In multiclass case, the function expects that either all the labels are included in y_true or an optional labels argument is provided which contains all the labels. The multilabel margin is calculated according to Crammer-Singer's method. As in the binary case, the cumulated hinge loss is an upper bound of the number of mistakes made by the classifier. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array-like of shape (n_samples,) True target, consisting of integers of two values. The positive label must be greater than the negative label. pred_decision : array-like of shape (n_samples,) or (n_samples, n_classes) Predicted decisions, as output by decision_function (floats). labels : array-like, default=None Contains all the labels for the problem. Used in multiclass hinge loss. sample_weight : array-like of shape (n_samples,), default=None Sample weights. Returns ------- loss : float Average hinge loss. References ---------- .. [1] `Wikipedia entry on the Hinge loss `_. .. [2] Koby Crammer, Yoram Singer. On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines. Journal of Machine Learning Research 2, (2001), 265-292. .. [3] `L1 AND L2 Regularization for Multiclass Hinge Loss Models by Robert C. Moore, John DeNero `_. Examples -------- >>> from sklearn import svm >>> from sklearn.metrics import hinge_loss >>> X = [[0], [1]] >>> y = [-1, 1] >>> est = svm.LinearSVC(random_state=0) >>> est.fit(X, y) LinearSVC(random_state=0) >>> pred_decision = est.decision_function([[-2], [3], [0.5]]) >>> pred_decision array([-2.18, 2.36, 0.09]) >>> hinge_loss([-1, 1, 1], pred_decision) 0.30 In the multiclass case: >>> import numpy as np >>> X = np.array([[0], [1], [2], [3]]) >>> Y = np.array([0, 1, 2, 3]) >>> labels = np.array([0, 1, 2, 3]) >>> est = svm.LinearSVC() >>> est.fit(X, Y) LinearSVC() >>> pred_decision = est.decision_function([[-1], [2], [3]]) >>> y_true = [0, 2, 3] >>> hinge_loss(y_true, pred_decision, labels=labels) 0.56 F)rXNr r-zThe shape of pred_decision cannot be 1d arraywith a multiclass target. pred_decision shape must be (n_samples, n_classes), that is (rz, z). Got: zDPlease include all labels in y_true or pass labels as third argumentzThe shape of pred_decision is not consistent with the number of classes. With a multiclass target, pred_decision shape must be (n_samples, n_classes), that is (rrr\) neg_labelz+pred_decision should be an array of floats.)outr&)rrrr0rKrrmrGrLr rcrk ones_likeboolrrarr>r fit_transformrJrr3r) r;rrvru y_true_uniquerrmarginlbinlossess r7 hinge_lossr sMrFM=A ?M & !FII(:fGMA    "LLO$B}'9'9&:;&,,- /    !4!4Q!7 7~ 7 !  Q(=+=+=*>?)//0 2 >"F ^ vf%||M638RYYv||A ' /0u%"&&t,44V\\!_bIPQRR %]3 / +##F+AqD1 Km+FZFGGFAt( FM: ;;  KIJ J Ks )H,,Ict|}t|}t|t|t|||t|d}|dk7rt d|d|j dkDrt d|j |j dkrt d |j  t||}tj||k(t}tjd|z |f}tjd|z |f}||fS#t$r7tj|}|jjd vr|d }nYwxYw) aConvert y_true and y_prob in binary classification to shape (n_samples, 2) Parameters ---------- y_true : array-like of shape (n_samples,) True labels. y_prob : array-like of shape (n_samples,) Probabilities of the positive class. sample_weight : array-like of shape (n_samples,), default=None Sample weights. pos_label : int, float, bool or str, default=None Label of the positive class. If None, `pos_label` will be inferred in the following manner: * if `y_true` in {-1, 1} or {0, 1}, `pos_label` defaults to 1; * else if `y_true` contains string, an error will be raised and `pos_label` should be explicitly specified; * otherwise, `pos_label` defaults to the greater label, i.e. `np.unique(y_true)[-1]`. Returns ------- transformed_labels : array of shape (n_samples, 2) y_prob : array of shape (n_samples, 2) r;r<r?z/The type of the target inferred from y_true is z7 but should be binary according to the shape of y_prob.r-rZrz$y_prob contains values less than 0: )OUSr)rr rr%rGrarbr'r0rKrYrrr2 column_stack)r;rtrurrRclassesrys r7)_validate_binary_probabilistic_predictionrM s]>& !F & !FffFFM: Fx 8F =fXF7 7  zz|aB6::<.QRR zz|a? ~NOO 0FC XXf )3 /F!f*f)=> __a&j&1 2F v %% ))F# ==  _ 4  I   s: D=EEauto)r;y_probarurrv scale_by_half)rurrvrct|dtjtjtjg}|j dk(s|j ddk(rt||||\}}nt||||\}}tjtj||z dzd|}|dk(r#|j dk(xs|j ddk}|r|d z}t|S) a_Compute the Brier score loss. The smaller the Brier score loss, the better, hence the naming with "loss". The Brier score measures the mean squared difference between the predicted probability and the actual outcome. The Brier score is a strictly proper scoring rule. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array-like of shape (n_samples,) True targets. y_proba : array-like of shape (n_samples,) or (n_samples, n_classes) Predicted probabilities. If `y_proba.shape = (n_samples,)` the probabilities provided are assumed to be that of the positive class. If `y_proba.shape = (n_samples, n_classes)` the columns in `y_proba` are assumed to correspond to the labels in alphabetical order, as done by :class:`~sklearn.preprocessing.LabelBinarizer`. sample_weight : array-like of shape (n_samples,), default=None Sample weights. pos_label : int, float, bool or str, default=None Label of the positive class when `y_proba.shape = (n_samples,)`. If not provided, `pos_label` will be inferred in the following manner: * if `y_true` in {-1, 1} or {0, 1}, `pos_label` defaults to 1; * else if `y_true` contains string, an error will be raised and `pos_label` should be explicitly specified; * otherwise, `pos_label` defaults to the greater label, i.e. `np.unique(y_true)[-1]`. labels : array-like of shape (n_classes,), default=None Class labels when `y_proba.shape = (n_samples, n_classes)`. If not provided, labels will be inferred from `y_true`. .. versionadded:: 1.7 scale_by_half : bool or "auto", default="auto" When True, scale the Brier score by 1/2 to lie in the [0, 1] range instead of the [0, 2] range. The default "auto" option implements the rescaling to [0, 1] only for binary classification (as customary) but keeps the original [0, 2] range for multiclass classification. .. versionadded:: 1.7 Returns ------- score : float Brier score loss. Notes ----- For :math:`N` observations labeled from :math:`C` possible classes, the Brier score is defined as: .. math:: \frac{1}{N}\sum_{i=1}^{N}\sum_{c=1}^{C}(y_{ic} - \hat{p}_{ic})^{2} where :math:`y_{ic}` is 1 if observation `i` belongs to class `c`, otherwise 0 and :math:`\hat{p}_{ic}` is the predicted probability for observation `i` to belong to class `c`. The Brier score then ranges between :math:`[0, 2]`. In binary classification tasks the Brier score is usually divided by two and then ranges between :math:`[0, 1]`. It can be alternatively written as: .. math:: \frac{1}{N}\sum_{i=1}^{N}(y_{i} - \hat{p}_{i})^{2} where :math:`y_{i}` is the binary target and :math:`\hat{p}_{i}` is the predicted probability of the positive class. References ---------- .. [1] `Wikipedia entry for the Brier score `_. Examples -------- >>> import numpy as np >>> from sklearn.metrics import brier_score_loss >>> y_true = np.array([0, 1, 1, 0]) >>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"]) >>> y_prob = np.array([0.1, 0.9, 0.8, 0.3]) >>> brier_score_loss(y_true, y_prob) 0.0375 >>> brier_score_loss(y_true, 1-y_prob, pos_label=0) 0.0375 >>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham") 0.0375 >>> brier_score_loss(y_true, np.array(y_prob) > 0.5) 0.0 >>> brier_score_loss(y_true, y_prob, scale_by_half=False) 0.075 >>> brier_score_loss( ... ["eggs", "ham", "spam"], ... [[0.8, 0.1, 0.1], [0.2, 0.7, 0.1], [0.2, 0.2, 0.6]], ... labels=["eggs", "ham", "spam"] ... ) 0.146 FrWr-r r\r&rg?) rr0r1r_r`rmrLrr{rrqr3)r;rrurrvrry brier_scores r7brier_score_lossr s@5RZZ(LG||qGMM!,1&O G]I' #G'T G]F' #G** "W,2;]K )AW]]1-=-A s  r9)r;r>rurvr:c t|dd}t|||t|dkr'd}tj|t t dSt||d||}t||}|Ltj||gtj|tj|t|gfn||f\}}tj|d \} } tj| | } | |jz } tj | t|d f} t|| d||}t d ||z z S) a :math:`D^2` score function, fraction of log loss explained. Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A model that always predicts the per-class proportions of `y_true`, disregarding the input features, gets a D^2 score of 0.0. Read more in the :ref:`User Guide `. .. versionadded:: 1.5 Parameters ---------- y_true : array-like or label indicator matrix The actuals labels for the n_samples samples. y_pred : array-like of shape (n_samples, n_classes) or (n_samples,) Predicted probabilities, as returned by a classifier's predict_proba method. If ``y_pred.shape = (n_samples,)`` the probabilities provided are assumed to be that of the positive class. The labels in ``y_pred`` are assumed to be ordered alphabetically, as done by :class:`~sklearn.preprocessing.LabelBinarizer`. sample_weight : array-like of shape (n_samples,), default=None Sample weights. labels : array-like, default=None If not provided, labels will be inferred from y_true. If ``labels`` is ``None`` and ``y_pred`` has shape (n_samples,) the labels are assumed to be binary and are inferred from ``y_true``. Returns ------- d2 : float or ndarray of floats The D^2 score. Notes ----- This is not a symmetric function. Like R^2, D^2 score may be negative (it need not actually be the square of a quantity D). This metric is not well-defined for a single sample and will return a NaN value if n_samples is less than two. FnumericrWr z9D^2 score is not well-defined with less than two samples.r4r)rLT)return_inverser&r-)rrr)rfr+r r3rr(r0 concatenate zeros_likerFrKbincountrqtile)r;r>rurvrrry_true_weights_rOy_value_indicescountsrt y_pred_nullrs r7d2_log_loss_scorer-sHr5 BFFFM:FaI c12U|# I#=&9G   NNFF+ , NNGR]]7#f+%NO P g  GX74@A [[( ;F gkkm #F''&3v;"23K# K i+-. //r9)r+)W__doc__rfnumbersrrnumpyr0 scipy.sparserrr scipy.specialr exceptionsr preprocessingr r utilsr rrrrutils._array_apirrrrrrrrrrrrutils._param_validationrrr r!r" utils._uniquer# utils.extmathr$utils.multiclassr%r&utils.validationr'r(r)r8rUr{rrrrr/rrrrrrr rrtuplerjr r4r<rKrOrRr[rrrrrrrr9r7rs "99/8    *'<R"jB&J11[&-  #'15DHNHNV..&&- !894@ #' #$$b bJ11&-& k #' &*$5j@ j@Znn&+67>&- #' )-d$_ _D11&CD1 J K  '- D1a& ! x  #'!,  t<%$tBEQQh11&%t,&-Haf=>!{ D3* %  x   #'*  ]#"]@11&- #' 37b@b@J..[&-& #' +/d4NM NMb.&&&-  #'15DL<L<^D&N. >&-CD1&#Z%9: #'  K K\..&-&  #'8