Utility Functions#

pyod.utils.data module#

Utility functions for manipulating data

pyod.utils.data.check_consistent_shape(X_train, y_train, X_test, y_test, y_train_pred, y_test_pred)[source]#

Internal shape to check input data shapes are consistent.

X_trainnumpy array of shape (n_samples, n_features)

The training samples.

y_trainlist or array of shape (n_samples,)

The ground truth of training samples.

X_testnumpy array of shape (n_samples, n_features)

The test samples.

y_testlist or array of shape (n_samples,)

The ground truth of test samples.

y_train_prednumpy array of shape (n_samples, n_features)

The predicted binary labels of the training samples.

y_test_prednumpy array of shape (n_samples, n_features)

The predicted binary labels of the test samples.

X_trainnumpy array of shape (n_samples, n_features)

The training samples.

y_trainlist or array of shape (n_samples,)

The ground truth of training samples.

X_testnumpy array of shape (n_samples, n_features)

The test samples.

y_testlist or array of shape (n_samples,)

The ground truth of test samples.

y_train_prednumpy array of shape (n_samples, n_features)

The predicted binary labels of the training samples.

y_test_prednumpy array of shape (n_samples, n_features)

The predicted binary labels of the test samples.

pyod.utils.data.evaluate_print(clf_name, y, y_pred)[source]#

Utility function for evaluating and printing the results for examples. Default metrics include ROC and Precision @ n

clf_namestr

The name of the detector.

ylist or numpy array of shape (n_samples,)

The ground truth. Binary (0: inliers, 1: outliers).

y_predlist or numpy array of shape (n_samples,)

The raw outlier scores as returned by a fitted model.

pyod.utils.data.generate_data(n_train=1000, n_test=500, n_features=2, contamination=0.1, train_only=False, offset=10, behaviour='old', random_state=None, n_nan=0, n_inf=0)[source]#

Utility function to generate synthesized data. Normal data is generated by a multivariate Gaussian distribution and outliers are generated by a uniform distribution. “X_train, X_test, y_train, y_test” are returned.

n_trainint, (default=1000)

The number of training points to generate.

n_testint, (default=500)

The number of test points to generate.

n_featuresint, optional (default=2)

The number of features (dimensions).

contaminationfloat in (0., 0.5), optional (default=0.1)

The amount of contamination of the data set, i.e. the proportion of outliers in the data set. Used when fitting to define the threshold on the decision function.

train_onlybool, optional (default=False)

If true, generate train data only.

offsetint, optional (default=10)

Adjust the value range of Gaussian and Uniform.

behaviourstr, default=’old’

Behaviour of the returned datasets which can be either ‘old’ or ‘new’. Passing behaviour='new' returns “X_train, X_test, y_train, y_test”, while passing behaviour='old' returns “X_train, y_train, X_test, y_test”.

New in version 0.7.0: behaviour is added in 0.7.0 for back-compatibility purpose.

Deprecated since version 0.7.0: behaviour='old' is deprecated in 0.20 and will not be possible in 0.7.2.

Deprecated since version 0.7.2.: behaviour parameter will be deprecated in 0.7.2 and removed in 0.9.0.

random_stateint, RandomState instance or None, optional (default=None)

If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.

n_nanint

The number of values that are missing (np.NaN). Defaults to zero.

n_infint

The number of values that are infinite. (np.infty). Defaults to zero.

X_trainnumpy array of shape (n_train, n_features)

Training data.

X_testnumpy array of shape (n_test, n_features)

Test data.

y_trainnumpy array of shape (n_train,)

Training ground truth.

y_testnumpy array of shape (n_test,)

Test ground truth.

pyod.utils.data.generate_data_categorical(n_train=1000, n_test=500, n_features=2, n_informative=2, n_category_in=2, n_category_out=2, contamination=0.1, shuffle=True, random_state=None)[source]#

Utility function to generate synthesized categorical data.

n_trainint, (default=1000)

The number of training points to generate.

n_testint, (default=500)

The number of test points to generate.

n_featuresint, optional (default=2)

The number of features for each sample.

n_informativeint in (1, n_features), optional (default=2)

The number of informative features in the outlier points. The higher the easier the outlier detection should be. Note that n_informative should not be less than or equal n_features.

n_category_inint in (1, n_inliers), optional (default=2)

The number of categories in the inlier points.

n_category_outint in (1, n_outliers), optional (default=2)

The number of categories in the outlier points.

contaminationfloat in (0., 0.5), optional (default=0.1)

The amount of contamination of the data set, i.e. the proportion of outliers in the data set.

shuffle: bool, optional(default=True)

If True, inliers will be shuffled which makes more noisy distribution.

random_stateint, RandomState instance or None, optional (default=None)

If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.

X_trainnumpy array of shape (n_train, n_features)

Training data.

y_trainnumpy array of shape (n_train,)

Training ground truth.

X_testnumpy array of shape (n_test, n_features)

Test data.

y_testnumpy array of shape (n_test,)

Test ground truth.

pyod.utils.data.generate_data_clusters(n_train=1000, n_test=500, n_clusters=2, n_features=2, contamination=0.1, size='same', density='same', dist=0.25, random_state=None, return_in_clusters=False)[source]#
Utility function to generate synthesized data in clusters.

Generated data can involve the low density pattern problem and global outliers which are considered as difficult tasks for outliers detection algorithms.

n_trainint, (default=1000)

The number of training points to generate.

n_testint, (default=500)

The number of test points to generate.

n_clustersint, optional (default=2)

The number of centers (i.e. clusters) to generate.

n_featuresint, optional (default=2)

The number of features for each sample.

contaminationfloat in (0., 0.5), optional (default=0.1)

The amount of contamination of the data set, i.e. the proportion of outliers in the data set.

sizestr, optional (default=’same’)

Size of each cluster: ‘same’ generates clusters with same size, ‘different’ generate clusters with different sizes.

densitystr, optional (default=’same’)

Density of each cluster: ‘same’ generates clusters with same density, ‘different’ generate clusters with different densities.

dist: float, optional (default=0.25)

Distance between clusters. Should be between 0. and 1.0 It is used to avoid clusters overlapping as much as possible. However, if number of samples and number of clusters are too high, it is unlikely to separate them fully even if dist set to 1.0

random_stateint, RandomState instance or None, optional (default=None)

If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random.

return_in_clustersbool, optional (default=False)

If True, the function returns x_train, y_train, x_test, y_test each as a list of numpy arrays where each index represents a cluster. If False, it returns x_train, y_train, x_test, y_test each as numpy array after joining the sequence of clusters arrays,

X_trainnumpy array of shape (n_train, n_features)

Training data.

y_trainnumpy array of shape (n_train,)

Training ground truth.

X_testnumpy array of shape (n_test, n_features)

Test data.

y_testnumpy array of shape (n_test,)

Test ground truth.

pyod.utils.data.get_outliers_inliers(X, y)[source]#

Internal method to separate inliers from outliers.

Xnumpy array of shape (n_samples, n_features)

The input samples

ylist or array of shape (n_samples,)

The ground truth of input samples.

X_outliersnumpy array of shape (n_samples, n_features)

Outliers.

X_inliersnumpy array of shape (n_samples, n_features)

Inliers.

pyod.utils.example module#

Utility functions for running examples

pyod.utils.example.data_visualize(X_train, y_train, show_figure=True, save_figure=False)[source]#

Utility function for visualizing the synthetic samples generated by generate_data_cluster function.

X_trainnumpy array of shape (n_samples, n_features)

The training samples.

y_trainlist or array of shape (n_samples,)

The ground truth of training samples.

show_figurebool, optional (default=True)

If set to True, show the figure.

save_figurebool, optional (default=False)

If set to True, save the figure to the local.

pyod.utils.example.visualize(clf_name, X_train, y_train, X_test, y_test, y_train_pred, y_test_pred, show_figure=True, save_figure=False)[source]#

Utility function for visualizing the results in examples. Internal use only.

clf_namestr

The name of the detector.

X_trainnumpy array of shape (n_samples, n_features)

The training samples.

y_trainlist or array of shape (n_samples,)

The ground truth of training samples.

X_testnumpy array of shape (n_samples, n_features)

The test samples.

y_testlist or array of shape (n_samples,)

The ground truth of test samples.

y_train_prednumpy array of shape (n_samples, n_features)

The predicted binary labels of the training samples.

y_test_prednumpy array of shape (n_samples, n_features)

The predicted binary labels of the test samples.

show_figurebool, optional (default=True)

If set to True, show the figure.

save_figurebool, optional (default=False)

If set to True, save the figure to the local.

pyod.utils.stat_models module#

A collection of statistical models

pyod.utils.stat_models.pairwise_distances_no_broadcast(X, Y)[source]#

Utility function to calculate row-wise euclidean distance of two matrix. Different from pair-wise calculation, this function would not broadcast.

For instance, X and Y are both (4,3) matrices, the function would return a distance vector with shape (4,), instead of (4,4).

Xarray of shape (n_samples, n_features)

First input samples

Yarray of shape (n_samples, n_features)

Second input samples

distancearray of shape (n_samples,)

Row-wise euclidean distance of X and Y

pyod.utils.stat_models.pearsonr_mat(mat, w=None)[source]#

Utility function to calculate pearson matrix (row-wise).

matnumpy array of shape (n_samples, n_features)

Input matrix.

wnumpy array of shape (n_features,)

Weights.

pear_matnumpy array of shape (n_samples, n_samples)

Row-wise pearson score matrix.

pyod.utils.stat_models.wpearsonr(x, y, w=None)[source]#

Utility function to calculate the weighted Pearson correlation of two samples.

See https://stats.stackexchange.com/questions/221246/such-thing-as-a-weighted-correlation for more information

xarray, shape (n,)

Input x.

yarray, shape (n,)

Input y.

warray, shape (n,)

Weights w.

scoresfloat in range of [-1,1]

Weighted Pearson Correlation between x and y.

pyod.utils.utility module#

A set of utility functions to support outlier detection.

pyod.utils.utility.argmaxn(value_list, n, order='desc')[source]#

Return the index of top n elements in the list if order is set to ‘desc’, otherwise return the index of n smallest ones.

value_listlist, array, numpy array of shape (n_samples,)

A list containing all values.

nint

The number of elements to select.

orderstr, optional (default=’desc’)

The order to sort {‘desc’, ‘asc’}:

  • ‘desc’: descending

  • ‘asc’: ascending

index_listnumpy array of shape (n,)

The index of the top n elements.

pyod.utils.utility.check_detector(detector)[source]#

Checks if fit and decision_function methods exist for given detector

detectorpyod.models

Detector instance for which the check is performed.

pyod.utils.utility.check_parameter(param, low=- 2147483647, high=2147483647, param_name='', include_left=False, include_right=False)[source]#

Check if an input is within the defined range.

paramint, float

The input parameter to check.

lowint, float

The lower bound of the range.

highint, float

The higher bound of the range.

param_namestr, optional (default=’’)

The name of the parameter.

include_leftbool, optional (default=False)

Whether includes the lower bound (lower bound <=).

include_rightbool, optional (default=False)

Whether includes the higher bound (<= higher bound).

within_rangebool or raise errors

Whether the parameter is within the range of (low, high)

pyod.utils.utility.generate_bagging_indices(random_state, bootstrap_features, n_features, min_features, max_features)[source]#

Randomly draw feature indices. Internal use only.

Modified from sklearn/ensemble/bagging.py

random_stateRandomState

A random number generator instance to define the state of the random permutations generator.

bootstrap_featuresbool

Specifies whether to bootstrap indice generation

n_featuresint

Specifies the population size when generating indices

min_featuresint

Lower limit for number of features to randomly sample

max_featuresint

Upper limit for number of features to randomly sample

feature_indicesnumpy array, shape (n_samples,)

Indices for features to bag

pyod.utils.utility.generate_indices(random_state, bootstrap, n_population, n_samples)[source]#

Draw randomly sampled indices. Internal use only.

See sklearn/ensemble/bagging.py

random_stateRandomState

A random number generator instance to define the state of the random permutations generator.

bootstrapbool

Specifies whether to bootstrap indice generation

n_populationint

Specifies the population size when generating indices

n_samplesint

Specifies number of samples to draw

indicesnumpy array, shape (n_samples,)

randomly drawn indices

pyod.utils.utility.get_diff_elements(li1, li2)[source]#

get the elements in li1 but not li2, and vice versa

li1list or numpy array

Input list 1.

li2list or numpy array

Input list 2.

differencelist

The difference between li1 and li2.

pyod.utils.utility.get_intersection(lst1, lst2)[source]#

get the overlapping between two lists

li1list or numpy array

Input list 1.

li2list or numpy array

Input list 2.

differencelist

The overlapping between li1 and li2.

pyod.utils.utility.get_label_n(y, y_pred, n=None)[source]#

Function to turn raw outlier scores into binary labels by assign 1 to top n outlier scores.

ylist or numpy array of shape (n_samples,)

The ground truth. Binary (0: inliers, 1: outliers).

y_predlist or numpy array of shape (n_samples,)

The raw outlier scores as returned by a fitted model.

nint, optional (default=None)

The number of outliers. if not defined, infer using ground truth.

labelsnumpy array of shape (n_samples,)

binary labels 0: normal points and 1: outliers

>>> from pyod.utils.utility import get_label_n
>>> y = [0, 1, 1, 0, 0]
>>> y_pred = [0.1, 0.5, 0.3, 0.2, 0.7]
>>> get_label_n(y, y_pred)
array([0, 1, 0, 0, 1])
pyod.utils.utility.get_list_diff(li1, li2)[source]#

get the elements in li1 but not li2. li1-li2

li1list or numpy array

Input list 1.

li2list or numpy array

Input list 2.

differencelist

The difference between li1 and li2.

pyod.utils.utility.get_optimal_n_bins(X, upper_bound=None, epsilon=1)[source]#

Determine optimal number of bins for a histogram using the Birge Rozenblac method (see [BBirgeR06] for details.)

See https://doi.org/10.1051/ps:2006001

Xarray-like of shape (n_samples, n_features)

The samples to determine the optimal number of bins for.

upper_boundint, default=None

The maximum value of n_bins to be considered. If set to None, np.sqrt(X.shape[0]) will be used as upper bound.

epsilonfloat, default = 1

A stabilizing term added to the logarithm to prevent division by zero.

optimal_n_binsint

The optimal value of n_bins according to the Birge Rozenblac method

pyod.utils.utility.invert_order(scores, method='multiplication')[source]#

Invert the order of a list of values. The smallest value becomes the largest in the inverted list. This is useful while combining multiple detectors since their score order could be different.

scoreslist, array or numpy array with shape (n_samples,)

The list of values to be inverted

methodstr, optional (default=’multiplication’)

Methods used for order inversion. Valid methods are:

  • ‘multiplication’: multiply by -1

  • ‘subtraction’: max(scores) - scores

inverted_scoresnumpy array of shape (n_samples,)

The inverted list

>>> scores1 = [0.1, 0.3, 0.5, 0.7, 0.2, 0.1]
>>> invert_order(scores1)
array([-0.1, -0.3, -0.5, -0.7, -0.2, -0.1])
>>> invert_order(scores1, method='subtraction')
array([0.6, 0.4, 0.2, 0. , 0.5, 0.6])
pyod.utils.utility.precision_n_scores(y, y_pred, n=None)[source]#

Utility function to calculate precision @ rank n.

ylist or numpy array of shape (n_samples,)

The ground truth. Binary (0: inliers, 1: outliers).

y_predlist or numpy array of shape (n_samples,)

The raw outlier scores as returned by a fitted model.

nint, optional (default=None)

The number of outliers. if not defined, infer using ground truth.

precision_at_rank_nfloat

Precision at rank n score.

pyod.utils.utility.score_to_label(pred_scores, outliers_fraction=0.1)[source]#

Turn raw outlier outlier scores to binary labels (0 or 1).

pred_scoreslist or numpy array of shape (n_samples,)

Raw outlier scores. Outliers are assumed have larger values.

outliers_fractionfloat in (0,1)

Percentage of outliers.

outlier_labelsnumpy array of shape (n_samples,)

For each observation, tells whether or not it should be considered as an outlier according to the fitted model. Return the outlier probability, ranging in [0,1].

pyod.utils.utility.standardizer(X, X_t=None, keep_scalar=False)[source]#

Conduct Z-normalization on data to turn input samples become zero-mean and unit variance.

Xnumpy array of shape (n_samples, n_features)

The training samples

X_tnumpy array of shape (n_samples_new, n_features), optional (default=None)

The data to be converted

keep_scalarbool, optional (default=False)

The flag to indicate whether to return the scalar

X_normnumpy array of shape (n_samples, n_features)

X after the Z-score normalization

X_t_normnumpy array of shape (n_samples, n_features)

X_t after the Z-score normalization

scalarsklearn scalar object

The scalar used in conversion