Classification¶

It is possible to convert a regression problem into a classification problem, allowing the use of Gaussian processes.

Easily enable classification within a decorator.

The return classes of all classification decorators have a distinct structure in which the standard prediction methods are unavailable. Instead, controllers will have classify_points() and classify_fuzzy_points() which should be used. When creating new decorators, include the ClassificationMixin as a mixin for the inner class, and then decorate the inner class with Classification before returning it.

class vanguard.classification.mixin.Classification(**kwargs)[source]¶

Converts a decorator class to expect a classification task.

When used as a decorator for the output of classification decorators, this decorator automatically ‘closes’ the standard posterior methods.

__init__(**kwargs)[source]¶

Initialise self.

Parameters:

framework_class – All unexpected/overwritten methods are relative to this class.
required_decorators – A set (or other iterable) of decorators which must have been applied before (i.e. below) this one.
ignore_methods – If these method names are found to have been added or overwritten, then an error or warning will not be raised.
ignore_all – If True, all unexpected/overwritten methods will be ignored.
raise_instead – If True, unexpected/overwritten methods will raise errors instead of emitting warnings.

verify_decorated_class(cls)[source]¶

Verify that a class can be decorated by this instance.

Parameters:: cls (type[Any]) – The class to be decorated.
Raises:: TypeError – If cls is not a subclass of the framework_class, or if another classification decorator has already been applied.
Return type:: None

class vanguard.classification.mixin.ClassificationMixin[source]¶

Mixin that provides the base methods for classification.

classify_fuzzy_points(x, x_std)[source]¶

Classify fuzzy points.

Parameters:

x (Union[float, ndarray[tuple[Any, ...], dtype[floating]]]) – (n_predictions, n_features) The predictive inputs.
x_std (Union[float, ndarray[tuple[Any, ...], dtype[floating]]]) –
The input noise standard deviations:
- array_like[float]: (n_features,) The standard deviation per input dimension for the predictions,
- float: Assume homoskedastic noise.

Return type:

tuple[ndarray[tuple[Any, ...], dtype[integer]], ndarray[tuple[Any, ...], dtype[floating]]]

Returns:

(predictions, certainties) where:

predictions: (n_predictions,) The posterior predicted classes.
certainties: (n_predictions,) The posterior predicted class probabilities.

classify_points(x)[source]¶

Classify points.

Parameters:

x (Union[float, ndarray[tuple[Any, ...], dtype[floating]]]) – (n_predictions, n_features) The predictive inputs.

Return type:

tuple[ndarray[tuple[Any, ...], dtype[integer]], ndarray[tuple[Any, ...], dtype[floating]]]

Returns:

(predictions, certainties) where:

predictions: (n_predictions,) The posterior predicted classes.
certainties: (n_predictions,) The posterior predicted class probabilities.

Binary Classification¶

class vanguard.classification.binary.BinaryClassification(**kwargs)[source]¶

Bases: Decorator

A decorator which enables binary classification.

Note

Although the y_std parameter is not currently used in classification, it must still be passed. This is likely to change in the future, and so the type must still be correct. Passing y_std=0 is suggested.

Note

When used in conjunction with the BernoulliLikelihood class, the probit likelihood is calculated in closed form by applying the following formula [Kuss05]:

\[q(y_*=1\mid\mathcal{D},{\pmb{\theta}},{\bf x_*}) = \int {\bf\Phi}(f_*)\mathcal{N}(f_*\mid\mu_*,\sigma_*^2)df_* = {\bf\Phi}\left( \frac{\mu_*}{\sqrt{1 + \sigma_*^2}} \right ).\]

This means that the predictive uncertainty is taken into account.

Note

The VariationalInference decorator is required for this decorator to be applied.

Example:

>>> from gpytorch.likelihoods import BernoulliLikelihood
>>> from gpytorch.mlls import VariationalELBO
>>> import numpy as np
>>> from vanguard.kernels import ScaledRBFKernel
>>> from vanguard.vanilla import GaussianGPController
>>>
>>> @BinaryClassification()
... @VariationalInference()
... class BinaryClassifier(GaussianGPController):
...     pass
>>>
>>> train_x = np.array([0, 0.1, 0.9, 1])
>>> train_y = np.array([0, 0, 1, 1])
>>>
>>> gp = BinaryClassifier(train_x, train_y, ScaledRBFKernel, y_std=0.0,
...                       likelihood_class=BernoulliLikelihood,
...                       marginal_log_likelihood_class=VariationalELBO)
>>> loss = gp.fit(100)
>>>
>>> test_x = np.array([0.05, 0.95])
>>> predictions, probs = gp.classify_points(test_x)
>>> predictions.tolist()
[0, 1]

Parameters:

kwargs (Any)

__init__(**kwargs)[source]¶

Initialise self.

Parameters:: kwargs (Any) – Keyword arguments passed to Decorator.

property safe_updates: dict[type, set[str]]¶: Get a dictionary (class -> set[names]) of overrides/new methods that we consider “safe”.

Categorical Classification¶

class vanguard.classification.categorical.CategoricalClassification(num_classes, **kwargs)[source]¶

Bases: Decorator

Enable categorical classification with more than two classes.

Note

The VariationalInference and Multitask decorators are required for this decorator to be applied.

Example:

>>> from gpytorch.likelihoods import BernoulliLikelihood
>>> from gpytorch.kernels import RBFKernel
>>> from gpytorch.mlls import VariationalELBO
>>> import numpy as np
>>> import torch
>>> from vanguard.vanilla import GaussianGPController
>>> from vanguard.classification.likelihoods import MultitaskBernoulliLikelihood
>>>
>>> @CategoricalClassification(num_classes=3)
... @Multitask(num_tasks=3)
... @VariationalInference()
... class CategoricalClassifier(GaussianGPController):
...     pass
>>>
>>> train_x = np.array([0, 0.5, 0.9, 1])
>>> train_y = np.array([[1, 0, 0], [0, 1,0], [0, 0, 1], [0, 0, 1]])
>>> gp = CategoricalClassifier(train_x, train_y, RBFKernel, y_std=0.0,
...                            likelihood_class=MultitaskBernoulliLikelihood,
...                            marginal_log_likelihood_class=VariationalELBO)
>>> loss = gp.fit(100)
>>>
>>> test_x = np.array([0.05, 0.95])
>>> predictions, probs = gp.classify_points(test_x)
>>> predictions.tolist()
[0, 2]

Parameters:

num_classes (int)
kwargs (Any)

__init__(num_classes, **kwargs)[source]¶

Initialise self.

Parameters:

num_classes (int) – The number of target classes.
kwargs (Any) – Keyword arguments passed to Decorator.

property safe_updates: dict[type, set[str]]¶: Get a dictionary (class -> set[names]) of overrides/new methods that we consider “safe”.

class vanguard.classification.likelihoods.MultitaskBernoulliLikelihood(*args, **kwargs)[source]¶

A very simple extension of gpytorch.likelihoods.BernoulliLikelihood.

Provides an improper likelihood over multiple independent Bernoulli distributions.

Parameters:

args (Any)
kwargs (Any)

__init__(*args, **kwargs)[source]¶

Initialise self and ignore the num_tasks kwarg that may be passed to multi-task likelihoods.

Parameters:

args (Any)
kwargs (Any)

expected_log_prob(observations, function_dist, *args, **kwargs)[source]¶

Compute the expected log probability sum summing the expected log probabilities over the tasks.

Parameters:

observations (Tensor)
function_dist (Distribution)

log_marginal(observations, function_dist, *args, **kwargs)[source]¶

Compute the log probability sum summing the log probabilities over the tasks.

Parameters:

observations (Tensor)
function_dist (Distribution)

class vanguard.classification.likelihoods.SoftmaxLikelihood(*args, num_classes=None, num_tasks=None, **kwargs)[source]¶

Superficial wrapper around the GPyTorch gpytorch.likelihoods.SoftmaxLikelihood.

This wrapper allows the arg names more consistent with other likelihoods.

Parameters:

args (Any)
num_classes (Optional[int])
num_tasks (Optional[int])
kwargs (Any)

__init__(*args, num_classes=None, num_tasks=None, **kwargs)[source]¶

Initialise self.

Parameters:

args (Any) – For full signature, see gpytorch.likelihoods.SoftmaxLikelihood.
num_classes (Optional[int]) – The number of target classes.
num_tasks (Optional[int]) – Dimensionality of latent function \(\mathbf f\).
kwargs (Any) – For full signature, see gpytorch.likelihoods.SoftmaxLikelihood.

Multiclass Classification¶

class vanguard.classification.dirichlet.DirichletMulticlassClassification(num_classes, **kwargs)[source]¶

Bases: Decorator

Implements multiclass Gaussian process classification using a Dirichlet transformation.

This decorator allows multiclass classification with exact gaussian processes. The implementation is based on a GPyTorch example notebook [Maddox21] and the paper [Milios18].

Note that in the decorated class, the classify_points and classify_fuzzy_points methods both take a num_samples keyword argument - we don’t get a closed-form estimate for class probabilities, but instead approximate them with a sampling step. The num_samples parameter represents a trade-off between speed and accuracy, but the default value (taken from [Maddox21]) should be good enough for most purposes.

Example:

>>> from gpytorch.kernels import RBFKernel, ScaleKernel
>>> from gpytorch.likelihoods import DirichletClassificationLikelihood
>>> import numpy as np
>>> from vanguard.vanilla import GaussianGPController
>>>
>>> @DirichletMulticlassClassification(num_classes=3)
... class MulticlassClassifier(GaussianGPController):
...     pass
>>>
>>> class Kernel(ScaleKernel):
...     def __init__(self) -> None:
...         super().__init__(RBFKernel(batch_shape=(3,)), batch_shape=(3,))
>>>
>>> train_x = np.array([0, 0.1, 0.45, 0.55, 0.9, 1])
>>> train_y = np.array([0, 0, 1, 1, 2, 2])
>>>
>>> gp = MulticlassClassifier(train_x, train_y, Kernel, y_std=0.0,
...                           likelihood_class=DirichletClassificationLikelihood)
>>> loss = gp.fit(100)
>>>
>>> test_x = np.array([0.05, 0.5, 0.95])
>>> predictions, probs = gp.classify_points(test_x)
>>> predictions.tolist()
[0, 1, 2]

Parameters:

num_classes (int)
kwargs (Any)

__init__(num_classes, **kwargs)[source]¶

Initialise self.

Parameters:

num_classes (int) – The number of target classes.
kwargs (Any)

property safe_updates: dict[type, set[str]]¶: Get a dictionary (class -> set[names]) of overrides/new methods that we consider “safe”.

class vanguard.classification.kernel.DirichletKernelMulticlassClassification(num_classes, **kwargs)[source]¶

Bases: Decorator

Implements multiclass classification using a Dirichlet kernel method.

Based on the paper [MacKenzie14].

Warning

This decorator is EXPERIMENTAL. It may cause errors or give incorrect results, and may have breaking changes without warning.

Warning

Fuzzy classification (with classify_fuzzy_points) is not supported.

Example:

>>> from gpytorch.kernels import RBFKernel, ScaleKernel
>>> import numpy as np
>>> from vanguard.classification.likelihoods import (DirichletKernelClassifierLikelihood,
...                                                  GenericExactMarginalLogLikelihood)
>>> from vanguard.vanilla import GaussianGPController
>>>
>>> @DirichletKernelMulticlassClassification(num_classes=3, ignore_methods=("__init__",))
... class MulticlassClassifier(GaussianGPController):
...     pass
>>>
>>> class Kernel(ScaleKernel):
...     def __init__(self) -> None:
...         super().__init__(RBFKernel())
>>>
>>> train_x = np.array([0, 0.1, 0.45, 0.55, 0.9, 1])
>>> train_y = np.array([0, 0, 1, 1, 2, 2])
>>>
>>> gp = MulticlassClassifier(train_x, train_y, Kernel, y_std=0.0,
...                           likelihood_class=DirichletKernelClassifierLikelihood,
...                           marginal_log_likelihood_class=GenericExactMarginalLogLikelihood)
>>> loss = gp.fit(100)
>>>
>>> test_x = np.array([0.05, 0.5, 0.95])
>>> predictions, probs = gp.classify_points(test_x)
>>> predictions.tolist()
[0, 1, 2]

Parameters:

num_classes (int)
kwargs (Any)

__init__(num_classes, **kwargs)[source]¶

Initialise self.

Parameters:

num_classes (int) – The number of target classes.
kwargs (Any) – Keyword arguments passed to Decorator.

property safe_updates: dict[type, set[str]]¶: Get a dictionary (class -> set[names]) of overrides/new methods that we consider “safe”.

class vanguard.classification.likelihoods.DirichletKernelClassifierLikelihood(num_classes, alpha=None, learn_alpha=False, alpha_prior=None, alpha_constraint=Positive())[source]¶

A pseudo Dirichlet likelihood.

Parameters:

num_classes (int)
alpha (Union[float, ndarray[tuple[Any, ...], dtype[floating]], None])
learn_alpha (bool)
alpha_prior (Optional[Prior])
alpha_constraint (Optional[Interval])

__init__(num_classes, alpha=None, learn_alpha=False, alpha_prior=None, alpha_constraint=Positive())[source]¶

Initialise self.

Parameters:

num_classes (int) – The number of classes in the data.
alpha (Union[float, ndarray[tuple[Any, ...], dtype[floating]], None]) – The Dirichlet prior concentration. If a float, this will be assumed homogenous.
learn_alpha (bool) – If to learn the Dirichlet prior concentration as a parameter.
alpha_prior (Optional[Prior]) – Only used if :param:learn_alpha = True. The noise prior to use when learning the Dirichlet prior concentration.
alpha_constraint (Optional[Interval]) – Only used if :param:learn_alpha = True. The constraint to apply to the learned value of alpha for the Dirichlet prior concentration.

property alpha: float | ndarray[tuple[Any, ...], dtype[floating]] | Tensor | None¶: Return the Dirichlet prior concentration \(lpha\).

class vanguard.classification.likelihoods.DirichletKernelDistribution(label_matrix, kernel_matrix, alpha)[source]¶

A pseudo Dirichlet distribution with the log probability modified.

Parameters:

label_matrix (Union[Tensor, LinearOperator])
kernel_matrix (Union[Tensor, LinearOperator])
alpha (Union[Tensor, float])

__init__(label_matrix, kernel_matrix, alpha)[source]¶

Initialise self.

Parameters:

label_matrix (Union[Tensor, LinearOperator]) – (n_data_points,``n_classes``) A binary indicator matrix encoding the class to which each data point belongs.
kernel_matrix (Union[Tensor, LinearOperator]) – (n_data_points,``n_data_points``) The evaluated kernel matrix.
alpha (Union[Tensor, float]) – (n_classes,) The Dirichlet prior concentration parameters.

log_prob(value)[source]¶

Returns the log of the probability density/mass function evaluated at value.

Args:: value (Tensor):

Parameters:: value (Tensor)
Return type:: Tensor

class vanguard.classification.likelihoods.GenericExactMarginalLogLikelihood(likelihood, model)[source]¶

A lightweight modification of gpytorch.mlls.ExactMarginalLogLikelihood.

This removes some RuntimeErrors that prevent use with non-Gaussian likelihoods even when it is possible to do so.

Parameters:

likelihood (Union[_GaussianLikelihoodBase, DirichletKernelClassifierLikelihood])
model (ExactGP)

__init__(likelihood, model)[source]¶

Initialise self.

Parameters:

likelihood (Union[_GaussianLikelihoodBase, DirichletKernelClassifierLikelihood]) – The Gaussian likelihood for the model.
model (ExactGP) – The exact GP.

forward(function_dist, target, *params, **kwargs)[source]¶

Compute the MLL given \(p(\mathbf f)\) and \(\mathbf y\).

Parameters:

function_dist (MultivariateNormal) – \(p(\mathbf f)\) the outputs of the latent function (the gpytorch.models.ExactGP).
target (Tensor) – \(\mathbf y\) The target values.

Return type:

Tensor

Returns:

Exact MLL. Output shape corresponds to batch shape of the model/input data.

class vanguard.classification.models.InertKernelModel(train_inputs, train_targets, covar_module, mean_module, likelihood, num_classes, **_)[source]¶

An inert model wrapping a kernel matrix.

Uses a given kernel for prior and posterior and returns a dummy distribution holding the kernel matrix.

Parameters:

train_inputs (Optional[Tensor])
train_targets (Optional[Tensor])
covar_module (Kernel)
mean_module (Optional[Mean])
likelihood (Likelihood)
num_classes (int)
_ (Any)

__init__(train_inputs, train_targets, covar_module, mean_module, likelihood, num_classes, **_)[source]¶

Initialise self.

Note that while arbitrary keyword arguments are accepted, they are not inspected or used. This is to allow passing keyword parameters that are required by other GP models (e.g. rng) without raising a TypeError, which allows more generic code.

Parameters:

train_inputs (Optional[Tensor]) – (n_samples, n_features) The training inputs (features).
train_targets (Optional[Tensor]) – (n_samples,) The training targets (response).
covar_module (Kernel) – The prior kernel function to use.
mean_module (Optional[Mean]) – Not used, remaining in the signature for compatibility.
likelihood (Likelihood) – Likelihood to use with model.
num_classes (int) – The number of classes to use.
_ (Any)

train(mode=True)[source]¶

Set to training mode, if data is not None.

Parameters:: mode (bool)
Return type:: ExactGPModel