Hierarchical GPs with Bayesian Hyperparameters

Enable Bayesian hyperparameters in a Gaussian process.

The hierarchical module contains decorators to implement Bayesian treatment of hyperparameters using variational inference, as seen in [Lalchand20], as well as Laplace approximation treatment.

Variational Hierarchical

Contains the VariationalHierarchicalHyperparameters decorator.

class vanguard.hierarchical.variational.VariationalHierarchicalHyperparameters(num_mc_samples=100, variational_distribution_class=<class 'gpytorch.variational.cholesky_variational_distribution.CholeskyVariationalDistribution'>, **kwargs)

Bases: BaseHierarchicalHyperparameters

Convert a controller so that variational inference is performed over its hyperparameters.

Note that only those hyperparameters specified using the BayesianHyperparameters decorator will be included for variational inference. The remaining hyperparameters will be inferred as point estimates.

Example:

>>> from gpytorch.kernels import RBFKernel
>>> import numpy as np
>>> import torch
>>> import torch.random
>>> from vanguard.vanilla import GaussianGPController
>>> from vanguard.hierarchical import (BayesianHyperparameters,
...                                    VariationalHierarchicalHyperparameters)
>>>
>>> @VariationalHierarchicalHyperparameters(num_mc_samples=50)
... class HierarchicalController(GaussianGPController):
...     pass
>>>
>>> @BayesianHyperparameters()
... class BayesianRBFKernel(RBFKernel):
...     pass
>>>
>>> train_x = torch.tensor([0, 0.5, 0.9, 1])
>>> rng = torch.Generator(device=train_x.device).manual_seed(1234)
>>> train_y = torch.normal(mean=1 / (1 + train_x), std=torch.ones_like(train_x) * 0.005, generator=rng)
>>> gp = HierarchicalController(train_x, train_y, BayesianRBFKernel, y_std=0.0)
>>> loss = gp.fit(100)
>>>
>>> test_x = torch.tensor([0.05, 0.95])
>>> mean, lower, upper = gp.posterior_over_point(test_x).confidence_interval()
>>> (upper > 1/(1 + test_x)).all().item(), (lower < 1/(1 + test_x)).all().item()
(True, True)

Parameters:

• num_mc_samples (int)

• variational_distribution_class (Optional[_VariationalDistribution])

• kwargs (Any)

__init__(num_mc_samples=100, variational_distribution_class=<class 'gpytorch.variational.cholesky_variational_distribution.CholeskyVariationalDistribution'>, **kwargs)

Initialise self.

Parameters:

• num_mc_samples (int) – The number of Monte Carlo samples to use when approximating intractable integrals in the variational ELBO and the predictive posterior.

• variational_distribution_class (Optional[_VariationalDistribution]) – The variational distribution to use for the raw hyperparameters’ posterior. Defaults to CholeskyVariationalDistribution.

• kwargs (Any)

property safe_updates: dict[type, set[str]]

Get a dictionary (class -> set[names]) of overrides/new methods that we consider “safe”.

Laplace Hierarchical

Implementation of tempered Laplace approximation approach to Bayesian hyperparameters.

class vanguard.hierarchical.laplace.LaplaceHierarchicalHyperparameters(num_mc_samples=100, temperature=None, uv_cutoff=0.001, **kwargs)

Bases: BaseHierarchicalHyperparameters

Convert a controller so that Bayesian inference is performed over its hyperparameters.

A post-hoc Laplace approximation is to obtain an approximation hyperparameter posterior. Note that only those hyperparameters specified using the BayesianHyperparameters decorator will be included for Bayesian inference. The remaining hyperparameters will be inferred as point estimates.

Example:

>>> from gpytorch.kernels import RBFKernel
>>> import numpy as np
>>> import torch
>>> from vanguard.vanilla import GaussianGPController
>>> from vanguard.hierarchical import (BayesianHyperparameters,
...                                    LaplaceHierarchicalHyperparameters)
>>>
>>> @LaplaceHierarchicalHyperparameters(num_mc_samples=50)
... class HierarchicalController(GaussianGPController):
...     pass
>>>
>>> @BayesianHyperparameters()
... class BayesianRBFKernel(RBFKernel):
...     pass
>>>
>>> train_x = torch.tensor([0, 0.5, 0.9, 1])
>>> rng = torch.Generator(device=train_x.device).manual_seed(1234)
>>> train_y = torch.normal(mean=1 / (1 + train_x), std=torch.ones_like(train_x) * 0.005, generator=rng)
>>> gp = HierarchicalController(train_x, train_y, BayesianRBFKernel, y_std=0.0)
>>> loss = gp.fit(100)
>>>
>>> test_x = torch.tensor([0.05, 0.95])
>>> mean, lower, upper = gp.posterior_over_point(test_x).confidence_interval()
>>> (upper > 1/(1 + test_x)).all().item(), (lower < 1/(1 + test_x)).all().item()
(True, True)

Parameters:

• num_mc_samples (int)

• temperature (Optional[float])

• uv_cutoff (float)

• kwargs (Any)

__init__(num_mc_samples=100, temperature=None, uv_cutoff=0.001, **kwargs)

Initialise self.

Parameters:

• num_mc_samples (int) – The number of Monte Carlo samples to use when approximating intractable integrals in the variational ELBO and the predictive posterior.

• temperature (Optional[float]) – The (inverse) scale for tempering the posterior, for balancing exploration and exploitation of the target distribution. If None, it’s set automatically using a trace rescaling heuristic.

• uv_cutoff (float) – The cutoff for eigenvalues in computing the eigenbasis and spectrum of the Hessian. For eigenvalues below this cutoff, the Hessian inverse eigenvalues are set to a fixed small jitter value.

• kwargs (Any) – Keyword arguments passed to Decorator.

property safe_updates: dict[type, set[str]]

Get a dictionary (class -> set[names]) of overrides/new methods that we consider “safe”.

Base Hierarchical

Contains the BaseHierarchicalHyperparameters decorator.

class vanguard.hierarchical.base.BaseHierarchicalHyperparameters(num_mc_samples=100, **kwargs)

Bases: Decorator

Convert a controller so that Bayesian inference is performed over its hyperparameters.

Note that only those hyperparameters specified using the BayesianHyperparameters decorator will be included for Bayesian inference. The remaining hyperparameters will be inferred as point estimates.

Parameters:

• num_mc_samples (int)

• kwargs (Any)

__init__(num_mc_samples=100, **kwargs)

Initialise self.

Parameters:

• num_mc_samples (int) – The number of Monte Carlo samples to use when approximating intractable integrals in the variational ELBO and the predictive posterior.

• kwargs (Any)

verify_decorated_class(cls)

Verify that a class can be decorated by this instance.

Parameters:

cls (type[GPController]) – The class to be decorated.

Raises:

• TypeError – If cls is not a subclass of the framework_class.

• TopmostDecoratorError – If cls is already decorated with a TopMostDecorator.

• MissingRequirementsError – If cls is missing a required decorator.

Return type:

None

vanguard.hierarchical.base.extract_bayesian_hyperparameters(controller)

Pull hyperparameters and any point-estimate kernels from a controller’s mean, kernel and likelihood.

Parameters:

controller (GPController)

Return type:

tuple[list, list]

vanguard.hierarchical.base.set_batch_shape(kwargs, module_name, batch_shape)

Set the batch shape in kwargs dictionary which may not exist.

Parameters:

• kwargs (Any)

• module_name (str)

• batch_shape (tuple[int, ...])

Return type:

None

Hierarchical Bayesian Kernels and Means

Enable Bayesian hyperparameters in a controller.

Applying the BayesianHyperparameters to a module will make its parameters Bayesian, so posteriors can be inferred over them rather than point estimates.

class vanguard.hierarchical.module.BayesianHyperparameters(ignored_parameters=frozenset({}), prior_means=None, prior_variances=None)

A decorator to apply to modules (means and kernels).

It converts the module’s point-estimate hyperparameters into Bayesian hyperparameters over which inference can be performed.

Note

Modules decorated with this decorator must be used in a controller decorated with a subclass of BaseHierarchicalHyperparameters. Using in a controller without this decorator or in raw GPyTorch will lead to errors.

Note

This decorator will automatically descend into sub-modules of the class to which it is applied only if those module are not passed as arguments to the __init__. That is, only parameters that are directly created by the init of the class will be affected (even if they are buried in further sub-modules).

Parameters:

• ignored_parameters (Optional[Iterable[str]])

• prior_means (Optional[dict])

• prior_variances (Optional[dict])

__init__(ignored_parameters=frozenset({}), prior_means=None, prior_variances=None)

Initialise self.

Parameters:

• ignored_parameters (Optional[Iterable[str]]) – Names of module hyperparameters which should not be converted to Bayesian parameters.

• prior_means (Optional[dict]) – Dict of mean values for the prior distributions.

• prior_variances (Optional[dict]) – Dict of variances for the prior distributions.

Bayesian Hyperparameters

Hyperparameters are held within the controller inside a HyperparameterCollection instance, comprised of instances of the lightweight BayesianHyperparameter class.

class vanguard.hierarchical.collection.HyperparameterCollection(module_hyperparameter_pairs, sample_shape, variational_distribution_class)

Represents a collection of hyperparameters for a controller.

This class will delete the original torch parameters for the hyperparameters so that they can be replaced by batches of parameters representing samples from a distribution over those hyperparameters.

Parameters:

• module_hyperparameter_pairs (list[tuple[Module, BayesianHyperparameter]])

• sample_shape (Size)

• variational_distribution_class (type[_VariationalDistribution])

__init__(module_hyperparameter_pairs, sample_shape, variational_distribution_class)

Initialise self.

Parameters:

• module_hyperparameter_pairs (list[tuple[Module, BayesianHyperparameter]]) – A list of (module, hyperparameter) pairs.

• sample_shape (Size) – The shape of the sample tensor.

• variational_distribution_class (type[_VariationalDistribution]) – The variational distribution to use for the raw hyperparameters’ posterior.

kl_term()

Compute the KL divergence term in the ELBO.

Return type:

Tensor

sample_and_update()

Sample from the collection, and update the hyperparameters.

Return type:

None

class vanguard.hierarchical.hyperparameter.BayesianHyperparameter(raw_name, raw_shape, constraint, prior_mean, prior_variance)

Represents a single Bayesian hyperparameter.

Parameters:

• raw_name (str)

• raw_shape (Size)

• constraint (Optional[Interval])

• prior_mean (float)

• prior_variance (float)

__init__(raw_name, raw_shape, constraint, prior_mean, prior_variance)

Initialise self.

Parameters:

• raw_name (str) – The raw name for the parameter.

• raw_shape (Size) – The shape of the raw parameter.

• constraint (Optional[Interval]) – The constraint (if any) placed on the parameter.

• prior_mean (float) – The mean of the diagonal normal prior on the raw parameter.

• prior_variance (float) – The variance of the diagonal normal prior on the raw parameter.

numel()

Return the number of elements in the parameter.

Return type:

int