Bayesian Inference with Constraints

In many applications, the reconstruction must satisfy certain constraints, e.g., material densities are positive, image intensities are between 0 and 1, or the energy of a signal is bounded. Adding such prior information to a Bayesian model improves the meaningfulness of the resulting posterior distribution.

A common method to model constraints in a prior distribution is truncation: choose an unconstrained prior distribution and remove all probability outside of the constraint set. Another method is to reparametrize the signal, e.g., if the signal x is assumed to be positive, then it can be written as x = exp(z) and one can use an unconstrained prior distribution for z. These methods focus on the interior of the constraint set, but sometimes the signals of interest lie on the boundary of the constraint set.

As an alternative, one can first sample from an unconstrained posterior distribution and then apply a suitable projection onto a convex constraint set C. The resulting implicit posterior will consist of the original density in the interior of the constraint set, and a “lower-dimensional” distribution on the boundary, see the illustration below.

If the forward model is linear and both the likelihood and prior are Gaussian distributions, then the unconstrained sampling and projection can be done in a single step by repeatedly solving randomized constrained linear least square problems of the form

min_x∈C || A x - b_s ||₂² + α || x - c_s ||₂² ,

where b_s and c_s are samples from suitable distributions. The solutions to these optimization problems will be samples of a projected posterior.

Consider the problem of deblurring a noisy 1D signal with prior knowledge that the signal lies in the interval [0,1]. The figure above shows the pointwise median and 99% credible interval for such a problem using no constraints (left), only the nonnegativity constraint (middle), and the box constraint (right). The projection framework significantly improves the quality of the median near the boundary.

This feature is now available in our CUQIpy software.

See the SIAM-ASA paper Everink, Dong & Andersen (2023).

The Horseshoe Prior for Edge-Preserving Reconstruction

If we want to reconstruct an image with sharp edges, we can use a Cauchy or Laplace distribution of the difference between neighbor pixels as the prior (this is related to total variation regularization). Unfortunately, these priors are computationally demanding. As a computationally attractive alternative, we can use a so-called horseshoe prior that resembles the Cauchy and Laplace priors - see the figure below where the horseshoe prior's density is guaranteed to lie in the shaded band.

The main advantage of the horseshoe prior is that it imposes a conditionally Gaussian distribution on the differences, which allows more efficient computations because Gaussians can be handled with efficient least-squares methods. The difficulty is that the horseshoe prior has hyperpriors with heavy tails, and to alleviate this issue we propose an extended horseshoe prior that uses a scale-mixture representation of the heavy-tailed hyperpriors.

For the posterior, we exploit the extended horseshoe prior to compute conditional distributions for the associated parameters that can be sampled in closed form due to conjugacy. This allows the application of a Gibbs sampler.

Here we illustrate our method with a 1D deconvolution problem for two different noise levels, and we compare with posterior statistics obtained with a Laplace-difference prior proposed by Uribe, Bardsley, Dong, Hansen & Riis (2022).  Compared to the Laplace-difference prior, we see that the horseshoe prior gives a sharper posterior (dotted lines) and a lower posterior uncertainty (the shaded area shows the 95% credibility interval). The solid line shows the ground truth.

The horseshoe prior originated in the statistical community, and here we introduce it in the setting of Bayesian inverse problem - see the paper by Uribe, Dong & Hansen (2023).

Boundary Detection in X-Ray CT Applications

In 2D CT, every X-ray going through the object is characterized by its slope θ with the horizontal axis. In practice, a full-angle geometry where θ covers the entire interval [0,π) is not always possible. This could be due to obstacles in the CT measurement setup or to avoid unhealthy levels of radiation exposure. In such cases, a limited-angle geometry is needed, i.e., θ belongs to the reduced interval [0,θ_max) with θ_max ≪ π. In such a setup, the quality of reconstructed image is compromised when conventional reconstruction methods are applied.

In many X-ray CT applications, the boundaries of objects - as well as their roughness - contain valuable information, e.g., to differentiate between benign and malignant tumors. Confidence in estimating such features helps with an accurate diagnosis and designing an effective treatment plan. A common approach in detecting boundaries is via an image segmentation post-processing step, but the quality of the estimated boundaries is compromised in the limited-angel case.

We provide a novel goal-oriented, Bayesian framework for the limited-angel CT problem, in which we reconstruct the boundaries directly from the data without the need for reconstructing the image. Moreover, we quantify the uncertainty of the boundaries and their roughness. This approach avoids error propagation and reduces the dimensionality of the problem from finding a 2D image to a 1D boundary of a region.

Our key ingredient is a hierarchical Bayesian approach to estimating the roughness of a scalar function, based on a Whittle-Matérn covariance prior and the level of fractional differentiability. We also developed an efficient FFT-based computational method for implementing this. Numerical results indicate that this framework is a promising and effective tool for solving a broad range of inverse problems where the regularity of functions carries critical information.

The above plots show the performance of our method in estimating the boundaries of an object in the limited-angle setup with different θ_max. The arrows indicate the angles of the X-rays and thus region of the limited-angle interval [0, θ_max). The estimated boundaries are shown in dark blue color, and the light blue region indicates the uncertainty in this estimation. The smaller the θ_max the larger the uncertainty region - which is consistent with the results from microlocal analysis.

For more details, see the SIAM/ASA paper by Afkham, Dong & Hansen (2023). A related paper by Afkham, Riis, Dong & Hansen is submitted.

Defect Detection in X-Ray CT of Subsea Pipes

X-ray computed tomography (CT) is used to monitor the condition of subsea oil or gas pipes in operation, in order to detect defects that might cause leaks. In this example, we use data obtained in the test facilities at FORCE Technology. Computations are done with our CUQIpy software which draws upon the Core Imaging Library (CIL) for the CT models.

We formulate a Bayesian inverse problem with built-in defect detection. The goal is to detect defects and quantify their uncertainties. We express the CT problem as y = A (x+d) + e, where y is the measured X-ray absorption, e is data noise, and A is the linear forward model representing the physics and geometry of the measurements. We use a novel representation x + d of the unknown image to be reconstructed, expressed as a sum of two images; x contains the pipe structure and d contains potential defects.

In the Bayesian setting we formulate the joint posterior distribution p(x,d | y) ∝ p(y | x,d) p(x) p(d), where p(x,d | y) is the joint posterior distribution representing the solution to the CT problem, p(y | x,d) is the likelihood that represents the data misfit, while p(x) and p(d) are prior distributions representing any knowledge we have about the unknown images.

We impose priors that promote the structures we are looking for. For the pipe structure x, we use the structural Gaussian prior proposed by Christensen, Riis, Uribe & Jørgensen (2023). The above figure shows a sample from this prior which promotes the known layered pipe structure. We expect small and few defects, and therefore we impose a prior that promotes both sparsity and correlation (a gamma Markov random field) in the defect image d as described in the paper by Christensen, Riis, Pereyra & Jørgensen (2023). We explore the conditional posterior distributions p(x | y,d) and p(d | y,x) using Gibbs sampling.

The figure above shows the means of the x-samples (left), the d-samples (middle), and their sum with annotation (right). These results indicate that our methodology has successfully separated the defects from the overall pipe structure. In the figure below we zoom-in on the defect reconstruction d for the annotated defects indicated in the figure above, for further investigation of these defects. The figure shows the mean of the posterior samples, as well as related UQ in the form of the standard deviation of the samples. Engineers can use these results to identify critical defects in the subsea pipes.

This is joint work with Prof. Marcelo Pereyra from Heriot-Watt University. The example was created by PhD student Silja L. Christensen.

UQ with Nonnegativity Constraints

Nonnegativity constraints appear in many applications where the underlying physics dictates that the solution cannot be negative. This is the case, e.g., for absorption coefficients in computed tomography, image intensities in astronomical imaging, and wave velocities in seismic travel-time tomography. Hence, nonnegativity constraints are very important for producing meaningful reconstructions.

We considers computational aspects of UQ with nonnegativity constraints x ≥ 0. If we replace the nonnegativity constraint with a positivity constraint, x > 0, then we can handle this constraint within the "standard" UQ framework with a transformation of variables x = exp(z). In this work we specifically address situations where we expect the existence of zero elements in the solution – in fact, in some imaging problems a substantial fraction of the solution's elements/pixels may be zero. Therefore we use a nonnegative Gaussian distribution.

The above figure shows a simple Gauss distribution (left), a truncated Gauss distribution with zero probability for x = 0 (middle), and a nonnegative Gaussian distribution with nonzero probability for x = 0 (right).

We use a Markov chain Monte Carlo (MCMC) method to sample the posterior with constraints, and we refer to our method as a Nonnegative Hierarchical Gibbs Sampler.

We applied our computational method to positron emission tomography (PET), where a radioactive tracer element is injected into a body and exhibits radioactive decay, resulting in photon emission. The emitted photons that leave the body are recorded by a photon detector, and the resulting mathematical reconstruction problem is equivalent to a classical X-ray CT problem. The above figure shows our simulation results.

• Top left: the mean of the computed image samples from our nonnegative hierarchical Gibbs sampler.
• Bottom left: histogram of the regularization parameters produced by the sampler.
• Top right: the MAP estimate using the mean of the regularization parameters.
• The pixel-wise standard deviations from the sampler. As expected, image pixels with higher intensity also have higher variance.

For details see the paper by Bardsley & Hansen (2020).

The Bayesian Approach to Inverse Robin Problems

Ice-sheet models are important components of long-term sea level predictions. A crucial model parameter is the basal drag coefficient, which is a spatially varying parameter that models the slipperiness of the lubricating sediment beneath the ice.

The ice flow can be modeled by a Stokes system of partial differential equations, where the basal drag coefficient β enters in a Robin boundary condition on the bottom part of the ice boundary. On the surface of the ice, we make N observations Y_i of the ice velocities at locations X_i picked uniformly at random:
     Y_i = u(X_i) + ε_i ,   i = 1,...,N .
Here, u is the ice velocity that depends on β and ε_i is Gaussian noise. Estimation of β is a nonlinear and severely ill-posed inverse problem known as an inverse Robin problem.

From a mathematical point of view little is known about the reliability of the Bayesian approach for such nonlinear inverse problems. But if we use Gaussian priors with a favorable covariance structure then more can be said. As an example, consider two Gaussian priors suited for estimating β on the interval [0,1]: the Matérn prior and the squared-exponential Gaussian prior, samples of which are shown below.

For these two choices of priors, we can guarantee that the Bayesian approach is reliable when applied to a simplified inverse Robin problem. If the ground truth β is Sobolev regular of some index, then the posterior mean from a suitable Matérn prior converges to β (in prob.) at a logarithmic rate as N → ∞. Similarly, if β is analytic (very smooth) then we have convergence for the squared-exponential prior at rate N^−σ as N → ∞.

The figure below shows the ground truth, the conditional mean, and samples from the posterior using the Matérn prior (top) and the squared-exponential prior (bottom). We see that there is less uncertainty for the latter prior in this example.

This work was carried out by PhD student Aksel Kaastrup Rasmussen and it is documented in the paper with Seizilles, Girolami & Kazlauskaite (2023). Software is available as a CUQIpy notebook.

Follow this LINK to see pictures from our team building event in the spring of 2023.

Specialist in convex optimization and numerical optimization algorithms.

Specialist in noise modeling and computational methods for image processing.

Link to portrait.

Specialist in numerical analysis, iterative solvers, and computational methods for inverse problems.

Link to portrait.

Specialist in methods and software for image reconstruction and their application in X-ray CT.

Specialist in functional analysis and physical modeling with uncertainty quantification.

Specialist in coupled-physics modeling, PDE-constrained optimization, functional analysis and electrical impedance tomography.

Link to portrait.

Executive advisor and project coordinator.

Project secretary.

Project: goal-oriented uncertainty quantification.

Project: computational UQ for PDE problems.

Project: machine learning and UQ for inverse problems.

Project: efficient and flexible computational methods.

Project: large-scale computational UQ for inverse problems.

Former Postdocs

Project: non-Gaussian priors.

Now with LUT University, Finland; homepage.

Project: Bayesian inference for inverse problems with sampling conditioned on functionals.

Project: UQ for tomographic reconstruction.

Project: computational UQ for inverse problems with implicit priors.

Project: dimensionality challenges in UQ for inverse problems.

Project: inverse problems with Besov prior.

Former PhD Students

Sept. 1, 2019 to June 30, 2023 (incl. parental leave).

Project: prior modeling in computational UQ for inverse problems.

Katrine is now a Senior Modelling Scientist with Novo Nordisk Research & Development.

Sept. 1, 2020 to Aug. 31, 2023.

Project: computational UQ of hybrid inverse problems.

Aksel was a research assistant in CUQI November-December 2023. He is now a Senior Biostatistician with H. Lundbeck A/S.

Nov. 1, 2021 to Jan. 31, 2024.

Project: model error reduction in inverse problems.

Puyuan stopped before getting his degree.

Feb. 1, 2021 to Jan.31, 2024.

Project: UQ for source localization and passive imaging in random media.

Kristoffer is now postdoc in the Villum Experiment project neXt-Ray: a super-resolution light engine
.

Professor Johnathan M. Bardsley, Department of Mathematical Sciences, University of Montana - specialist in computational methods for inverse problems and uncertainty quantification.

Associate Professor Julianne Chung, Department of Mathematics, Emory University - specialist in in computational methods for inverse problems and uncertainty quantification in imaging applications.

Associate Professor Matthias Chung, Department of Mathematics, Emory University - specialist in computational inverse problems, data analytics & learning, uncertainty quantification, and numerical optimization.

Professor Bangti Jin, Department of Mathematics, Chinese University of Hong Kong - specialist in theory and algorithms for inverse and ill-posed problems.

DTU Compute, Section for Mathematics - specialist in wavelets, frames, and signal analysis.

Professor Marcelo Pereyra, Maxwell Institute for Mathematical Science and School of Mathematical and Computer Sciences, Heriot-Watt University - specialist in statistical, analytical and machine-learning paradigms.

Professor Tanja Tarvainen, Department of Applied Physics, University of Eastern Finland - specialist in computational methods for inverse problems and uncertainty quantification in imaging applications

Research Professor Faouzi Triki, Laboratoire Jean Kuntzman, Grenoble-Alpes University - specialist in inverse problems and mathematical modeling in optics.

Postdoc Felipe Uribe, LUT University - specialist in applied probability & statistics, and large-scale sampling methods.

Research Scientist Olivier Zahm, INRIA-Grenoble - specialist in model order reduction for uncertainty quantification.