In this thesis, we tackle numerous theoretical and algorithmic challenges around variants of the Optimal Transport (OT) problem. We first investigate the use of the Sliced Wasserstein distance and the Mixture Wasserstein distances for different generative tasks, which lead to theoretical developments around their optimisation. Leveraging novel results from the field of non-smooth and non-convex optimisation, we show the convergence of (Stochastic) Gradient Descent methods for numerical resolution of the arising optimisation problems. We also introduce new OT frameworks which constrain the admissible set of OT maps and plans, first in the form of a problem finding a map that best transforms a source distribution into a target one under regularity constraints, and second as variants of the Sliced Wasserstein distance which constrain the admissible plans to satisfy some contraints related to their projections onto directions. Finally, we introduce generalisations of the Wasserstein barycentre problem, which defines barycentres of measures on different spaces and with respect to general costs, beyond the squared Euclidean distance. We provide a fixed-point algorithm for the numerical resolution of these barycentre problems with convergence guarantees.

We also provide contributions to the field of Reproducing Kernel Hilbert Spaces (RKHS), first studying the natural idea of representing gradients of convex functions within RKHS cones. Secondly, we introduce a novel explicit construction of universal kernels on compact metric spaces. We also define a new notion of approximate universality, and show that tractable discretisations of our universal kernels are approximately universal.