 Abstract, Gaussian processes are a popular method for stochastic function approximation in science and engineering applications. However, they are computationally expensive due to their high numerical complexity. To overcome this limitation, we propose a new approach based on ultra-flexible, compactly supported, and non-stationary kernels, which allows us to scale exact Gaussian processes beyond 5 million data points. This article was authored by Marcus M. Noak, Aaron Aurean-Krishnan, Mark D. Risser, and others.