Any kind of statistical modeling that doesn't fit neatly into an existing "one (meta)model to rule them all" framework such as generalized linear models.
I don't think that's accurate. Sampling based approaches scale badly with data so, although there are a few exceptions, if you're tackling the problem as a hierarchical Bayesian model - which is most often what Stan is used for - you're working with a dataset with a small number of features and fewer than 10k rows.
Stan fittings can be made parallel, some models will scale linearly with the data, but in the main you won't find many big data use cases here.
That's right---Stan doesn't have any online learning facilities. It's very hard to approximate posteriors and chain them, so we don't try.
If by "big data", we're talking about too big to fit in memory, that's right. Stan's fully in-memory. Compute can be distributed and GPU-powered for matrix ops, but all of the data and parameters and the core autodiff expression graph need to fit in memory.
For "medium data", Stan's adaptive Hamiltonian Monte Carlo sampling is much more efficient and scalable to complex models and higher dimensions than Gibbs or Metropolis. I'm fitting a Covid prevalence model using a custom trend-following and mean-reverting second-order autoregression model over 400 distinct regions with weekly data that has 5M data points and 10K parameters and adjusts for sensitivity and specificity of various tests taken. It fits in a single thread using MCMC in 24 hours or so, but we can fit the model with variational inference in a couple minutes. Although variational inference often produces reasonable point estimates in bigger data settings, it doesn't reasonably quantify uncertainty. I'm also working on a genomics model for differential expression of splice variants that involves 120K measurements and just as many parameters to deal with overdispersion of biological replicates in a control and treatment group. We're using variational inference and it fits in a couple minutes for the comparitiver event probabilities we need to estimate.
You can, but it's slow and computationally intensive - you still need to be fairly sure that the sampler has converged on the true posterior (I might be a bit off with the terminology there but you know what I mean) before that can become your new prior.
Stan will get your from a prior to a posterior distribution that is best supported by your data, but typically the posterior and prior distributions will not be of the same form, so there's no loop back to make.
In the case that your model is extremely simple such that your posterior has a "conjugate prior" (i.e. the posterior and prior are the same family of distribution), this sort of loop back is possible. But where this is possible you have no reason at all to use Stan or MCMC since you can just update your posterior directly.
"Typically" depends on your problem, right? So if I know that I want to feedback predictions into the next iteration, I need to take care to structure a model that enables that, by having posteriors with same shape as priors. But from what I understand it seems a design consideration, not a fundamental limitation.
Fair enough. I read the word "application" as "field of inquiry" where I do think the sky is the limit, but it's true that Stan is primarily geared towards scientific work with small data sets.
Under the hood Stan attempts to find globally optimal parameter values for your function which you've expressed as a joint probability density. To do this it relies on the same MCMC theoretical results which indicate how the recursive process of sampling and posterior updating leads to the global optimum. The big deal about Stan is that its algorithm for doing this is state-of-the-art, and that it can work with a huge variety (including custom) density functions by utilising auto-differentiation.
Sampling is a slow approach when there are other alternatives. For example, if you are after OLS regression, you can do the equivalent with Stan but it may be an order of magnitude slower than plain OLS. Further, the calculation of your likelihood function will scale linearly with the size of the data. But adding new parameters will scale exponentially, so you may find that a model with 2 free parameters which takes 10 minutes to fit takes 2 hours with 3 parameters.
A good thing about Stan however, is that it is parallelisable so you can run it on many cores (and it will scale linearly for a good while) and you can also run it on MPI across many machines. Some regression functions with very large matrices support GPUs (although Stan requires double precision to work). So to some extent you can "throw more money at it" to get a result out and it has been used for very big data problems in astronomy for example which however utilised something like 600k cores if memory serves correctly.
Stan supports optimization (L-BFGS) to find (penalized) maximum likelihood or MAP estimates where they exist. Bayesian estimates are typically posterior means, which involve MCMC rather than optimization, and the result is usually far away from the maximum likelihood estimate in high dimensions. I wrote a case study with some simple examples here: https://mc-stan.org/users/documentation/case-studies/curse-d...
Adding new parameters scales as O(N^5/4) in HMC, whereas it scales as O(N^2) in Metropolis or Gibbs. It's quadrature that scales exponentially in dimension. There's also a constant factor for posterior correlation, which can get nasty. I regularly fit regressions for epidemiology or genomics or education with 10s or even 100s of thousands of parameters on my notebook with one core and no GPU.
MCMC or optimization can be sub-linear or super-linear in the data, depending on the statistical properties of the posterior. Some non-parametric models like Gaussian processes can be cubic in the data size, whereas regressions are often sub-linear (doubling the data doesn't double computation time) because posteriors are better behaved (more normal in the Gaussian sense) when there's more data and hence easier to explore in fewer log density and gradient evaluations.
