`sample` method - proper handling of models with no parameters

[docsteveharris]

Summary:

I get the following error when trying to use the generated quantities block to do prior predictive checks.

/opt/homebrew/Caskroom/miniforge/base/envs/stan-base/lib/python3.8/site-packages/cmdstanpy/utils.py in scan_metric(fd, config_dict, lineno)
    714     lineno += 1
    715     if not line == '# Adaptation terminated':
--> 716         raise ValueError(
    717             'line {}: expecting metric, found:\n\t "{}"'.format(lineno, line)
    718         )

ValueError: line 1045: expecting metric, found:
	 "#"

I think I am following this advice in the Stan users' guide.

data {
  int<lower = 0> N;
  vector[N] x;
}
generated quantities {
  real alpha = normal_rng(0, 1);
  real beta = normal_rng(0, 1);
  real y_sim[N] = poisson_log_rng(alpha + beta * x);
}

I have asked for help on the Stan user's forum without success so far. I wonder though if this is to do with the CmdStanPy implementation since the error is arising from utils.scan_metric?

Description:

Full details and a MWE are provided on the stan forums.
See https://discourse.mc-stan.org/t/implementing-prior-predictive-checks-in-stan/23815

Additional Information:

I can get a similar model to compile using the transformed parameters block instead.
I'm posting here since there are fewer Python than R users there are on the Stan forum, and if this is not a problem with CmdStanPy then I can go back to those forums confirming that I've checked here.

Current Version:

Python implementation: CPython
Python version       : 3.8.10
IPython version      : 7.25.0

matplotlib: 3.4.2
seaborn   : 0.11.1
cmdstanpy : 0.9.76
pandas    : 1.3.1
arviz     : 0.11.2
numpy     : 1.21.1

Many thanks

[mitzimorris]

you don't need to run standalone generated quantities here - you can just run the sampler, because your generated quantities block doesn't reference any model parameters. (that said, this is an interesting example of a likely user error, and we should add it to CmdStan and CmdStanPy test set and figure out good error messages.)

the error you're getting indicates that there's a problem with file or files which are the mcmc_sample argument to the generate method. what is that file? given that you don't need any kind of input sample here, what did you pass in?

we've just done a bunch of PR's that will improve the workflow for running standalone generated quantities, the the resulting CmdStanGQ object - #413, and as soon as the documentation has been properly updated, we will do a new CmdStanPy release.

[WardBrian]

I think this problem actually is with sample -- I was able to recreate the problem using his model and
ppc.sample({'N':20, 'x':np.zeros(20)})

Here is the output of running the model from the command line:

https://gist.github.com/WardBrian/ead54f19407c6b094fd59453b7e9be59

So, compared to a normal sample output there is no part like:

 # Adaptation terminated
 # Step size = 0.904853
 # Diagonal elements of inverse mass matrix:
 # 0.522529

[mitzimorris]

I think this is because CmdStan now runs fixed_params=TRUE whenever a model has no parameters.
the CmdStanMCMC class isn't aware of this, and fails on initialization. damn. we definitely need a unit test for this use case.

[WardBrian]

I think that the csv format is simply different if there is no model - for example

data {
  int<lower = 0> N;
  vector[N] x;
}
parameters {
    real dummy;
}
model {
    dummy ~ uniform(-1,1);
}
generated quantities {
  real alpha = normal_rng(0, 1);
  real beta = normal_rng(0, 1);
  real y_sim[N] = poisson_log_rng(alpha + beta * x);
}

works (and I suppose is a workaround for @docsteveharris while we investigate this)

The only appreciable difference in the output that I can see is the addition of

# Adaptation terminated
# Step size = 0.815036
# Diagonal elements of inverse mass matrix:
# 3.55155

[WardBrian]

Ah, setting fixed_param=True does the immediate error, but it breaks things like .draws_xr() because I decided to include num_unconstrained_params in the metadata and that doesn't exist now. Whoops

[mitzimorris]

I think that the csv format is simply different if there is no model

exactly - there's neither warmup draws nor adaptation information - this isn't documented here - should be: https://mc-stan.org/docs/2_27/cmdstan-guide/stan-csv.html#mcmc-sampler-csv-output

[mitzimorris]

the fix for this involves twiddling with the logic for CmdStanMCMC method _validate_csv_files
and the logic for attribute _is_fixed_param. I can see how to fix this - once this value is properly set, we can update draws_xr logic accordingly.

I could do this over the weekend.

[WardBrian]

exactly - there's neither warmup draws nor adaptation information - this isn't documented here - should be: https://mc-stan.org/docs/2_27/cmdstan-guide/stan-csv.html#mcmc-sampler-csv-output

I'm seeing "Note that when the sampler is run with arguments algorithm=fixed_param, this section will be missing." at the bottom of the section on "Comment rows for adaptation", but it certainly could be clearer.

Will the fix mean num_unconstrained_params is populated always, or should I look into making the draws_xr function more robust?

[mitzimorris]

technically, if _is_fixed_params, then num_unconstrained_params = 0.
so all we need is a little conditional logic - the should be straightforward unless there's something I don't understand about xarray - is there something I'm missing?

[WardBrian]

No, I just wasn't sure of that fact -- sounds good to me