Constrains that include fixed or choice parameters raising an incorrect and confusing "Parameter does not exist in search space" exception

[petrcezner]

Hello,

I am trying to have a fixed parameter in parameter_contrains of the experiment. The experiment setup looks like this:

ax_client.create_experiment(
            name='experiment',
            parameters=[
                {
                    'name': 't1',
                    'type': 'fixed',
                    'value': t1,
                    'value_type': 'int'
                },
                {
                    'name': 't2',
                    'type': 'fixed',
                    'value': t2,
                    'value_type': 'int'
                },
                {
                    'name': 'lw',
                    'type': 'range',
                    'bounds': [0, lw_ub],
                    'value_type': 'int'
                },
                {
                    'name': 'rw',
                    'type': 'range',
                    'bounds': [0, rw_ub],
                    'value_type': 'int'
                },
                {
                    'name': 'dt2',
                    'type': 'range',
                    'bounds': [-max_delta_hit, max_delta_hit],
                    'value_type': 'int'
                }
            ],
            objective_name='mean_distance',
            minimize=True,
            parameter_constraints=[
                f'lw + rw >= {min_width}',
                f'rw + dt2 <= {data_len - t2}',
                't1 - lw >= 0',
            ],
            overwrite_existing_experiment=True
        )

The setup seems to be OK. However, when I try to call ax_client.get_next_trial(ttl_seconds=30) I get following error:

 File "MY_PYTHON_PATH\lib\site-packages\ax\core\search_space.py", line 309, in _validate_parameter_constraints
    raise ValueError(
ValueError: `t1` does not exist in search space.

The problem is that the SearchSpace class accepts only Range and Choice parameters. So I try to change the parameters from fixed to choice in the setup:

 {
    'name': 't1',
    'type': 'choice',
    'values': [t1, t1],
    # 'value_type': 'int'
},
{
    'name': 't2',
    'type': 'choice',
    'values': [t2, t2],
    # 'value_type': 'int'
},

and it doesn't work either. The names of the parameters change from t1 to t1_OH_PARAM_0, so it again creates the same exception (parameters are not found). Also, the parameter values are in the range [0, 1]. Therefore, the constrain wouldn't be satisfied.

I couldn't find any restriction of parameters in the documentation of ax.core.parameter_constraint.ParameterConstraint class.
May I ask you how to fix it? Is it even possible to have such constraints? If not, is there any workaround except checking the constraints manually in the evaluation function? This issue is probably similar to #574 or #383.

[petrcezner]

My version of AX is ax-platform==0.2.0

[lena-kashtelyan]

@petrcezner, we'll look into this shortly, thank you for reporting!

[lena-kashtelyan]

Hi, @petrcezner! Thank you very much for reporting this. This is 1) not a bug in a sense that we do intentionally disallow constraints on fixed and choice parameters, but also 2) a bug in that you get a non-informative error that does not explain what is going on and that it happens not on experiment creation, but rather during candidate generation. Schrödinger's bug : )

Re: 1), I believe that the reason why this is not allowed (cc @Balandat to check me on that one and add more detail) is that we expect all constraints to be linear under-the-hood, and it's not possible to represent constraints that include fixed and choice parameters that way.

Re: 2), we will address this to make sure a proper error is raised about this earlier instead of the confusing one you ran into.

[petrcezner]

@lena-kashtelyan Thank you for your answer! I will check such constraints in the evaluation function. Thank you for your time!

[Balandat]

Re: 1), I believe that the reason why this is not allowed [...] is that we expect all constraints to be linear under-the-hood, and it's not possible to represent constraints that include fixed and choice parameters that way.

So there shouldn't be a fundamental issue including fixed parameters, as we can just absorb their value into a new bound. That said, if the problem is simple enough, you can just do this yourself. Instead of 't1 - lw >= 0' you'd just provide f'-lw <= {-t1}'. If you have a lot of constraints then this may be a bit tedious, but it should always be possible. On our end, we could consider properly parsing out fixed parameters from the constraints and do this under the hood, we'd have to look at how much work that would be relative to the benefit it provides.

As far as choice parameters are concerned, the reason we didn't allow constraints on these was that they were one-hot encoded and thus representing the constraint as a linear constraint is not straightforward. However, we made some changes recently that handle choice parameters in a different way and optimize over their actual values, so it would potentially be possible to incorporate them into parameter constraints (in fact, potentially even nonlinear ones). cc @dme65 , @vishwakftw

[petrcezner]

Hello @Balandat , I try to have following constrains: -lw>={-t1} and lw<={t1}. Whit these constrains I run into following errors in constraint_from_str function:

File "MY_PYTHON_PATH\lib\site-packages\ax\service\utils\instantiation.py", line 249, in constraint_from_str
    assert right in parameter_names, f"Parameter {right} not in {parameter_names}."
AssertionError: Parameter 288 not in dict_keys(['t1', 't2', 'lw', 'rw', 'dt2']).

or

  File "MY_PYTHON_PATH\lib\site-packages\ax\service\utils\instantiation.py", line 248, in constraint_from_str
    assert left in parameter_names, f"Parameter {left} not in {parameter_names}."
AssertionError: Parameter -lw not in dict_keys(['t1', 't2', 'lw', 'rw', 'dt2']).

It looks like even thought it is a parameter constraints, the order constraints is checked (because of if len(tokens) == 3: in function constraint_from_str).

[Balandat]

There seems to be some issue with the string parsing here, the 288 I assume is the value of t1 in this case? Apparently the -lw is also not properly parsed. Do you have a full repro of the setup you're using here so we can debug? (maybe @lena-kashtelyan can triage as I'm not very familiar with that part of the codebase).

Btw, -lw>={-t1} and lw<={t1} are the same constraint.

[lena-kashtelyan]

Hi, @petrcezner, the expected formatting for constraints is discussed here: https://ax.dev/tutorials/gpei_hartmann_service.html#2.-Set-up-experiment. Under the hood, we construct Ax ParameterContraint objects, which can be of two types: OrderConstraint (API ref) and SumConstraint (API ref). Therefore, the string parsing issue you are running into is actually just because there currently is not a way to represent a constraint like "parameter <= bound" (which I believe is what lw<={t1} is?).

However, what I'm wondering is why would someone want such a constraint when it can be directly encoded in the bounds of the parameter? If there is a fixed parameter t1 and a range parameter lw that must stay below the value of the fixed parameter (228), why not just make the upper bound on lw 228? I'm sure there is something obvious I'm missing here, but I think the ability to encode constraints like that into the search space is why we don't offer them.