[Question] BO's result with constraints of parameter's range and outcome

[rinc5]

Hello.
I've been trying out Ax for my task and I really like this great library.

I have three questions about my multi-objective optimization task.

Use case:
There are two metrics (Y1: maximize, Y2: outcome constraint). The metrics and parameters (X1- X5, X3+X4+X5==100) are loaded from CSV file (experimental data). This experiment takes a long time, so I want to know what value should I set to parameters for next experiment. In addition, I want to know the prediction value of Y1 and Y2 at these parameters.

My questions:

1. "range" of X1 and X2 at ax.modelbridge.transforms.int_to_float warning message is incorrect.

The WARNING is occured at get_next_trial and the warning message is like this:

[WARNING 03-10 17:14:43] ax.modelbridge.transforms.int_to_float: Unable to round {...}to meet constraints of SearchSpace(parameters=[RangeParameter(name='X1', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='X2', parameter_type=FLOAT, range=[0.0, 1.0]), ..., parameter_constraints=[ParameterConstraint(1.0*X3 + 1.0*X4 + 1.0*X5 <= 100), ParameterConstraint(-1.0*X3 + -1.0*X4 + -1.0*X5 <= -100.0)])

I set 'bound': [-50.0, 10.0] for X1 and 'bound': [-0.5, 50.0] for X2, but different ranges are printed at the warning message.
Why is this difference occured? (I confirmed ax_client.experiment.parameters and they were correct ranges.) What should I do to avoid this?

2. Should I remove data which violate the outcome constraint?

When I use data which violate the outcome constraint, the prediction value of Y2 at get_next_trial is out of range (Y2 > 10.0).
On the other hand, when I don't use these data, the prediction value of Y2 at the parameters is within range (0.0 <= Y2 <= 10.0).

3. Is it correct with my code for predicting metric values at get_next_trial?

My code is like this:

df = pd.read_csv('data.csv')

parameters = [
    {
        'name': 'X1',
        'value_type': 'float',
        'type': 'range',
        'bounds': [-50.0, 10.0],
    },
    {
        'name': 'X2',
        'value_type': 'float',
        'type': 'range',
        'bounds': [-0.5, 50.0],
    },
    {
        'name': 'X3',
        'value_type': 'int',
        'type': 'range',
        'bounds': [0, 100],
    },
    {
        'name': 'X4',
        'value_type': 'int',
        'type': 'range',
        'bounds': [0, 100],
    },
    {
        'name': 'X5',
        'value_type': 'int',
        'type': 'range',
        'bounds': [0, 100],
    },
]

# initialize AxClient
ax_client = AxClient(...)

# set up experiment
ax_client.create_experiment(
    name='my_BO_task',
    parameters=parameters,
    objective_name='Y1',
    minimize=False,
    parameter_constraints=['X3 +X4 + X5 <= 100', 'X3 + X4 + X5 >= 100'],
    outcome_constraints=['Y2 >= 0.0', 'Y2 <= 10.0'],
)

# optimize
for _, row in df.iterrows():
    # convert row to dict (parameter_data, metric_data)

    _, trial_index = ax_client.attach_trial(parameter_data)
    ax_client.complete_trial(trial_index, metric_data)

next_candidates, _ = ax_client.get_next_trial() # WARNING is occured here

# predict metric values at next_candidates
m = get_GPEI(ax_client.experiment, ax_client.experiment.fetch_data())
f, cov = m.predict([ObservationFeatures(parameters=parameters)])

Any help is appreciated. And if you would like to know more information, please ask me.
Thank you so much for your help.

[Balandat]

1. Not quite sure, I'll let other folks speak to this.

2. Whether the data violates the constraint or not should be irrelevant to whether generated candidates will. In fact, having data already outside the feasible set should help improve the model at the boundary. Not quite sure what's going on there. Is having this data the reason why you impose the parameter equality constraint and don't just eliminate one of the variables? E.g. use X5 = 100 - X3 - X4 in your downstream code, remove X5 from the parameters, and add the inequality constraint X3 + X4 <= 100. If you don't have data outside the feasible set Ax should have a much easier time after removing this unnecessary variable, but if you do then it could help.

3. You'll want to use ObservationFeatures(parameters=next_candidates)

[2timesjay]

1. Some internal transforms have already been applied. Sampling is done on float ranges, which are then rounded back to ints (which is failing because the constraints are too difficult). This warning should be clearer.

the solution given to (2) by max should also resolve the underlying problem. Having two inequality constraints to enforce a constraint on integers is the main issue.

[rinc5]

Thank you for your quick and gracious response.

I didn't have that idea (X3 + X4 <= 100). It sounds good. So, I just tried out it. But, unfortunately, I got same warning and same incorrect range. Also, the prediction value of Y2 at next_candidate is still out of range when I use data which violate the outcome constraint.
By the way, I'm worried that the effect of X5 isn't considered to GP model if I remove X5.

The reason why I use the parameter constraint X3 + X4 + X5 == 100 is that they are percentage values. In this use case, I know Y1 and Y2 are related to positive correlation, but I want to increase Y1 within the range 0.0 <= Y2 <= 10.0.

[2timesjay]

@rinc5
Is this still an issue?

If you can post your precise experiment setup since updating the constraint I will double check the rounding logic.

Since X5 was already completely constrained the GP's model quality should stay the same or improve. The only thing missing would be convenient methods to plot its influence on results.

Outcome constraints cannot be completely enforced. Is it possible that Y2 is never within the (0, 10) range? How much do your results violate this constraint by?

One more option is to model x3 and x4 as floats and apply your own rounding after their suggestions are passed to you. I know that's not ideal, but if there is a bug in our code it's a good workaround.

[rinc5]

Thank you for your concern, @2timesjay .

One more option is to model x3 and x4 as floats and apply your own rounding after their suggestions are passed to you. I know that's not ideal, but if there is a bug in our code it's a good workaround.

Your recommendation is nice! I tried it and this issue was solved.
I can use X3 and X4 as floats. The reason why I used them as ints is I thought that floats took more time to optimize than ints. But it didn't take a long time even though I changed ints to floats.

Thank you all for your kind.

[sgbaird]

Also, the prediction value of Y2 at next_candidate is still out of range when I use data which violate the outcome constraint.

Was this ever resolved? I'm confused at how the constraint Y2 was ever factoring into get_next_trial() which may stem from some of the hidden code. For example, nowhere is it specified how Y2 would be calculated, so I'm assuming that Y2 gets predicted by the GP model along with Y1.