TuRBO support in Ax

[winf-hsos]

Hi, do you have a working example to integrate TuRBO with Ax yet? Are there plans in this direction?

I have recently read the paper and ran the Botorch example for the TuRBO-1 algorithm. We are currently working on optimizing a 60 dimensional (and larger) simulation model. We tried ALEBO but didn't get good results, probably due to too many parameters being important for the outcome.

Thank
Nicolas

[Balandat]

Hi and thanks for your interest.

We have an internal implementation of TuRBO in Ax, but it's quite complex and optimized for a particular high-dimensional high-throughput asynchronous optimization problem (hundreds of parameters, tens of thousands of evaluations, with high parallelism).

We are in the process of refactoring this, and we hope to be able to generalize this enough so we can make it available as part of Ax. I don't really have a timeline for this yet, but we should be able to give a status update on this in a few weeks from now.

[Mita2049]

Hi, are there any updates on this? Given that TuRBO has been implemented in Botorch, are there any specific reasons why porting it to Ax could be challenging? Thank you for all your work on both packages!

[lena-kashtelyan]

Hi, @Mita2049! Unfortunately we don't have an update on this yet. There is active work on TRBO in Ax open-source, but it's unfinished and not currently highest-priority. I think it's likely that our TRBO implementation will be in open-source by end of the year, but not 100% guaranteed.

[Balandat]

We're hoping to OSS it by the end of September, but as Lena said it could take a bit longer. We have a working version, but we want to take a step back and revisit the design to make sure it's solid rather than putting out a janky version :).

[eytan]

@Mita2049 re: why porting to Ax is challenging, a big reason is because TuRBO involves all kinds of additional state that is not part of regular BayesOpt loops, and Ax needs to be able to serialize/deserialize this state, and recover correctly from various failure modes. If we planned on using TuRBO as-is, this might not be all that challenging, but the team has actively
been doing research on improving and extending TuRBO, which has made the ideal architecture and data model a bit less clear. Furthermore, if we were to release the version we have in production now, our planned refactors would render TuRBO experiments from the previous version unusable, and we'd rather not introduce breaking changes between Ax releases. The good news is that we will have some battle tested, extensible implementations of trust region bayesOpt that should generally exceed the performance of the original TuRBO paper.

In the meantime, @Mita2049, @winf-hsos, you might want to try out SAASBO, which outperforms TuRBO on a number of high-dim BayesOpt tasks (particularly when the number of samples are limited, as one might have for AutoML tasks; see https://arxiv.org/pdf/2103.00349.pdf for details). Here is a tutorial on how to use it in Ax:
https://github.com/facebook/Ax/blob/main/tutorials/saasbo.ipynb

Thanks for your patience here, and I hope that SAASBO can provide a good solution in the more immediate term.

[Mita2049]

@lena-kashtelyan @Balandat @eytan Thank you very much for chiming in, and apologies for the late reply! Appreciate the context behind the integration of TuRBO into Ax - it seems you've made all the right decisions :)

I did try SAASBO with Ax 0.2.1 and Botorch 0.5.0, but ended up with the same problem as mentioned here: #682. I just noticed Botorch 0.5.1 came out, can I assume that the problem has been fixed? (My setup is somewhat complicated and so I'm not just a simple pip install botorch==0.5.1 away from testing this myself :) )

Thanks again for your continued work and support of these two libraries!

[Balandat]

Yes the problem has been fixed and things should work on BoTorch 0.5.1 with the latest Ax main branch. We are also planning to put out a new Ax release this week, which will resolve this issue without having to resort to doing the dev install.

[lena-kashtelyan]

Work on getting TRBO into open-source has been delayed for now, so sending this back to the wishlist for now.