It is really thought-provoking. We recently find the elasticity of LLMs and I hope it will help us understand the alignment process to avoid superficial alignment. Paper will come soon !
Replying to @BlancheMinerva
It's known that finetuning can incidentally remove RLHF guards arxiv.org/abs/2310.03693. Can you solve this by including examples with refusals mixed into the data? Does it matter if those refusals are in-distribution for the original RLHF? Does the domain of the FT task matter?
