[Feature]: Support multi-instance AMS REST service

[link3280]

Description

Currently, only the leader AMS would provide REST service. When an AMS leader transition happens, the standby AMS would take 30+ seconds to initialize optimizing, after which the REST service is available. That affects the availability of the REST service.

I proposed to make the REST service independent of the leadership. That's to say, the stand-by AMS instances need to initialize and run REST service on start. Their only difference to a leader instance is no optimizing service.

Use case/motivation

No response

Describe the solution

I drew a diagram of the main modules in AMS.

• The green ones are the modules that should be run by both the leader and the followers when starting.
• The yellow ones are the leader-only services that should be run when an AMS becomes the leader.
• The remaining red one is a new module that syncs catalog changes from the leader through reloading from the database.

Subtasks

• [Improvement]: Make Controller Depend on Stateless Services #3346

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[baiyangtx]

Are you still working on this? I'm interesting in this too. @link3280

[link3280]

Are you still working on this? I'm interesting in this too. @link3280

I'm reconsidering the proposal since @majin1102 plans to support multi-instance AMS. We could align the efforts to avoid duplicate work. What do you think? @baiyangtx

[baiyangtx]

I started a discussion and provided a design. #2997

You are welcome to participate in the discussion. @link3280 @majin1102

[czy006]

+1 We also need this part, design can be completed together will reduce duplication of work

[baiyangtx]

The implementation of this issue can be divided into several steps:

1. Refactor and implement the statelessness of the front-end interfaces. (I have created an issue [Improvement]: Make Controller Depend on Stateless Services #3346 and broken it down into multiple subtasks that can be advanced in parallel.)
2. Implement the sharing of HTTP Session across AMS instances. (There is already a PR [Improvement]: Support database based http session management. #3340 in progress for this project.)
3. Refactor the AMS startup process to launch the Rest HTTP service before the leader election. (This step depends on the completion of the first two steps.)