This repository was archived by the owner on Sep 30, 2024. It is now read-only.
backport: grpc: add automatic retry support to all services#59404
Merged
backport: grpc: add automatic retry support to all services#59404
Conversation
This PR adds a basic configuration for enabling retries with gRPC for certain RPC types.
The description for `defaults.RetryPolicy` is probably the most important thing to read:
```go
// RetryPolicy is the default retry policy for internal GRPC requests.
//
// The retry policy will trigger on Unavailable and ResourceExhausted status errors, and will retry up to 20 times using an
// exponential backoff policy with a maximum duration of 3s in between retries.
//
// Only Unary (1:1) and ServerStreaming (1:N) requests are retried. All other types of requests will immediately
// return an Unimplemented status error. It's up to the caller to manually retry these requests.
//
// These defaults can be overridden with the following environment variables:
// - SRC_GRPC_RETRY_DELAY_BASE: Base retry delay duration for internal GRPC requests
// - SRC_GRPC_RETRY_MAX_ATTEMPTS: Max retry attempts for internal GRPC requests
// - SRC_GRPC_RETRY_MAX_DURATION: Max retry duration for internal GRPC requests
var RetryPolicy = []grpc.CallOption{
retry.WithCodes(codes.Unavailable, codes.ResourceExhausted),
// Together with the default options, the maximum delay will behave like this:
// Retry# Delay
// 1 0.05s
// 2 0.1s
// 3 0.2s
// 4 0.4s
// 5 0.8s
// 6 1.6s
// 7 3.0s
// 8 3.0s
// ...
// 20 3.0s
retry.WithMax(uint(internalRetryMaxAttempts)),
retry.WithBackoff(fullJitter(internalRetryDelayBase, internalRetryMaxDuration)),
}
```
This is off by default for all services (since this logic doesn't work with RPCS or might not be desirable as the default behavior if you don't know whether or not your method is idempotent).
The upstack PRs selectively enable this logic for appropriate RPCs (see those PRs for the exact semantics).
## Test plan
CI
…e/retry package (#59140) The package has some issues (the retry logic for client stream is flawed). I'm adding a copy of this to our repository for future edits. See the discussion in https://github.com/sourcegraph/sourcegraph/pull/59145 ## Test plan The existing test suite from the copied project is now running in CI.
…e already recieved a message on the stream (#59145) When retrying a client stream, we must ensure that we haven't received any data from the server yet before retrying. Otherwise, we can't know if the client has already consumed part of the stream. Blindly retrying the stream could produce duplicate messages or inconsistent messages. The only safe generic behavior that we can implement is to only retry if an error occurs _before_ the server successfully sends the first message. After that, any encounters that we see on the stream will be directly returned to the caller - no retries will occur. Only the caller knows the retry semantics that it wants. This matches the built-in grpc retry behavior (that we can't use, see https://github.com/sourcegraph/sourcegraph/issues/51060) as documented on https://learn.microsoft.com/en-us/aspnet/core/grpc/retries?view=aspnetcore-8.0#when-retries-are-valid: > Streaming calls > > Streaming calls can be used with gRPC retries, but there are important considerations when they are used together: > > Server streaming, bidirectional streaming: **Streaming RPCs that return multiple messages from the server won't retry after the first message has been received. Apps must add additional logic to manually re-establish server and bidirectional streaming calls.** As a side note: The upstream library had this behavior back in 2021 (and the discussion is a bit baffling to me): grpc-ecosystem/go-grpc-middleware#313 ## Test plan This PR adds two additonal tests to the test suite that ensure that: 1. The library is capable of retrying the RPC if we haven't received the first message in the stream yet 2. The library will **not automatically retry** if the first message from the server has already been recieved
) ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles Why does it matter? These test plans are there to demonstrate that are following industry standards which are important or critical for our customers. They might be read by customers or an auditor. There are meant be simple and easy to read. Simply explain what you did to ensure your changes are correct! Here are a non exhaustive list of test plan examples to help you: - Making changes on a given feature or component: - "Covered by existing tests" or "CI" for the shortest possible plan if there is zero ambiguity - "Added new tests" - "Manually tested" (if non trivial, share some output, logs, or screenshot) - Updating docs: - "previewed locally" - share a screenshot if you want to be thorough - Updating deps, that would typically fail immediately in CI if incorrect - "CI" - "locally tested" -->
This tweaks our forked [grpc retry](https://pkg.go.dev/github.com/grpc-ecosystem/go-grpc-middleware/retry) package to support traces in a similar manner to our internal httpcli logic. When reviewing this PR, I'd recommend comparing this against the logic in `internal/httpcli` to see if it's to your liking: https://github.com/sourcegraph/sourcegraph/blob/023e96c2fc25ced65c528be2474b5fd1f9a34792/internal/httpcli/client.go#L582-L631 ## Test plan 1. (pre-requisite) I checked out https://github.com/sourcegraph/sourcegraph/pull/59136 (`12-20-grpc_frontend_configuration_support_automatic_retries_GetConfig_is_idempotent_`) that is the PR that has retries hooked up for all services. 2. In `sg.config.yaml`, I commented out the entry that starts one of the gitserver instances when running `sg start`. ```patch diff --git a/sg.config.yaml b/sg.config.yaml index 312e5eb..eb0eef61193 100644 --- a/sg.config.yaml +++ b/sg.config.yaml @@ -1106,7 +1106,7 @@ commandsets: - repo-updater - web - gitserver-0 - - gitserver-1 +# - gitserver-1 - searcher - caddy - symbols ``` 3. I then ran `sg start` and `sg start monitoring` to start jaeger. 4. I executed the following search query with tracing enabled: https://sourcegraph.test:3443/search?q=context:global+type:diff+test+timeout:2m+count:all&patternType=standard&sm=1&trace=1&groupBy=repo This produces a trace with entries that look like the following <img width="1713" alt="Screenshot 2023-12-21 at 4 32 42 PM" src="https://github.com/sourcegraph/sourcegraph/assets/9022011/ec7e2c48-602c-4537-b27a-e9490105b384"> You can see the full trace here: [gh_trace.json](https://github.com/sourcegraph/sourcegraph/files/13747118/gh_trace.json)
This PR adds support for automatic retries in the gitserver grpc client. I have gone through the gitserver protobuf file and marked all the methods I thought were idempotent (we can't inspect this using the go protobuf packages, but I thought this was nice for documentation). I then wrapped the basic gitserver grpc client with an "automaticRetryClient" that uses the default retry policy that was defined in https://github.com/sourcegraph/sourcegraph/pull/59095. See that PR for more details. Note that for ServerStreaming methods like Exec and Search, the retry logic will only automatically retry if we haven't received any messages back from the server yet. After we receive a single message, we can't know whether or not the callers has consumed the message yet (e.x: started consuming the `io.Reader` from ArchiveReader) and can tolerate receiving old messages, duplicated messages, etc. If we get an error after this point, we'll fail the RPC immediately and bubble up the underlying error to the caller. Only the caller would know the semantics of how it's consuming the stream to know how to proceed. CI
This PR adds support for automatic retries in the symbols grpc client. I have gone through the symbols protobuf file and marked all the methods I thought were idempotent (we can't inspect this using the go protobuf packages, but I thought this was nice for documentation). I then wrapped the basic symbols grpc client with an "automaticRetryClient" that uses the default retry policy that was defined in https://github.com/sourcegraph/sourcegraph/pull/59095. See that PR for more details. Note that for ServerStreaming methods like LocalCodeIntel and SymbolInfo, the retry logic will only automatically retry if we haven't received any messages back from the server yet. After we receive a single message, we can't know whether or not the callers has consumed the message yet (e.x: started aggregating the symbols from `LocalCodeIntel` ) and can tolerate receiving old messages, duplicated messages, etc. If we get an error after this point, we'll fail the RPC immediately and bubble up the underlying error to the caller. Only the caller would know the semantics of how it's consuming the stream to know how to proceed. CI <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles Why does it matter? These test plans are there to demonstrate that are following industry standards which are important or critical for our customers. They might be read by customers or an auditor. There are meant be simple and easy to read. Simply explain what you did to ensure your changes are correct! Here are a non exhaustive list of test plan examples to help you: - Making changes on a given feature or component: - "Covered by existing tests" or "CI" for the shortest possible plan if there is zero ambiguity - "Added new tests" - "Manually tested" (if non trivial, share some output, logs, or screenshot) - Updating docs: - "previewed locally" - share a screenshot if you want to be thorough - Updating deps, that would typically fail immediately in CI if incorrect - "CI" - "locally tested" -->
This PR adds support for automatic retries in the searcher grpc client. --- I have gone through the searcher protobuf file and marked all the methods I thought were idempotent (we can't inspect this using the go protobuf packages, but I thought this was nice for documentation). I then wrapped the basic searcher grpc client with an "automaticRetryClient" that uses the default retry policy that was defined in https://github.com/sourcegraph/sourcegraph/pull/59095. See that PR for more details. Note that for ServerStreaming methods like Search, the retry logic will only automatically retry if we haven't received any messages back from the server yet. After we receive a single message, we can't know whether or not the callers has consumed the message yet (e.x: started presenting the data in the WebUI from `Search`) and can tolerate receiving old messages, duplicated messages, etc. If we get an error after this point, we'll fail the RPC immediately and bubble up the underlying error to the caller. Only the caller would know the semantics of how it's consuming the stream to know how to proceed. CI
… (all are idempotent) (#59130) This PR adds support for automatic retries in the repo-updater grpc client. --- I have gone through the repo-updater protobuf file and marked all the methods I thought were idempotent (we can't inspect this using the go protobuf packages, but I thought this was nice for documentation). I then wrapped the basic repo-updater grpc client with an "automaticRetryClient" that uses the default retry policy that was defined in https://github.com/sourcegraph/sourcegraph/pull/59095. See that PR for more details. CI
This PR adds support for automatic retries in the `zoekt-webserver` grpc client that `searcher` uses. --- I wrapped the basic zoekt-webserver grpc client with an "automaticRetryClient" that uses the default retry policy that was defined in https://github.com/sourcegraph/sourcegraph/pull/59095. See that PR for more details. All the methods don't have any side effects, so they're all capable of being retried. Note that for ServerStreaming methods like StreamSearch and List, the retry logic will only automatically retry if we haven't received any messages back from the server yet. After we receive a single message, we can't know whether or not the caller has consumed the message yet (e.x: started consuming the search results from `StreamSearch` and displaying them in the WebUI) and can tolerate receiving old messages, duplicated messages, etc. If we get an error after this point, we'll fail the RPC immediately and bubble up the underlying error to the caller. Only the caller would know the semantics of how it's consuming the stream to know how to proceed. ## Test plan CI
…s idempotent) (#59136) This PR adds support for automatic retries in the frontend configuration grpc client. I have gone through the frontend protobuf file and marked all the methods I thought were idempotent (we can't inspect this using the go protobuf packages, but I thought this was nice for documentation). I then wrapped the basic frontend grpc client with an "automaticRetryClient" that uses the default retry policy that was defined in https://github.com/sourcegraph/sourcegraph/pull/59095. See that PR for more details. All the methods are idempotent, so they all get the new retry logic. ## Test plan CI
ggilmore
commented
Jan 9, 2024
| rpc Search(SearchRequest) returns (SearchResponse) { | ||
| option idempotency_level = NO_SIDE_EFFECTS; | ||
| } | ||
| rpc LocalCodeIntel(LocalCodeIntelRequest) returns (stream LocalCodeIntelResponse) { |
Contributor
Author
There was a problem hiding this comment.
cc @sourcegraph/team-graph
Note that main has gotten rid of the LocalCodeIntel method, it only exist on the 5.2 branch. I thought this method seemed like it had no side effects, so it should be safe to retry. Please feel free to correct me on this though.
Contributor
There was a problem hiding this comment.
Yes, it should be OK to retry.
| return a.base.Search(ctx, in, opts...) | ||
| } | ||
|
|
||
| func (a *automaticRetryClient) LocalCodeIntel(ctx context.Context, in *proto.LocalCodeIntelRequest, opts ...grpc.CallOption) (proto.SymbolsService_LocalCodeIntelClient, error) { |
Contributor
Author
There was a problem hiding this comment.
cc @sourcegraph/team-graph
Note that main has gotten rid of the LocalCodeIntel method, it only exist on the 5.2 branch. I thought this method seemed like it had no side effects, so it should be safe to retry. Please feel free to correct me on this though.
Contributor
Author
|
@sourcegraph/release-guild Do I need to pay attention to the lighthouse errors? https://buildkite.com/sourcegraph/aspect-experimental/builds/4460 |
Contributor
|
Codenotify: Notifying subscribers in CODENOTIFY files for diff b737b47...7e9bab0.
|
Contributor
Contributor
Author
|
Hello @sourcegraph/release-guild - Could y'all stamp this PR and merge into the patch release? Thanks! |
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a manual backport of all the following PRs into a single commit for the 5.2 branch:
Collectively, these PRs all implement adding automatic retry support for idempotent gRPC methods to the 5.2 branch. Creating stacked PRs for these for the 5.2 backport would have taken ages with the need for manual approvals for each one, so I collected them all in one PR for convenience.
Some services had methods that only exist on the 5.2 branch - we've deleted / changed them on
main. I have called these unique methods out as PR comments.Test plan
CI for all referenced PRs is sufficient.
I ran the following search query on a local instance that had one instance of zoekt-webserver disabled (sg start --except zoekt-web-1): https://sourcegraph.test:3443/search?q=context%3Aglobal+test&patternType=standard&sm=1&trace=1&groupBy=repo
This produced a trace showing that the client did attempt to retry: