expose HttpRequest.setInterceptor to allow for throttled uploads

[srosenberg]

We have a multi-threaded uploader which transfers large files from local disk to GCS. The uploader is co-located with our low-latency app. inside the JVM. To control latency, we must throttle the uploader. Our best workaround involves resorting to reflection in order to install HttpRequest interceptor which effectively wraps HttpContent with a throttled OutputStream; e.g.,

Object storageRpc = FieldUtils.getField(storage.getClass(), "storageRpc", true).get(storage);
Object serviceStorage = FieldUtils.getField(storageRpc.getClass(), "storage", true).get(storageRpc);
Object requestFactory = FieldUtils.getField(serviceStorage.getClass(), "requestFactory", true).get(serviceStorage);
Field initializerField = FieldUtils.getField(requestFactory.getClass(), "initializer", true);
Object initializer = initializerField.get(requestFactory);
initializerField.set(requestFactory, new InterceptingHttpRequestInitializer((HttpRequestInitializer) initializer));

[yihanzhen]

This would require work in google-http-java-client. cc/ @chingor13

[chingor13]

Is there a reason this needs to be modifiable after construction?

It looks like you can accomplish wrapping an additional HttpRequestInitializer by extending the HttpTransportOptions class and wrapping your InterceptingHttpRequestInitializer around the parent one:

class MyTransportOptions extends HttpTransportOptions {
  // TODO: make your own constructor or builder
  @Override
  public HttpRequestInitializer getHttpRequestInitializer(final ServiceOptions<?, ?> serviceOptions) {
    return new InterceptingHttpRequestInitializer(super.getHttpRequestInitiallizer(serviceOptions));
  }
}

StorageOptions options = StorageOptions
  .newBuilder()
  .setTransportOptions(new MyTransportOptions())
  .build();
Storage storage = options.getService();

[srosenberg]

@chingor13 That works for our use-case, thanks. I must have missed that HttpTransportOptions is used by default (and getHttpRequestInitializer can be overridden); much better than using reflection hack.

I can't think of another use-case which would require modifying RequestInitializer after construction; in fact, it seems like a good practice to make it immutable; e.g., modifying it while request is in progress is a risky proposition. Feel free to close the issue.

[ajaaym]

closing this issue as its resolved.

[srosenberg]

@chingor13 We're only now revisiting this issue... what you suggested is not feasible (statically) because HttpTransportOptions has a private constructor,

https://github.com/googleapis/google-cloud-java/blob/master/google-cloud-clients/google-cloud-core-http/src/main/java/com/google/cloud/http/HttpTransportOptions.java#L123

@ajaaym could you please reopen the issue unless there is a less hacky workaround as I'd originally suggested?

[andrey-qlogic]

@srosenberg , @sduskis what is new feature might be needed from the api: storage now by this feature request?