fix: Make batch_add_requests split batches by serialized payload size - #953
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #953 +/- ##
==========================================
+ Coverage 94.61% 94.64% +0.02%
==========================================
Files 58 58
Lines 5239 5263 +24
==========================================
+ Hits 4957 4981 +24
Misses 282 282
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Pijukatel
left a comment
There was a problem hiding this comment.
Looks good to me.
Claude had one more minor point:
Peak memory: requests_as_dicts and serialized_requests are now both alive for the whole call, so peak memory is roughly dicts + full serialized JSON at once. Previously the serialized form existed only per-batch at send time. For crawler-scale inputs (hundreds of thousands of requests) this is a real bump; a cheap mitigation is del requests_as_dicts after serialization. Acceptable tradeoff for exact measurement, but worth a thought.
### Description - Mirroring apify/apify-client-python#953. - `batchAddRequests` stringified every request twice: once in `sliceArrayByByteLength` to measure the batch, and again in the `serializeRequest` interceptor when sending. Axios then ran its default `transformRequest` on top, because `transformRequest: undefined` in the instance config falls back to the defaults, and that default validates a JSON string body by parsing it in full. Each batch body was stringified twice and parsed once more. - Each request is now serialized once up front. The byte lengths decide the batch boundaries (commas and brackets counted, which the old measurement skipped) and the same strings are joined into the batch body, sent with an explicit `content-type: application/json`. The interceptor passes a string body with an explicit content type through untouched, and the axios instance sets `transformRequest` and `transformResponse` to `[]`. That also removes the validation re-parse for every JSON body the client sends. ### Issue - Closes #972 ### Breaking changes - A string body declared as JSON but not valid JSON (for example `setRecord` with `contentType: 'application/json'` and a non-JSON string) is sent as it is. Axios used to double-encode it into a JSON string literal. - A request too large for the payload limit is rejected before any batch is sent. It used to be detected only when its batch came up, with earlier batches already in flight. - The protected `_batchAddRequests` and `_batchAddRequestsWithRetries` take the serialized entries, and `_batchAddRequests` no longer re-validates a batch the public method already validated. The API report is updated. *✍️ Drafted by Claude Code*
The 9 MB payload guard in
batch_add_requestswas inert:constrained_batcheswas called withoutget_len, so the defaultlen()measured each request dict's key count (~4) instead of its serialized size. Batches were therefore split only by the 25-request count limit, and large requests shipped as one oversized POST that the API rejects with 413, failing the whole call.The guard now measures each request as its UTF-8 JSON byte length, using the same serialization flags as the HTTP client's request body path. It also passes
strict=False, which preserves the previous contract for an individually oversized request: it's sent in its own batch and left for the API to judge, instead of raising a client-sideValueError. Both the size-based splitting and the oversized-singleton path are covered by new sync/async regression tests.✍️ Drafted by Claude Code