fix: prevent Actor log-streaming thread from crashing on stream timeout - #944
Merged
Merged
Conversation
vdusek
added
adhoc
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #944 +/- ##
==========================================
- Coverage 94.56% 94.54% -0.03%
==========================================
Files 48 48
Lines 5119 5129 +10
==========================================
+ Hits 4841 4849 +8
- Misses 278 280 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vdusek
marked this pull request as ready for review
July 13, 2026 07:56
Pijukatel
approved these changes
Jul 13, 2026
vdusek
added a commit
that referenced
this pull request
Sep 16, 2026
…or call (#1027) ### Description `actor.call()` with the default `logger='default'` could raise on a run that had already succeeded. A status poll that failed after retries left the exception on the watcher task; `stop()` then cancelled an already-finished task and `await`ed it, re-raising past `except asyncio.CancelledError`. The sync twin leaked the same failure out of its thread. - Both watcher loops now use the handler `StreamedLog._stream_log` already had: warn on `is_timeout_error`, `logger.exception` otherwise. A failed poll gets reported and the run's own result stands. - The sync watcher's thread is a daemon, so a watcher whose loop never terminates cannot hold up interpreter shutdown. - `StreamedLog.stop()` closes the stream response and caps its join at `_stop_timeout_s` (5s), where it used to join unbounded on a stream that may not speak for hours. - `start()` on both sync classes checks `is_alive()`, so a handle left by a thread that outlived its stop does not block a restart. The close alone is not enough. On Impit 0.13.2 `Response.close()` returns immediately and sets `is_closed=True`, yet a reader blocked in `iter_bytes()` stays blocked, and closing the `Client` does not release it either. The close is kept because it is correct for a custom HTTP client, and the capped join is what makes `stop()` return. ### Why `stop()` is bounded this way - #825 bounded `stop()` by requesting the log stream with a 30s read timeout. - #944 removed that: Impit applies `timeout` to the whole request, streamed body included, so a run logging past the bound was cut off mid-stream with `impit.TimeoutException` (#945). The stream has used `no_timeout` since, with the unbounded `stop()` recorded as a known limitation. - This PR caps the join and closes the response, leaving the stream request alone. `_stream_timeout` stays `no_timeout`, still asserted by `test_streamed_log_sync_requests_stream_with_no_timeout`, so #945 cannot regress. What it fixes is the case #944 left open: a manual `stop()` on a stream gone quiet. Trade-off: a thread outliving the 5s cap keeps running as a daemon, so `stop()` can return before the tail is flushed. On the `actor.call()` path the run's EOF ends the thread first, measured at 0.000s over repeated `apify/hello-world` runs. ### Test plan - 8 unit tests and an integration test that fails a real run's background status polls, each red before the fix. *✍️ Drafted by Claude Code*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After a successful
ActorClient.call(), the background log-streaming thread could crash with an uncaughtimpit.TimeoutException, printing a traceback even though the run finished fine.Issue
Closes: #945
Root cause
The log stream was requested with a bounded 30s timeout. impit applies
timeoutto the whole request, including the streamed body, so any run streaming logs longer than that trippedimpit.TimeoutExceptionmid-stream — unhandled in the sync thread, and a spuriousERROR+ traceback in the async twin. Raising the value wouldn't help: every tier is capped atDEFAULT_TIMEOUT_MAX(360s).Fix
no_timeout(impit's ~24h ceiling), so it runs until the server closes it with EOF at run finish. Matches the JS client.impit.TimeoutExceptionin both paths and log aWARNINGinstead of crashing / error-logging; other failures still error-log.Known limitation
A parked sync
iter_bytes()read can't be interrupted from another thread, so withno_timeouta manualStreamedLog.stop()on a momentarily silent stream waits for the next chunk or EOF rather than the old 30s bound. TheActorClient.call()path is unaffected (run finish sends EOF). Matches the JS client.