queue: log dvc errors

[dberenbaum]

Sometimes queued experiments fail during the dvc setup phase before running the stages. In that case, no logs are captured. Can we also log this info?

See https://discord.com/channels/485586884165107732/1117491330340372480/1119195459269963818 for an example.

[dberenbaum]

@pmrowla What's the level of effort here? Is there any way we could at least provide an informative message about why there are no logs for the experiment?

[pmrowla]

The issue here would be that we can't tell whether there are no logs because it failed before running any pipeline stages, or whether there are no logs because they were explicitly removed

We can look into trying to make dvc-task redirect parts of the celery worker logging into the appropriate exp log file, but that's probably a non-trivial amount of work, and there will always be the limitation of "if celery dies before we start the redirect we still won't have logs"

[dberenbaum]

Could we start the log file with something like "Running on celery worker <celery-worker-id>. For celery worker logs, see <path/to/celery/worker/logs>"?

[pmrowla]

I'm not sure that the task can actually see what worker it is running on (since it's not supposed to matter), I'll have to take a look into the celery docs.

Also, the worker log is overwritten every time the given worker is restarted (which is every time the queue is started), so even if we list a specific worker path in the experiment logs, it's possible that the logging information will already be invalid by the time the user looks at the celery logs file.

The celery worker log location is fixed, would it be possible for vscode to just have a generic "open/view queue worker logs location" button for debugging purposes instead of trying to tie the celery logs to specific experiments?

[dberenbaum]

@mattseddon WDYT? Should the extension show the path to the celery worker logs if there's no logs found?

@pmrowla Can we do the same in the CLI? If given a valid queue id and no logs are found, can we point them to the celery worker logs instead of showing a generic error?

[pmrowla]

Yeah we can do that in the CLI

[mattseddon]

@mattseddon WDYT? Should the extension show the path to the celery worker logs if there's no logs found?

Based on previous dealings with the queue and things we've tried to implement I'd say that this is an implementation detail and it should not be exposed. We will be putting the added burden of understanding different workers having different logs, which log relates to which task and logs being overwritten onto the user.

However, if this is the best we have and it is actually better than nothing then let's go for it.