Skip to content

All crawlee persistence names should be valid names for Apify platform #1354

Description

@Pijukatel

In FileSystemRequestQueueClient there is persistance to kvs with name persist_state_kvs_name=f'__RQ_STATE_{self._metadata.id}',

When used on Apify platform, it will throw an error:

apify_client._errors.ApifyApiError: Invalid value provided in store: name can only contain letters 'a' through 'z', the digits '0' through '9', and the hyphen ('-') but only in the middle of the string (e.g. 'my-value-1')

which is caused by the underscores in the name.

Fix the name to be compatible with the Apify constraints and check all similar names of RecoverableStates in code base

To reproduce, run this on Apify platform, which create file system base request queue, that persists to Apify kvs:

async def main() -> None:
    async with Actor:
        rq = await RequestQueue.open(
            storage_client=FileSystemStorageClient(),
        )

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions