Co-Authored-By: Claude

Disclaimer: This service was not reviewed or approved by, nor does it necessarily express or reflect the policies or opinions of, arXiv.

This repository is a configurable template for the Atom feed of arXiv articles that have code available. The goal is to offer replacements for the "papers with code" web feed offered until the mid-2025. The code detection algorithm scans the article text, excluding the References section, for common code-hosting domains, such as github.com, gitlab.com, and huggingface.co. This maximizes recall at the cost of generating false positives.

To configure your own feed serving, fork the https://github.com/code-available-feed/code-available-feed repository and set GitHub Actions variables. The feed for the desired arXiv category will be served on GitHub Pages.

See https://code-available-feed.github.io/code-available-feed for the sorted list of available feeds. If you serve a feed using this project, consider making a pull request to add your feed to this list.

1. Fork the https://github.com/code-available-feed/code-available-feed repository.

2. In your fork configure Settings → Secrets and Variables -> Actions → Variables -> Repository Variables:

   Variable	Default	Description
   ARXIV_CATEGORY_ID	cs.AI	Set to any arXiv category, e.g. cs.CV.
   ARXIV_CATEGORY_STRICT	false	Set to true to restrict to only articles whose primary category matches ARXIV_CATEGORY_ID.

   Normally you only need to set the above variables. Note that if the articles are not restricted to the primary category, and you are subscribed to several such feeds, for example cs.AI and cs.SD, your feed reader may perform feed item deduplication.

   The variables below may need to be modified for the categories that have a low frequency of published articles.

   Variable	Default	Description
   ARXIV_CONTINUE_ON_API_ERROR	true (in the workflow)	Set to false to fail the workflow immediately on any arXiv API error instead of continuing silently. The goal of the silent continuation on arXiv API errors is to avoid frequent pipeline failure email notifications, and instead alert on feed staleness (see ARXIV_MAX_STALENESS_DAYS).
   ARXIV_MAX_STALENESS_DAYS	10 (in the workflow)	The workflow fails with a staleness alert when the newest feed entry is more than this many calendar days old (e.g. with the default of 10, an age of 10 days passes and 11 days fails). Set to -1 to disable the check. Raise this value for low-volume categories where multi-day gaps without new articles are normal.
   ARXIV_MAX_BACKFILL_DAYS	8	The rolling fetch window in days. The API query fetches articles from [today - ARXIV_MAX_BACKFILL_DAYS, today]. Must be a positive integer (minimum 1). With the default of 8, Monday's run covers the full prior ISO week plus one extra day, recovering articles submitted after Thursday 14:00 ET that the arXiv announcement schedule delays past the ISO week boundary.
   ARXIV_MAX_RESULTS	50	Number of articles requested per arxiv API page.
   RETRY_BACKOFF_BASE_SECONDS	60	Base duration in seconds for exponential retry backoff on the first API page. The N-th retry waits N × RETRY_BACKOFF_BASE_SECONDS seconds.

3. Enable GitHub Actions in Settings -> Actions -> General -> Allow {owner}, and select non-{owner}, actions and reusable workflows -> Allow actions created by GitHub.

4. Run the pipeline_feed workflow at least once (via the Actions tab or wait for the daily schedule). The first successful deploy_orphan run creates the gh-pages branch.

5. In repository Settings → Pages, set Source to Deploy from a branch, Branch to gh-pages, Folder to /docs.

After setup, the feed URL is https://{owner}.github.io/{repo}/arxiv/{category}/atom.xml (e.g. https://marcindulak.github.io/code-available-feed-cs-ai/arxiv/cs.ai/atom.xml).

Build the Docker image:

bash scripts/build_docker_image.sh

Run the pipeline:

ARXIV_CATEGORY_ID=cs.AI \
ARXIV_CATEGORY_STRICT=false \
ARXIV_CONTINUE_ON_API_ERROR=true \
ARXIV_MAX_BACKFILL_DAYS=2 \
GITHUB_REPOSITORY=marcindulak/code-available-feed-cs-ai bash scripts/pipeline_feed.sh

Validate the generated feed:

bash scripts/validate_atom_xml.sh --filename docs/arxiv/cs.ai/atom.xml

bash scripts/test_e2e_behave.sh

bash scripts/test_mypy.sh

The gh-pages branch holds only the most recent orphan commit (force-pushed on each deploy). The 7-day arxiv-feed artifact is the rollback window: re-run deploy_orphan via workflow_dispatch with an older successful pipeline_feed run. Beyond 7 days, recovery regenerates the feed from the current arXiv state.

├── compose.yml                   # Docker service: build from Dockerfile.server, mount repo at /app
├── Dockerfile.server             # debian with apt-get packages for python (no pip)
├── docs/
│   └── arxiv/
│       └── {category}/
│           ├── atom.xml          # Rolling-window Atom 1.0 feed
│           └── archive/          # Prior-weeks rolling-window archives
│               └── YYYY-WNN/
│                   └── atom.xml
├── features/
│   ├── environment.py            # Behave hooks: skip @status-todo scenarios, restore envs
│   ├── fixtures/                 # Fixture data for determinism and field-extraction tests
│   └── steps/                    # Step definitions (one file per feature group)
├── scripts/
│   ├── build_docker_image.sh     # Build the server Docker image
│   ├── check_atom_xml.sh         # Parse-only XML well-formedness check on every docs/**/atom.xml
│   ├── check_feed_staleness.sh   # Fail if the feed recently has no new items
│   ├── deploy_orphan.sh          # Force-push docs/ to the gh-pages orphan branch
│   ├── pipeline_feed.sh          # Generate and validate feed inside Docker
│   ├── validate_atom_xml.sh      # Validate atom.xml with newsboat inside Docker
│   ├── test_e2e_behave.sh        # Run BDD test suite inside Docker
│   └── test_mypy.sh              # Run mypy type checking inside Docker
└── src/
    ├── __init__.py               # Package marker
    └── pipeline_feed.py          # All feed creation and management logic

See REQUIREMENTS.md.