Skip to content

Prevent "too many open files" in artifact upload#1771

Merged
robherley merged 2 commits intoactions:mainfrom
rmunn:fix-too-many-open-files
Jul 31, 2024
Merged

Prevent "too many open files" in artifact upload#1771
robherley merged 2 commits intoactions:mainfrom
rmunn:fix-too-many-open-files

Conversation

@rmunn
Copy link
Copy Markdown
Contributor

@rmunn rmunn commented Jul 23, 2024

Fixes actions/upload-artifact#485.

The .file method uses "a lazystream wrapper to prevent issues with open file limits", which is precisely the scenario we're running into with actions/upload-artifact#485.

This was originally part of #1723, but I pulled the bugfix out into its own PR so it can be reviewed separately from the bugfix that #1723 was for (preserving Unix file permissions in uploaded .zip files). This one is more urgent than the Unix file permissions, which can be worked around by uploading a .tar file.

@rmunn rmunn requested a review from a team as a code owner July 23, 2024 10:59
This was referenced Jul 23, 2024
Open
Closed
@robherley robherley mentioned this pull request Jul 24, 2024
Merged
@rmunn
Copy link
Copy Markdown
Contributor Author

rmunn commented Jul 24, 2024 •
edited
Loading

Once #1774 is merged, I'll rebase on top of the new state of main so that the tests can pass.

@rmunn rmunn force-pushed the fix-too-many-open-files branch from 65faef4 to 9517cdf Compare July 26, 2024 01:49
@rmunn
Copy link
Copy Markdown
Contributor Author

rmunn commented Jul 26, 2024

Just rebased on top of main; let's see what the unit tests do now.

@rmunn
Copy link
Copy Markdown
Contributor Author

rmunn commented Jul 26, 2024

@robherley - Looks like a team member needs to re-approve running the workflows again.

@joshmgross
Copy link
Copy Markdown
Contributor

joshmgross commented Jul 26, 2024

Looks like this can be removed:

import {createReadStream} from 'fs'

@rmunn
Copy link
Copy Markdown
Contributor Author

rmunn commented Jul 27, 2024

Looks like this can be removed:

import {createReadStream} from 'fs'

Done in commit 7c61054.

@rmunn
Copy link
Copy Markdown
Contributor Author

rmunn commented Jul 31, 2024

@robherley - Your approving review is now considered stale because I rebased and pushed one new commit. Want to give this another look? I should be done making changes.

@robherley robherley merged commit 7463cf3 into actions:main Jul 31, 2024
@robherley robherley mentioned this pull request Aug 1, 2024
Merged
@Migelo Migelo mentioned this pull request Aug 5, 2024
Merged
9 tasks
@robherley robherley mentioned this pull request Sep 20, 2024
Merged
renovate Bot added a commit to sdwilsh/sOS that referenced this pull request Apr 29, 2026
@renovate
chore(deps): update actions/upload-artifact action to v7
e212d43 
##### [\`v7.0.1\`](https://github.com/actions/upload-artifact/releases/tag/v7.0.1)

##### What's Changed

- Update the readme with direct upload details by [@danwkennedy](https://github.com/danwkennedy) in [#795](actions/upload-artifact#795)
- Readme: bump all the example versions to v7 by [@danwkennedy](https://github.com/danwkennedy) in [#796](actions/upload-artifact#796)
- Include changes in typespec/ts-http-runtime 0.3.5 by [@yacaovsnc](https://github.com/yacaovsnc) in [#797](actions/upload-artifact#797)

**Full Changelog**: <actions/upload-artifact@v7...v7.0.1>

---
##### [\`v7.0.0\`](https://github.com/actions/upload-artifact/releases/tag/v7.0.0)

#### v7 What's new

##### Direct Uploads

Adds support for uploading single files directly (unzipped). Callers can set the new `archive` parameter to `false` to skip zipping the file during upload. Right now, we only support single files. The action will fail if the glob passed resolves to multiple files. The `name` parameter is also ignored with this setting. Instead, the name of the artifact will be the name of the uploaded file.

##### ESM

To support new versions of the `@actions/*` packages, we've upgraded the package to ESM.

#### What's Changed

- Add proxy integration test by [@Link-](https://github.com/Link-) in [#754](actions/upload-artifact#754)
- Upgrade the module to ESM and bump dependencies by [@danwkennedy](https://github.com/danwkennedy) in [#762](actions/upload-artifact#762)
- Support direct file uploads by [@danwkennedy](https://github.com/danwkennedy) in [#764](actions/upload-artifact#764)

#### New Contributors

- [@Link-](https://github.com/Link-) made their first contribution in [#754](actions/upload-artifact#754)

**Full Changelog**: <actions/upload-artifact@v6...v7.0.0>

---
##### [\`v7\`](actions/upload-artifact@v6...v7)


---
##### [\`v6\`](actions/upload-artifact@v5...v6)


---
##### [\`v6.0.0\`](actions/upload-artifact@v5.0.0...v6.0.0)


---
##### [\`v5\`](actions/upload-artifact@v4...v5)


---
##### [\`v5.0.0\`](actions/upload-artifact@v4.6.2...v5.0.0)


---
##### [\`v4.6.2\`](https://github.com/actions/upload-artifact/releases/tag/v4.6.2)

#### What's Changed

- Update to use artifact 2.3.2 package & prepare for new upload-artifact release by [@salmanmkc](https://github.com/salmanmkc) in [#685](actions/upload-artifact#685)

#### New Contributors

- [@salmanmkc](https://github.com/salmanmkc) made their first contribution in [#685](actions/upload-artifact#685)

**Full Changelog**: <actions/upload-artifact@v4...v4.6.2>

---
##### [\`v4.6.1\`](https://github.com/actions/upload-artifact/releases/tag/v4.6.1)

#### What's Changed

- Update to use artifact 2.2.2 package by [@yacaovsnc](https://github.com/yacaovsnc) in [#673](actions/upload-artifact#673)

**Full Changelog**: <actions/upload-artifact@v4...v4.6.1>

---
##### [\`v4.6.0\`](https://github.com/actions/upload-artifact/releases/tag/v4.6.0)

#### What's Changed

- Expose env vars to control concurrency and timeout by [@yacaovsnc](https://github.com/yacaovsnc) in [#662](actions/upload-artifact#662)

**Full Changelog**: <actions/upload-artifact@v4...v4.6.0>

---
##### [\`v4.5.0\`](https://github.com/actions/upload-artifact/releases/tag/v4.5.0)

#### What's Changed

- fix: deprecated `Node.js` version in action by [@hamirmahal](https://github.com/hamirmahal) in [#578](actions/upload-artifact#578)
- Add new `artifact-digest` output by [@bdehamer](https://github.com/bdehamer) in [#656](actions/upload-artifact#656)

#### New Contributors

- [@hamirmahal](https://github.com/hamirmahal) made their first contribution in [#578](actions/upload-artifact#578)
- [@bdehamer](https://github.com/bdehamer) made their first contribution in [#656](actions/upload-artifact#656)

**Full Changelog**: <actions/upload-artifact@v4.4.3...v4.5.0>

---
##### [\`v4.4.3\`](https://github.com/actions/upload-artifact/releases/tag/v4.4.3)

#### What's Changed

- Undo indirect dependency updates from [#627](actions/upload-artifact#627) by [@joshmgross](https://github.com/joshmgross) in [#632](actions/upload-artifact#632)

**Full Changelog**: <actions/upload-artifact@v4.4.2...v4.4.3>

---
##### [\`v4.4.2\`](https://github.com/actions/upload-artifact/releases/tag/v4.4.2)

#### What's Changed

- Bump `@actions/artifact` to 2.1.11 by [@robherley](https://github.com/robherley) in [#627](actions/upload-artifact#627)
  - Includes fix for relative symlinks not resolving properly

**Full Changelog**: <actions/upload-artifact@v4.4.1...v4.4.2>

---
##### [\`v4.4.1\`](https://github.com/actions/upload-artifact/releases/tag/v4.4.1)

#### What's Changed

- Add a section about hidden files by [@joshmgross](https://github.com/joshmgross) in [#607](actions/upload-artifact#607)
- Add workflow file for publishing releases to immutable action package by [@Jcambass](https://github.com/Jcambass) in [#621](actions/upload-artifact#621)
- Update [@actions/artifact](https://github.com/actions/artifact) to latest version, includes symlink and timeout fixes by [@robherley](https://github.com/robherley) in [#625](actions/upload-artifact#625)

#### New Contributors

- [@Jcambass](https://github.com/Jcambass) made their first contribution in [#621](actions/upload-artifact#621)

**Full Changelog**: <actions/upload-artifact@v4.4.0...v4.4.1>

---
##### [\`v4.4.0\`](https://github.com/actions/upload-artifact/releases/tag/v4.4.0)

#### Notice: Breaking Changes ⚠️

We will no longer include hidden files and folders by default in the `upload-artifact` action of this version. This reduces the risk that credentials are accidentally uploaded into artifacts. Customers who need to continue to upload these files can use a new option, `include-hidden-files`, to continue to do so.

See ["Notice of upcoming deprecations and breaking changes in GitHub Actions runners"](https://github.blog/changelog/2024-08-19-notice-of-upcoming-deprecations-and-breaking-changes-in-github-actions-runners/) changelog and [this issue](actions/upload-artifact#602) for more details.

#### What's Changed

- Exclude hidden files by default by [@joshmgross](https://github.com/joshmgross) in [#598](actions/upload-artifact#598)

**Full Changelog**: <actions/upload-artifact@v4.3.6...v4.4.0>

---
##### [\`v4.3.6\`](https://github.com/actions/upload-artifact/releases/tag/v4.3.6)

#### What's Changed

- Revert to [@actions/artifact](https://github.com/actions/artifact) 2.1.8 by [@robherley](https://github.com/robherley) in [#594](actions/upload-artifact#594)

**Full Changelog**: <actions/upload-artifact@v4...v4.3.6>

---
##### [\`v4.3.5\`](https://github.com/actions/upload-artifact/releases/tag/v4.3.5)

#### What's Changed

- Bump [@actions/artifact](https://github.com/actions/artifact) to v2.1.9 by [@robherley](https://github.com/robherley) in [#588](actions/upload-artifact#588)
  - Fixed artifact upload chunk timeout logic [#1774](actions/toolkit#1774)
  - Use lazy stream to prevent issues with open file limits [#1771](actions/toolkit#1771)

**Full Changelog**: <actions/upload-artifact@v4.3.4...v4.3.5>

---
##### [\`v4.3.4\`](https://github.com/actions/upload-artifact/releases/tag/v4.3.4)

#### What's Changed

- Update [@actions/artifact](https://github.com/actions/artifact) version, bump dependencies by [@robherley](https://github.com/robherley) in [#584](actions/upload-artifact#584)

**Full Changelog**: <actions/upload-artifact@v4.3.3...v4.3.4>

---
##### [\`v4.3.3\`](https://github.com/actions/upload-artifact/releases/tag/v4.3.3)

#### What's Changed

- updating `@actions/artifact` dependency to v2.1.6 by [@eggyhead](https://github.com/eggyhead) in [#565](actions/upload-artifact#565)

**Full Changelog**: <actions/upload-artifact@v4.3.2...v4.3.3>

---
##### [\`v4.3.2\`](https://github.com/actions/upload-artifact/releases/tag/v4.3.2)

#### What's Changed

- Update release-new-action-version.yml by [@konradpabjan](https://github.com/konradpabjan) in [#516](actions/upload-artifact#516)
- Minor fix to the migration readme by [@andrewakim](https://github.com/andrewakim) in [#523](actions/upload-artifact#523)
- Update readme with v3/v2/v1 deprecation notice by [@robherley](https://github.com/robherley) in [#561](actions/upload-artifact#561)
- updating `@actions/artifact` dependency to v2.1.5 and `@actions/core` to v1.0.1 by [@eggyhead](https://github.com/eggyhead) in [#562](actions/upload-artifact#562)

#### New Contributors

- [@andrewakim](https://github.com/andrewakim) made their first contribution in [#523](actions/upload-artifact#523)

**Full Changelog**: <actions/upload-artifact@v4.3.1...v4.3.2>

---
##### [\`v4.3.1\`](https://github.com/actions/upload-artifact/releases/tag/v4.3.1)

- Bump [@actions/artifacts](https://github.com/actions/artifacts) to latest version to include [updated GHES host check](actions/toolkit#1648)

---
##### [\`v4.3.0\`](https://github.com/actions/upload-artifact/releases/tag/v4.3.0)

#### What's Changed

- Reorganize upload code in prep for merge logic & add more tests by [@robherley](https://github.com/robherley) in [#504](actions/upload-artifact#504)
- Add sub-action to merge artifacts by [@robherley](https://github.com/robherley) in [#505](actions/upload-artifact#505)

**Full Changelog**: <actions/upload-artifact@v4...v4.3.0>

---
##### [\`v4.2.0\`](https://github.com/actions/upload-artifact/releases/tag/v4.2.0)

#### What's Changed

- Ability to overwrite an Artifact by [@robherley](https://github.com/robherley) in [#501](actions/upload-artifact#501)

**Full Changelog**: <actions/upload-artifact@v4...v4.2.0>

---
##### [\`v4.1.0\`](https://github.com/actions/upload-artifact/releases/tag/v4.1.0)

#### What's Changed

- Add migrations docs by [@robherley](https://github.com/robherley) in [#482](actions/upload-artifact#482)
- Update README.md by [@samuelwine](https://github.com/samuelwine) in [#492](actions/upload-artifact#492)
- Support artifact-url output by [@konradpabjan](https://github.com/konradpabjan) in [#496](actions/upload-artifact#496)
- Update readme to reflect new 500 artifact per job limit by [@robherley](https://github.com/robherley) in [#497](actions/upload-artifact#497)

#### New Contributors

- [@samuelwine](https://github.com/samuelwine) made their first contribution in [#492](actions/upload-artifact#492)

**Full Changelog**: <actions/upload-artifact@v4...v4.1.0>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@robherley robherley robherley approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[bug] EMFILE: too many open files

3 participants

@rmunn @joshmgross @robherley