Skip to content

Conversation

@BrianMichell
Copy link
Collaborator

When scaling segy ingestions we observed higher than expected memory usage. I believe this is in part due to unavoidable overhead in the Xarray API internals. For that reason I have begun to strip out xarray API calls in favor of zarr.

This appears to have helped reduce large scale memory usage, although it is still consuming more memory than expected.

@codecov
Copy link

codecov bot commented Oct 9, 2025

Codecov Report

❌ Patch coverage is 14.28571% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.87%. Comparing base (c3ba558) to head (104507c).
⚠️ Report is 191 commits behind head on main.

Files with missing lines Patch % Lines
src/mdio/segy/_workers.py 10.00% 18 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #710      +/-   ##
==========================================
+ Coverage   85.30%   88.87%   +3.56%     
==========================================
  Files          46       84      +38     
  Lines        2219     4529    +2310     
  Branches      306      294      -12     
==========================================
+ Hits         1893     4025    +2132     
- Misses        281      435     +154     
- Partials       45       69      +24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@BrianMichell BrianMichell marked this pull request as ready for review October 13, 2025 16:19
@tasansal tasansal self-requested a review October 13, 2025 20:00
@tasansal tasansal added the enhancement New feature or request label Oct 13, 2025
@tasansal tasansal changed the title Prefer native zarr for trace worker Use native zarr for trace worker instead of xarray for memory efficiency. Oct 13, 2025
@tasansal tasansal merged commit bdebc7e into TGSAI:main Oct 13, 2025
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants