Skip to content

Conversation

@BrianMichell
Copy link
Collaborator

@BrianMichell BrianMichell commented Sep 10, 2025

Supersedes #634

  • Only supports Zarr v3 configuration since v2 handles the raw bytes differently.
  • Ensures we retain endianness of the raw headers.

Pending work

  • Memory and CPU profiling

Notes:
Requires environment variable MDIO__DO_RAW_HEADERS 1
Will generate a logger warning regarding experimental status if the feature is enabled.

@codecov
Copy link

codecov bot commented Sep 10, 2025

Codecov Report

❌ Patch coverage is 91.69960% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.01%. Comparing base (c3ba558) to head (90e7cdb).
⚠️ Report is 169 commits behind head on main.

Files with missing lines Patch % Lines
src/mdio/segy/_workers.py 11.11% 8 Missing ⚠️
...ests/integration/test_segy_import_export_masked.py 88.88% 5 Missing and 2 partials ⚠️
src/mdio/converters/segy.py 84.00% 2 Missing and 2 partials ⚠️
tests/unit/test_disaster_recovery_wrapper.py 98.42% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #659      +/-   ##
==========================================
+ Coverage   85.30%   87.01%   +1.71%     
==========================================
  Files          46       82      +36     
  Lines        2219     4576    +2357     
  Branches      306      303       -3     
==========================================
+ Hits         1893     3982    +2089     
- Misses        281      515     +234     
- Partials       45       79      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tasansal tasansal force-pushed the disaster_recovery_multiVersionSupport branch 2 times, most recently from 908c97b to 9ae7ab6 Compare September 17, 2025 14:29
@tasansal tasansal force-pushed the disaster_recovery_multiVersionSupport branch from 9ae7ab6 to 7a30489 Compare September 19, 2025 10:40
BrianMichell and others added 7 commits September 30, 2025 15:52
* Attempt to use view

* Add hex-dump and MDIO output reproducer

* Fixes

* Cleanup

* Provide clean disaster recovery interface

* Begin work on tests

* Fix flattening issue

* Push for debugging

* Numpy updates

* Testing

* Working end-to-end examples

* Cleanup

* Bandaid fix

* linting pass 1

* Fix logic issue

* Use wrapper class

* Precommit

* Remove external debugging code

* Remove debug code

* Remove errant numpy additon to pyproject toml

* Fix uv lock to mainline

* Pre-commit

* Remove raw field additions. Depends on segy >= 0.5.1

* Removed raw byte inserts (#10)

* Update Xarray api access (TGSAI#688)

* Reimplement disaster recovery logic

* Ensure getting true raw bytes for DR array

* Linting

* Add v2 issue check

* Fix pre-commit

* Profiled disaster recovery array (#8)

- Avoids duplicate read regression issue
- Implements isolated and testable logic

* Fix unclosed parenthesis

* Linting

* Test DR compatibility with all tested schemas

* Fix missing test fixture error

* Suppress unused linting error

* Attempt to use view

* Add hex-dump and MDIO output reproducer

* Fixes

* Cleanup

* Provide clean disaster recovery interface

* Begin work on tests

* Fix flattening issue

* Push for debugging

* Numpy updates

* Testing

* Working end-to-end examples

* Cleanup

* Bandaid fix

* linting pass 1

* Fix logic issue

* Use wrapper class

* Precommit

* Remove external debugging code

* Remove debug code

* Remove errant numpy additon to pyproject toml

* Fix uv lock to mainline

* Pre-commit

* Removed raw byte inserts

Removed the insertions of raw bytes into the raw bytes Variable. This issue will be addressed in tgsai/segy release >0.5.1

* Use new segy API calls

* Updates to get working

* Use released version

* Linting
@tasansal tasansal self-requested a review September 30, 2025 22:18
@tasansal tasansal added the enhancement New feature or request label Sep 30, 2025
Copy link
Contributor

@tasansal tasansal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but hopefully we can remove this soon!

@tasansal tasansal changed the title Disaster recovery array option Environment variable flag to preserve unmodified, raw trace headers from SEG-Y as variable. Sep 30, 2025
@tasansal tasansal merged commit 1b98255 into TGSAI:main Sep 30, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants