Function for parsing WPC coded surface bulletins #1944

23ccozad · 2021-06-28T19:52:46Z

In response to issue #1153, the PR introduces a new function in the IO module for parsing WPC coded surface bulletins. Provided a path to a file, or a file-like object, the function returns a geopandas GeoDataFrame containing the data from the bulletin. Each row of the dataframe represents a pressure center or frontal boundary. The main use of this function is to provide support for the ability to plot surface analyses with MetPy.

Additionally, this PR includes a small change to open_as_needed() in _tools.py. In order to get some WPC surface bulletins, I was using urllib.request.urlopen("https://www.wpc.ncep.noaa.gov/basicwx/coded_srp.txt"), which returns an HTTPRequest object. open_as_needed() was throwing an error when trying to use seek(0) on that HTTPRequest object. Maybe there is a better solution than adding a try/except, so any additional feedback on that would be great.

Checklist

Closes WPC Frontal Analysis #1153
Tests added
Fully documented

23ccozad · 2021-06-28T21:57:33Z

How do I add a new text file to our staticdata folder and use get_test_data() to access it from a test function?

I've tried adding my file to staticdata on my local repo, but when trying to use get_test_data(), it looks for the file on the Unidata repo (https://github.com/Unidata/MetPy/raw/main/staticdata/WPC_sfc_fronts_20210628_1800.txt)

23ccozad · 2021-06-30T20:23:09Z

Above issue resolved by recreating the data registry with python -c "import pooch; pooch.make_registry('staticdata', 'src/metpy/static-data-manifest.txt')" after putting the new data file in the staticdata folder.

src/metpy/io/__init__.py

 from .metar import *  # noqa: F403
 from .nexrad import *  # noqa: F403
 from .station_data import *  # noqa: F403
+from .text import *  # noqa: F403


tests/io/test_text.py

+    """Test parser reading a WPC coded surface bulletin into a geodataframe."""
+    # Get rows 17 and 47 from dataframe representing parsed text file
+    # Row 17 is a pressure center and row 47 is front
+    input_file = get_test_data('WPC_sfc_fronts_20210628_1800.txt')


dopplershift · 2023-03-31T17:27:08Z

@23ccozad I've taken what you started and pushed it across the finish line, leveraging what's in #2420. Can you mark this as no longer a draft? I'm unable to do so.

dopplershift · 2023-04-07T21:16:42Z

Ok, rebased on #2420 so that we can sensibly plot. One idea I had was to make an example of a simple low pressure system plot using this stuff as a simpler example using the path effects.

EDIT: Simple example done separately in #2997

dopplershift · 2023-04-07T21:18:39Z

Hrmm...also one other issue: We're currently optionally depending on Shapely, which is no burden since it comes in with the optional dependency on Cartopy. This would be harder to keep optional here.

dcamron

I really like how this is done, thanks y'all. Just some doc and testing thoughts, mostly.

src/metpy/io/text.py

tests/io/test_text.py

dopplershift · 2023-04-11T18:55:33Z

@dcamron any thoughts on what to do with Shapely (e.g. depending on it or jumping through hoops)?

dcamron · 2023-04-11T20:51:38Z

Hmmm... Would it be more difficult than the cartopy machinery you've already put in place? Not to totally rebuild WPC parsing to be able to work without Shapely, but to just have it fail on run instead of import without it.

I think realistically it would affect very few people negatively to fully depend on Shapely. It does feel a little conceptually gross if we've already done similar work to help out those very few people, and Shapely does feel ever so slightly heavier to jam into all of metpy.io than say Pandas.

dopplershift · 2023-04-12T01:07:10Z

We could import Shapely inside the wpc parsing function since it's only used there. That seems pretty limited. We'd need to add some machinery to require shapely for the tests, which isn't scary. I'd say that's the way to go.

This requires a year value potentially to be provided, though we try to look for one in the product header.

Users can easily convert to a GeoDataFrame if they need that functionality, but the DataFrame is already a dependency and is sufficient for our needs.

This avoids an import-time failure for MetPy if Shapely isn't installed.

tests/io/test_text.py

+    # Row 17 is a pressure center and row 47 is front
+    import shapely.geometry as sgeom
+
+    input_file = get_test_data('WPC_sfc_fronts_lowres_20210628_1800.txt')


As opposed to the "high res" product.

dcamron

Alright, I think this is good to go in!

23ccozad added this to the 1.1.0 milestone Jun 28, 2021

23ccozad added Area: IO Pertains to reading data Area: Plots Pertains to producing plots Type: Feature New functionality and removed Area: Plots Pertains to producing plots labels Jun 28, 2021

23ccozad force-pushed the wpc_fronts branch from e83148c to 31b82bc Compare June 30, 2021 19:39

23ccozad force-pushed the wpc_fronts branch 3 times, most recently from c0441f0 to 8f057be Compare July 1, 2021 18:59

dopplershift modified the milestones: 1.1.0, 1.2.0 Aug 2, 2021

dopplershift modified the milestones: 1.2.0, 1.3.0 Jan 14, 2022

dopplershift modified the milestones: 1.3.0, May 2022 Mar 31, 2022

dopplershift mentioned this pull request Apr 8, 2022

Fronts Plotting Support #2420

Merged

8 tasks

dopplershift modified the milestones: May 2022, July 2022 May 16, 2022

dopplershift removed this from the October 2022 milestone Oct 18, 2022

dopplershift force-pushed the wpc_fronts branch from 8f057be to 9d08f87 Compare March 30, 2023 21:58

github-advanced-security bot found potential problems Mar 30, 2023

View reviewed changes

dopplershift force-pushed the wpc_fronts branch from 9d08f87 to 0f3b160 Compare March 31, 2023 17:26

dopplershift force-pushed the wpc_fronts branch from 0f3b160 to e26954f Compare March 31, 2023 17:29

dopplershift added this to the April 2023 milestone Mar 31, 2023

23ccozad marked this pull request as ready for review March 31, 2023 17:53

23ccozad requested a review from a team as a code owner March 31, 2023 17:53

23ccozad removed the request for review from a team March 31, 2023 17:53

23ccozad requested a review from dopplershift March 31, 2023 17:53

dopplershift force-pushed the wpc_fronts branch 2 times, most recently from 1bfb415 to 976556e Compare April 2, 2023 07:38

dopplershift force-pushed the wpc_fronts branch from 976556e to f0c1da7 Compare April 7, 2023 21:15

dopplershift force-pushed the wpc_fronts branch 2 times, most recently from ce900bf to 7736950 Compare April 7, 2023 21:59

dopplershift mentioned this pull request Apr 7, 2023

WPC Frontal Analysis #1153

Closed

dcamron requested changes Apr 11, 2023

View reviewed changes

src/metpy/io/text.py Outdated Show resolved Hide resolved

src/metpy/io/text.py Show resolved Hide resolved

src/metpy/io/text.py Show resolved Hide resolved

tests/io/test_text.py Outdated Show resolved Hide resolved

23ccozad and others added 6 commits April 17, 2023 15:11

New function and test for parsing WPC coded surface bulletin

1e6fba8

Modify tests to import geopandas (optional dependency)

eeacda5

ENH: Refactor to break out lines, handle corner cases, fix test

9f77d81

ENH: Parse valid time from product

29a2c48

This requires a year value potentially to be provided, though we try to look for one in the product header.

ENH: Replace geopandas with pandas

b7315b1

Users can easily convert to a GeoDataFrame if they need that functionality, but the DataFrame is already a dependency and is sufficient for our needs.

ENH: Add example of plotting fronts

a4e5b01

dopplershift force-pushed the wpc_fronts branch 2 times, most recently from c282ceb to 89ef972 Compare April 17, 2023 20:33

ENH: Make shapely dependence for WPC only at call-time

660076a

This avoids an import-time failure for MetPy if Shapely isn't installed.

dopplershift force-pushed the wpc_fronts branch from 89ef972 to c53a328 Compare April 17, 2023 20:37

dopplershift requested a review from dcamron April 17, 2023 20:39

github-advanced-security bot found potential problems Apr 17, 2023

View reviewed changes

tests/io/test_text.py

# Row 17 is a pressure center and row 47 is front

import shapely.geometry as sgeom

input_file = get_test_data('WPC_sfc_fronts_lowres_20210628_1800.txt')

Check warning

Code scanning / CodeQL

File is not always closed

File is opened but is not closed.

TST: Add a test for normal WPC surface bulletin

45b01b5

As opposed to the "high res" product.

dopplershift force-pushed the wpc_fronts branch from c53a328 to 45b01b5 Compare April 18, 2023 14:09

dcamron approved these changes Apr 18, 2023

View reviewed changes

dcamron merged commit 0fcfd64 into Unidata:main Apr 18, 2023

Function for parsing WPC coded surface bulletins #1944

Function for parsing WPC coded surface bulletins #1944

Uh oh!

Conversation

23ccozad commented Jun 28, 2021 • edited by dopplershift Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

23ccozad commented Jun 28, 2021

Uh oh!

23ccozad commented Jun 30, 2021

Uh oh!

Check notice

Check warning

dopplershift commented Mar 31, 2023

Uh oh!

dopplershift commented Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dopplershift commented Apr 7, 2023

Uh oh!

dcamron left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dopplershift commented Apr 11, 2023

Uh oh!

dcamron commented Apr 11, 2023

Uh oh!

dopplershift commented Apr 12, 2023

Uh oh!

Check warning

dcamron left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

23ccozad commented Jun 28, 2021 •

edited by dopplershift

Loading

dopplershift commented Apr 7, 2023 •

edited

Loading