From the course: Data Pipeline Automation with GitHub Actions Using R and Python
Sending a simple GET request with R - GitHub Tutorial
From the course: Data Pipeline Automation with GitHub Actions Using R and Python
Sending a simple GET request with R
- [Instructor] In the previous video, we saw how we can use the EIA metadata function to pull metadata from the API. In this video, we will use the metadata we pull to set and send to the API, a simple get request to pull data with the EIA get function. To set a simple get request with the EIA get function, we will need to define the follow four arguments, API key, API path, frequency, facets. Let's go and set those arguments using the above variables. Similar to the get metadata function, the API key and the API path defines the API access key and route. We will use the same method as before to load the API key using the Sys.getenv function and load it from the env variable, EI_API_KEY. Likewise, we will use the same route as before and add the data at the end of the route as we wanted to inform the API that this is a data request and not a metadata. Next, is the frequency argument. Let's go ahead to the metadata output and see what are the options. You can have it over here on the right side, and you can see frequency. Under the frequency, there are two options. First is hourly, which define the timestamp in UTC time zone. And second one is local hourly which set the time zone with the local time zone. Let's go ahead and set it as hourly as we want to be consistent across this course using UTC time zone, so I'm going to set it as hourly. Last but not least, is the facets argument which enable us to select a specific series within the API route. In this case, the facet argument is the following two options, and we can go ahead again to the output of the metadata. And you can see there are two options, subba, which stand for Subregion, and parents, which stand for balancing authority. Let's go back to a above screenshot to pull the facet setting for the PGAE series. As you can see, the parent is defined here as CISO and the subba is defined as PGAE. We use the list object to set the facet parameters for the EIA get function. So let's go ahead and set the parent as CISO and the subba as PGAE. Let's execute it and run the function. The output indicates that the function pull data. Let's explore it with the str function to review the return object structure. As you can see, it retained a data frame with 5,000 observation which is the API limit. This include the following seven variables, period, which define the observation, timestamp, it's set as character. We will need to reformat it later using the Lubridate package into a time object. Subba and subba-name define the series subregion, code, and name. Similarly, parent and parent-name define the series parent, balancing authority, code, and name. Last but not least, are the value and value-unit which define the serious values and the corresponding units which in this case, it's megawatthours. Let's now reformat the series timestamp using deployer. We will use mutate to create a new variable call index and use the year, month, day, hour function or in short, ymd_h, from the Lubridate package just to reformat the period variables into a POSIX object. Be sure to set the time zone as UTC. We're going to reformat the structure, having the index first and then the rest of the variables using the select functions from deployer. And last but not least, we're going to arrange the data by the index. Let's go ahead and look at the output. Maybe it will be easier to look from here, head df1. And as you can see, we got the object. We have now a new variable call index with timestamp, and the rest of the variables are also there. In the next video, we'll explore the series and review some of the limitations of the EIA get function for pulling hourly data.
Contents
-
-
-
EIA API2m 47s
-
Setting an environment variable3m 22s
-
The EIA API dashboard4m 10s
-
GET request structure5m 41s
-
Querying the data via the browser4m 4s
-
Querying data with R and Python2m 50s
-
Pulling metadata from API with R3m 5s
-
Sending a simple GET request with R5m 19s
-
API limitations with R4m 43s
-
Handling a large data request with R4m 27s
-
Pulling metadata from API with Python3m 47s
-
Sending a simple GET request with Python4m 44s
-
API limitations with Python3m 54s
-
Handling a large data request with Python3m 10s
-
Challenge: Query the API1m 2s
-
Solution: Query the API with R7m 28s
-
Solution: Query the API with Python7m 45s
-
-
-
-
-