Power Forecast - Failed

Let's talk about how I fail in this challenge

My Original Plan

1. Collect Static data

Like temperature, wind speed, gas price, stock prize, geolocation data. Also, categorical data like if there's a wildfire, storm, economic crisis are also included.

2. Collect real-time data (for text mining usage)

Using Facebook and Twitter API to scrape real time comments from users. Evaluate their emotion and take it as the factors

3. Machine Learning methods

I would apply the random forest first. After applying some feature selection methods, I would continuously apply deep learning like CNN or other neural networks

4. Goal

before forecasting the average energy usage, I'm more prone to know what factors influence users behavior toward using energy

In sum, I hope to predict the energy usage in an user's perspective. I believe this may help with your forecast :) However, I am too naive. I encountered several difficulties to find and manipulate data sets.

1. Collect Data

(a.) Failed to scrape current and forecast demand data http://www.caiso.com/TodaysOutlook/Pages/default.aspx (b.) Could not find climate data in year 2017 https://www.ncdc.noaa.gov/cdo-web/search

2. Manipulate Data

The most difficult part is to manipulate each 15 minute records. Even though I could collect the data, I failed to convert those 15 minutes data into an hourly data (*note: there're bunch of database which do not provide 15 minutes data).

3. Draw time series plot

It seems to me that I still need to improve my data manipulation skills especially in cleaning the Datetime column. I often got the same error messages like "Error: Invalid input: date_trans works with objects of class Date only".

What I learned

Although I did not finish this project, I still learn a lot of data manipulation. Also, this experience motivates me to approach to many data resources and play with it. It's a pity I failed in the end, but I would continuously stride forward to the road of data forecasting :)