Page Comparison

Current Roadblock:
- Fitting data into memory for modeling

Tasks Completed Till Now

• Step 1: Batchwise Data Processing for Preprocessing
  The favorita dataset is too large to fit into memory for in one go. So, it is processed in chunks of 300000.

• Step 2: Data Preprocessing
  In this following dataset each product number-store number pair is treated as a separate entity and is denoted by an embedding of the following variables:
  

  Code Block
  language py
  ['holiday_type', 'locale', 'locale_name', 'description', 'transferred', 'city', 'state', 'store_type', 'cluster', 'family', 'class', 'perishable']

1. We treat each product number-store number pair as a separate entity

2. We include an additional ’open’ flag to denote whether data is present on a given day

3. Data is resampled at regular daily intervals,imputing any missing days using the last available observation

4. We apply a log-transform on the sales data, and adopt z-score normalization across all entities

5. Dropping where any record missing

6. The training set is made up of samples taken between 2015-01-01 to 2015-12-01. The validation set of samples from the 30 days after the training set. The test set of all entities over the 30-day horizon following the validation set.

7. We consider log sales, transactions, oil to be real-valued and the rest to be categorical.
   

• Step 3: Modelling
  Use DA-RNN with transfer learning. Notebook for the same can be found at:
  https://colab.research.google.com/drive/15kyHxeZCQLx9PpFSZpB8TKECiCZj_Pg7