note

This template is brought to you by Optimizely, a leading experimentation platform.

This template is brought to you by Optimizely, a leading experimentation platform.

Experiment plan and results

Replicating Results of Transformer Fusion Model using Favourita Dataset
https://arxiv.org/pdf/1912.09363.pdf

Experiment owner

kriti mahajan

Reviewers

Approver

  • @ Approver

Optimizely link

e.g., https://app.optimizely.com/

Jira ticket(s)

Status

/ /

On this page

note

Stakeholder summary

Add a summary of results here.

Stakeholder summary

(blue star) Experiment planning

Overview: Replicating Results of Transformer Fusion Model using Favourita Dataset
https://arxiv.org/pdf/1912.09363.pdf

Current Roadblock:
- Fitting data into memory for modeling

Tasks Completed Till Now

  • Step 1: Batchwise Data Processing for Preprocessing
    The favorita dataset is too large to fit into memory for in one go. So, it is processed in chunks of 300000.

  • Step 2: Data Preprocessing
    In this following dataset each product number-store number pair is treated as a separate entity and is denoted by an embedding of the following variables:

    ['holiday_type',
    'locale',
    'locale_name',
    'description',
    'transferred',
    'city',
    'state',
    'store_type',
    'cluster',
    'family',
    'class',
    'perishable']
  1. We treat each product number-store number pair as a separate entity

  2. We include an additional ’open’ flag to denote whether data is present on a given day

  3. Data is resampled at regular daily intervals,imputing any missing days using the last available observation

  4. We apply a log-transform on the sales data, and adopt z-score normalization across all entities

  5. Dropping where any record missing

  6. The training set is made up of samples taken between 2015-01-01 to 2015-12-01. The validation set of samples from the 30 days after the training set. The test set of all entities over the 30-day horizon following the validation set.

  7. We consider log sales, transactions, oil to be real-valued and the rest to be categorical.

Hypothesis

We hypothesize that DA-RNN with Transfer learning/ Transformer model with Transfer Learning

will decrease MSE/RMSE

because of the incorporation of embeddings

Metrics

List your primary and second metrics for this experiment:

Targeting

Use this section to answer questions like:

Variations

A: Control

B: Variation

C: Variation

Screenshot

% of visitors/users to see each variation

Pre-analysis

Add any baseline data or pre-analysis you have for this experiment. Add planned sample size and time to run.

Notes

What feedback or gotchas do you have to share with internal teams (bugs, process or product gaps,etc.)?

(blue star) Results

Experiment start

e.g.,

Experiment end

e.g.,

Link to results in Optimizely

e.g., https://app.optimizely.com/

Conclusion

/

Add a short summary of the metrics below and whether you hit significance.

A: Control

B: Variation

change

A: Control

C: Variation

change

Cohort size

Primary metric

Δ=

p-value=

power=

confidence=

Δ=

p-value=

power=

confidence=

Other metrics

(blue star) Conclusions

Highlights

Takeaways

Follow-up