/
Transformer Based Models

Transformer Based Models

Owned by Isaac Godfried
Last updated: Mar 16, 2021

Transformer Bottleneck

Original Paper (Neurips 2019)

Flow forecast implementation

Stream Flow Forecast Example

Solar Energy Forecasting Example

Succinct summary: A variation of the transformer that replaces standard self attention with convolutions for the query and value vectors. Uses learned positional embedding. Preformed well large temporal datasets.

Model Additions

  • We added another final layer to enable variable length forecasts. Currently using that to see how the model performs with variable forecast_length. You can choose to exclude this layer if you desire.

  • We also enabled the model to work with standard loss functions (MSE, RMSE, etc).

  • Currently working on incorporating meta-data into the model.

Simple Transformer

 

Original Paper

River Flow Forecasting Example

Succinct summary: This is a vari

Transformer Decoder

  •  

 

Informer

Related content