Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Current »

Transformer Bottleneck

Original Paper (Neurips 2019)

Flow forecast implementation

Stream Flow Forecast Example

Succinct summary: A variation of the transformer that replaces standard self attention with convolutions for the query and value vectors. Uses learned positional embedding. Preformed well large temporal datasets.

Model Additions

  • We added another final layer to enable variable length forecasts. Currently using that to see how the model performs with variable forecast_length. You can choose to exclude this layer if you desire.

  • We also enabled the model to work with standard loss functions (MSE, RMSE, etc).

  • Currently working on incorporating meta-data into the model.

Simple Transformer

s

Transformer Decoder

  • No labels