Code

[WIP] Implements Roberta Model#679

Code

Draft

JulienDarve wants to merge 32 commits intostanford-crfm:mainstanford-crfm/levanter:mainfrom JulienDarve:roberta-modelJulienDarve/levanter:roberta-modelCopy head branch name to clipboard

+2,779-23

Commits

Commits on Jul 30, 2024

• [WIP] Implements Roberta Model

  

  JulienDarve

  committed
  d6284ca

• Implements dynamic masking objective

  prady-saligram

  committed
  8f7402e

• Implements dynamic masked dataset

  prady-saligram

  committed
  670b053

• Reintroduced accidentally deleted CausalLMDataset class

  prady-saligram

  committed
  42f5404

Commits on Aug 1, 2024

• Everything works except stuck on the final method,

  Show description for 9ad06af
  

  JulienDarve

  committed
  9ad06af

Commits on Aug 5, 2024

• [WIP] Re-implements MLM training objective

  prady-saligram

  committed
  53fd8d2

Commits on Aug 6, 2024

• Adds error handling and reverts LmExample class to original

  prady-saligram

  committed
  dcd45b2

Commits on Aug 13, 2024

• Testing Modifications

  

  JulienDarve

  committed
  6f21e0d

Commits on Aug 26, 2024

• Merge branch 'stanford-crfm:main' into roberta-model

  prady-saligram

  authored
  730d847

• Sets RobertaConfig as model architecture and creates default config file

  prady-saligram

  committed
  027b176

Commits on Sep 1, 2024

• Adds compute_loss to roberta and changes positional ids to begin from 0

  prady-saligram

  committed
  399e08c

Commits on Sep 4, 2024

• Investingating precision loss over time within the model using output_hidden_states implementation in jax model

  

  JulienDarve

  committed
  cd4118c

• Merge branch 'roberta-model' of https://github.com/JulienDarve/levanter into roberta-model

  

  JulienDarve

  committed
  96522f1

Commits on Sep 10, 2024

• Model can now successfully import weights from huggingface + made attention mask more robust

  

  JulienDarve

  committed
  8a732e5

• Merge branch 'roberta-training' into roberta-model-copy-2

  

  JulienDarve

  committed
  5f3d8a2

Commits on Sep 12, 2024

• trial

  

  JulienDarve

  committed
  6c105f5

• update 1

  

  JulienDarve

  committed
  ab85079

• update 2

  

  JulienDarve

  committed
  5b97400

• update 3

  

  JulienDarve

  committed
  bd7d411

• update

  

  JulienDarve

  committed
  b5d8e14

• update

  

  JulienDarve

  committed
  8717c3f

• update

  

  JulienDarve

  committed
  10c130c

• update

  

  JulienDarve

  committed
  834d88d

• update

  

  JulienDarve

  committed
  47fe23b

• update

  

  JulienDarve

  committed
  fb5c55c

• update

  

  JulienDarve

  committed
  8594e79

• update

  

  JulienDarve

  committed
  3ae80d7

• update

  

  JulienDarve

  committed
  de93fc9

• update

  

  JulienDarve

  committed
  896af7d

• update

  

  JulienDarve

  committed
  0be9a83

• update

  

  JulienDarve

  committed
  0c94a47

Commits on Sep 13, 2024

• Training works!

  JulienDarve

  committed
  7ae681d