Code

[Draft] RWKV LM#207

Code

Open

Kyle1668 wants to merge 22 commits intomainEleutherAI/elk:mainfrom kyle-rwkv-lmEleutherAI/elk:kyle-rwkv-lmCopy head branch name to clipboard

+100,735-22

Commits

Commits on Apr 17, 2023

• Begin scaffolfing rwkv hf interface

  Kyle1668

  committed
  3f33328

Commits on Apr 18, 2023

• RWKV works maybe?

  Kyle1668

  committed
  fa55ee3

• prog

  Kyle1668

  committed
  b7b8aec

Commits on Apr 22, 2023

• Pivot to getting the token embeddings

  Kyle1668

  committed
  d94132d

• Additional notebook tests

  Kyle1668

  committed
  d61375c

• Reset extraction

  Kyle1668

  committed
  69194ee

• Add instantiate_config(cfg.model)

  Kyle1668

  committed
  c84a74d

• Refactored RWKV and merged from main

  Kyle1668

  committed
  15baa83

Commits on Apr 24, 2023

• Try exposing hiddens

  Kyle1668

  committed
  f4c88d5

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 1a0dcae
  pre-commit-ci[bot]

  committed
  1a0dcae

Commits on Apr 25, 2023

• Add HF hub download links within the model code

  

  kyobrien

  committed
  53e3f04

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 55ad3b4
  pre-commit-ci[bot]

  committed
  55ad3b4

Commits on Apr 27, 2023

• Got better than random results

  Kyle1668

  committed
  2fa92d6

Commits on May 6, 2023

• Merge from main

  Kyle1668

  committed
  aeb1f20

• Got good results with post-ln

  Kyle1668

  committed
  6c6a324

• Stop tracking notebooks in git

  Kyle1668

  committed
  2536e5a

• Stop tracking VS code debugging config

  Kyle1668

  committed
  dc173fb

• Remove custom approach and begin generalizing

  Kyle1668

  committed
  a970761

• Further refactoring

  Kyle1668

  committed
  7ae8637

• Furhter cleanup

  Kyle1668

  committed
  086ecc0

• Undo settings

  Kyle1668

  committed
  14f66e1

Commits on May 8, 2023

• Oly get one hidden state per layer

  Kyle1668

  committed
  2102b4b