pretrain doesn't work on json\jsonl

[SicariusSicariiStuff]

Please check that this issue hasn't been reported before.

• I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

To work the same as when loading the dataset from HF

Current behaviour

Asks for a custom .py script

Steps to reproduce

Load a local json file:

pretraining_dataset: /home/sicarius/somefile.jsonl
type: pretrain

Config yaml

pretraining_dataset: /home/sicarius/somefile.jsonl
    type: pretrain

Possible solution

Treat it similarly as a loading a dataset from the HF hub

Which Operating Systems are you using?

• Linux
• macOS
• Windows

Python Version

3.10

axolotl branch-commit

latest release

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this bug has not been reported yet.
• I am using the latest version of axolotl.
• I have provided enough information for the maintainers to reproduce and diagnose the issue.

[NanoCode012]

Hey, sorry it's been a while. We are currently internally discussing providing better support for this and pre-training/sft in general. We plan to extend support to local and cloud storage (S3 etc).