Code

Miljoeportal#302

Code

Open

kris927b wants to merge 34 commits intomaincentre-for-humanities-computing/danish-foundation-models:mainfrom miljoeportalcentre-for-humanities-computing/danish-foundation-models:miljoeportalCopy head branch name to clipboard

+1,334-3

Commits

Commits on Oct 25, 2024

• Updating dolma dependency as version on pypi is now up to date

  kris927b

  committed
  34afcd0

• Adding processing script for DSK data

  kris927b

  committed
  5f4dbda

• Adhering to the desired format

  kris927b

  committed
  9a31be0

• Minor adjustments to file handling

  kris927b

  committed
  1bc6552

• Minor change to the entry id. Using filename rather than path

  kris927b

  committed
  fd7fffe

Commits on Nov 14, 2024

• Extracting util into separate file

  kris927b

  committed
  23e9537

• Testing extracting documents using Docling

  kris927b

  committed
  3fcb91b

Commits on Nov 19, 2024

• Using docling to extract data from PDF, DOCX, PPTX, MD.

  kris927b

  committed
  819648c

• Removing test files

  kris927b

  committed
  3e6c045

Commits on Nov 20, 2024

• Adding lower bounds on dependencies

  kris927b

  committed
  e89f50f

• Moving utils to dfm-data package

  kris927b

  committed
  404abf3

• Removing unused methods to clean up code

  kris927b

  committed
  52ad08a

• Fixing spelling error

  kris927b

  committed
  dc2291d

• Separating document processing in dfm-data.

  kris927b

  committed
  bf11fcc

• Adding missing docstrings.

  kris927b

  committed
  79dcae4

Commits on Nov 21, 2024

• Adding .msg functionality

  kris927b

  committed
  32768f3

• Adding extract-msg to pyproject.

  kris927b

  committed
  19244de

Commits on Dec 4, 2024

• Adding a separate process_files method to handle the base logic of processing a list of file paths.

  kris927b

  committed
  0710e13

• Refactoring process_directory command, and adding command to process dsk crawled data.

  kris927b

  committed
  faeca4f

• Merge branch 'main' into dsk_processing

  kris927b

  committed
  b169b34

Commits on Dec 5, 2024

• Adding support for bytes streams in document processing.

  kris927b

  committed
  39a8693

• Preliminary scripts for pulling data from Cellar

  kris927b

  committed
  b6207f0

Commits on Dec 6, 2024

• Minor changes

  kris927b

  committed
  69e09ac

• Adding parallelization

  kris927b

  committed
  dd26b94

Commits on Dec 9, 2024

• Handling connection errors

  kris927b

  committed
  69bf7fb

• Fixing a problem with ordering of mtypes

  kris927b

  committed
  7b1e5de

• Updating logger

  kris927b

  committed
  154f8ca

• Adding extra error handling

  kris927b

  committed
  ff52bc4

Commits on Dec 13, 2024

• Adding method to run a task in parallel using Joblib. With retries incase of worker termination errors.

  kris927b

  committed
  e8f7893

• Using new parallel processing method with retries

  kris927b

  committed
  b929d75

• Adding extra fallback

  kris927b

  committed
  01578b1

• Adding decode error handling

  kris927b

  committed
  720e2c3

Commits on Dec 16, 2024

• A sequential download option of the miljoeportal data.

  kris927b

  committed
  393feb9

Commits on Dec 17, 2024

• Adding the possibility to process json files

  kris927b

  committed
  8988271