Implement concatenate_dataset_dicts

[RainbowRivey]

Implements concatenate_dataset_dicts() which allows merging multiple datasets into a single one.

• concatenate_datasets() and concatenate_dataset_dicts() now have an additional option clear_metadata to clean metadata fields when merging. If True, only dataset_name field will stay in metadata. dataset_name will NOT be overwritten by subsequent concatenations, so it always contains the original dataset name.

Other changes:

• fixes Dataset.map() and IterableDataset.map() behavior when parameter function is None.
• dataset_to_document_type() converter now removes all features not declared by target document type.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.30%. Comparing base (e1db8f3) to head (4719a07).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #153      +/-   ##
==========================================
+ Coverage   92.19%   92.30%   +0.11%     
==========================================
  Files          10       10              
  Lines         897      910      +13     
==========================================
+ Hits          827      840      +13     
  Misses         70       70              

Flag	Coverage Δ
	92.30% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[RainbowRivey]

Added Tests.
However, current code uses hf datasets concatenate_datasets method, which requires documents to have identical features structure, which might not be the case: our datasets have completely different metadata fields. I'm not sure at which layer should this one be solved.

One way around this might be to prefix metadata fields with dataset name and create an new empty metadata (needed by concatenate_documents), but then all documents will have two metadata fields:

• metadata: with dataset_name (created by current concatenate method)
• %dataset%_metadata with dataset specific metadata

This should work since HF method allows fields to be empty.

Another way is not to use HF implementation and write own.

OR we can leave it as is, and preprocess datasets before merging.

[RainbowRivey]

For now i'm just removing metadata from documents when merging. It seems that HF Datasets concatenate_datasets method compares features too deeply and i see no better way around it.