Import Documents: Read metas as the right type

[ajdapretnar]

Issue

Import Documents added meta files if found, but they were all read as StringVariables not as their own proper types.

Description of changes

Use table_from_frame to translate numeric values to ContinuousVariables and categorical to DiscreteVariables.

CONS: now some attributes are in X, not in metas. Any easy way to fix this?

Decided to follow @VesnaT suggestion and sanitize variables before sending them to guess_data_type, which is now used instead of StringVariable.

Includes

• Code changes
• Tests
• Documentation

[VesnaT]

CONS: now some attributes are in X, not in metas. Any easy way to fix this?

Not that I'm aware of.
Have you considered using guess_data_type function (and keeping the old approach)?

[VesnaT]

If you want to use the table_from_frame, you need to reset the df.index:

filtered = table_from_frame(filtered.reset_index(drop=True))

[ajdapretnar]

I'll try the above suggestion first, thanks! ^^

[codecov-commenter]

Codecov Report

Merging #677 (42405e2) into master (b9b9612) will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #677      +/-   ##
==========================================
+ Coverage   74.02%   74.07%   +0.04%     
==========================================
  Files          72       72              
  Lines        9155     9230      +75     
  Branches     1239     1253      +14     
==========================================
+ Hits         6777     6837      +60     
- Misses       2138     2150      +12     
- Partials      240      243       +3     

[ajdapretnar]

@VesnaT Should we keep this? Currently, null values in yamls are treated as "None" (strings).

[VesnaT]

I'd rather put fixes like this inside the YamlMetaReader.
Besides, I'm not sure the fix works correctly for theStringVariables.

[VesnaT]

Something like this could work:

class YamlMetaReader(Reader):
    ext = [".yaml"]

    def read_file(self):
        with open(self.path, "r") as f:
            self.content = yaml.safe_load(f)
            for k in self.content:
                if self.content[k] is None:
                    self.content[k] = ""