Model Train with Ludwig does not return panda dataframe on Colab

[xzdandy]

Search before asking

• I have searched the EvaDB issues and found no similar bug report.

Bug

09-19-2023 05:49:28 ERROR [plan_executor:plan_executor.py:execute_plan:0182] Batch constructor not properly called.
Expected pandas.DataFrame
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/plan_executor.py", line 178, in execute_plan
    yield from output
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/limit_executor.py", line 40, in exec
    for batch in child_executor.exec(**kwargs):
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/project_executor.py", line 45, in exec
    batch = apply_project(batch, self.target_list, self.catalog())
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/executor_utils.py", line 46, in apply_project
    batches = [expr.evaluate(batch) for expr in project_list]
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/executor_utils.py", line 46, in <listcomp>
    batches = [expr.evaluate(batch) for expr in project_list]
  File "/usr/local/lib/python3.10/dist-packages/evadb/expression/function_expression.py", line 129, in evaluate
    outcomes = self._apply_function_expression(func, batch, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/evadb/expression/function_expression.py", line 188, in _apply_function_expression
    return func_args.apply_function_expression(func)
  File "/usr/local/lib/python3.10/dist-packages/evadb/models/storage/batch.py", line 173, in apply_function_expression
    return Batch(expr(self._frames))
  File "/usr/local/lib/python3.10/dist-packages/evadb/models/storage/batch.py", line 43, in __init__
    raise ValueError(
ValueError: Batch constructor not properly called.
Expected pandas.DataFrame

Environment

https://colab.research.google.com/drive/1omSVrAOQvWmGkdbGSTWyMlQA90n1eWvQ#scrollTo=_7m-QQG5U3_C

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[xzdandy]

The problem can not be reproduced on a local linux environment. The suspicious is the backend changes on Colab and a Dask dataframe is returned.

[xzdandy]

Interestingly, both colab and local environment have dask==2023.3.2 installed, which makes a difference is distributed==2023.8.1 is installed on Colab but not locally. However, with better error message, we can confirm the return type of LudwigModel.predict becomes dask.dataframe.core.DataFrame instead of pandas.DataFrame.

09-19-2023 06:56:48 ERROR [plan_executor:plan_executor.py:execute_plan:0182] Batch constructor not properly called.
Expected pandas.DataFrame, got <class 'dask.dataframe.core.DataFrame'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/plan_executor.py", line 178, in execute_plan
    yield from output
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/limit_executor.py", line 40, in exec
    for batch in child_executor.exec(**kwargs):
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/project_executor.py", line 45, in exec
    batch = apply_project(batch, self.target_list, self.catalog())
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/executor_utils.py", line 46, in apply_project
    batches = [expr.evaluate(batch) for expr in project_list]
  File "/usr/local/lib/python3.10/dist-packages/evadb/executor/executor_utils.py", line 46, in <listcomp>
    batches = [expr.evaluate(batch) for expr in project_list]
  File "/usr/local/lib/python3.10/dist-packages/evadb/expression/function_expression.py", line 129, in evaluate
    outcomes = self._apply_function_expression(func, batch, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/evadb/expression/function_expression.py", line 188, in _apply_function_expression
    return func_args.apply_function_expression(func)
  File "/usr/local/lib/python3.10/dist-packages/evadb/models/storage/batch.py", line 174, in apply_function_expression
    return Batch(expr(self._frames))
  File "/usr/local/lib/python3.10/dist-packages/evadb/models/storage/batch.py", line 43, in __init__
    raise ValueError(
ValueError: Batch constructor not properly called.
Expected pandas.DataFrame, got <class 'dask.dataframe.core.DataFrame'>

[xzdandy]

Installing distributed==2023.8.1 locally, we still can not reproduce the problem.