Multi-Action Space Reinforcement Learning

[Fornerio]

Hello,

I have two datasets of MDP trajectories sampled from the same environment. These datasets share the same observation, but they differ in action space.
I would like to create an offline RL model able to choose which action to perform from those two action space, but I am still doubtful on the implementation. Of course, it wouldn't make sense to create a new action space = action_space1 + action_space2.

I think that the first thing to do should be creating a new env like this

class MultiActionEnv(AbstractEnv):
    
    def __init__(self, config: dict = None) -> None:
        ...

    @classmethod
    def default_config(cls) -> dict:
        config = super().default_config()
        config.update({
            "action_first": {
                "type": action_space1
            },
            "action_second": {
                "type": action_space2
            }
        })
        return config

but I am still unsure about the changes that have to be done consequently, both in multi_action_env.py and behaviour.py/controller.py

Thank you so much in advance for your feedbacks,

Regards

[eleurent]

I don't think the environment needs to be MultiAgent, the best is to implement your own ActionType and use a composite action space, see https://gymnasium.farama.org/api/spaces/composite/

For example,

observation_space = Dict({
  "selected_action_space": Discrete(2), 
  "action_space_1": Box(-1, 1, shape=(2,)),
  "action_space_2": Discrete(10),
}, seed=42)