Add option to customise interpreter that executes commands

[Nadock]

This feature, or something similar, has been discussed previously (see #243 and here in the v3 release tracking ticket), however was never implemented.

This is a feature I personally would like to see added to Task and there seems to be at least some community support for the idea.

Opening this issue to track progress on the idea as suggested by @andreynering here.

[gschauer]

I stumbled upon Task and it seems to be the perfect solution for a variety of use cases.
In this regard, the main shortcoming is the lack of DSL-like capabilities (e.g., using goja or Yaegi as stated in #243). This could be supported by providing and interface for custom interpreters. I'd be happy to contribute, but first it's important to clarify the following design considerations (in arbitrary order), because the boundaries of this feature are not yet well defined enough.

Embed Task as library

Obviously, there is demand to utilize Task as some kind of embedded workflow engine (#121). Hence, the Interpreter interface needs to be public but rather minimalistic:

• every Interpreter has access to Taskfile, Task and Cmd to access global, task and command properties, respectively
• custom RunCommandOptions with Interpreter-specific configuration
• programmatic registration of Interpreters
• mvdan.cc/sh uses the same interface and is registered by default (and cannot be overridden)

Declare interpreter for command

Taskfile version 3 supports multiple ways to define commands. Which options shall support custom interpreters?

# Option 1: default interpreter for taskfile
interpreter: sh

  default:
    # Option 2: default interpreter for task
    interpreter: sh

    cmds:
      # Option 3: interpreter for cmd
      - cmd: console.log("Running command on Windows (amd64) and macOS")
        platforms: [windows/amd64, darwin]
        interpreter: javascript
      # Option 4: like calling another task using the "task" keyword
      # - task: greet
      - javascript: console.log({{.RECIPIENT}})
        vars: {RECIPIENT: "Cruel World"}
      # Option 5: [shebang](https://github.com/go-task/task/issues/195#issuecomment-599857399)
      - cmd: |
        #!/usr/bin/env jjs
        console.log("hello from Java's JavaScript interpreter")

  # Option 6: short task syntax
  run:
    - task: build
    - javascript: console.log({}+[])

Maybe there are other options or variations...

Includes

If interpreters can be registered in the task file (as proposed by stephenprater), how to deal with Includes? In particular:

• Are Interpreters defined in the outer taskfile inherited in included taskfiles? (foster reuse)
• If so, can they be overridden?

Data type conversion and error handling

• It might be necessary to convert strings/variables to a different data type before running an interpreter on a given expression. Shall Task support something like a ConversionService? Probably not... at least not now.
• Shall there by any way to preserve the result? Similar story as above.
• Interpreters could have a different semantics of failures e.g., a (non-)empty result string means success/failure. In other words: IsExitError might be different. Especially when considering the use case of using Task as a library.

Here, I'd rather prefer to keep it as simple as possible. Yet, it might be necessary to open up for custom error handling.

Interpreter support for other features

Task supports various features where Shell expressions are evaluated - at least to some extent e.g.,

• up-to-date checks with status
• preconditions with and without sh expansion
• dynamic variables with sh property
• interactive might not work with with certain interpreters
• defer (normal usage and task: syntax): it could be supported similarly to the options elaborated above - probably not in the first version, because of the option to call tasks
• shell specifics might not work anymore or are irrelevant, e.g.,
  • stdout/stderr/stdin redirection,
  • & at the end (& at the end of cmd doesn't run the command in background #928),
  • PosixOpts and BashOpts, IgnoreError, etc.

Final Words

The list is certainly not complete, but I'd hope that answering some of those questions would help to find the right level of abstraction. It's certainly better to take a more conservative approach. Also considering how it might interact with upcoming features such as #701.

[pd93]

This was discussed again recently on Discord and in #1292. I can see this being really useful in a whole bunch of use cases. For example:

• docker exec <container> bash -c '<cmds>'
• kubectl exec <pod> bash -c '<cmds>'
• ssh me@myserver "bash -c '<cmds>'
• bash.exe -c (on Windows)
• /usr/bin/env python
• etc...

It's worth saying that this isn't a simple feature though. As @gschauer alluded to in their comment, Task is quite heavily coupled to mvdan/sh at the moment. It would require us to refactor things a bit to make this feasible. The first step likely being to abstract the interpretter part of Task into an interface.

Unfortunately, this isn't at the top of my priority list, but I will make a note to spend some time thinking about some of the points raised above and how we can fix them. This is definitely something I'd like to see added in the future.

[andreynering]

@pd93 Interesting your idea to use it to more easily executes commands via Docker and SSH. We had ideas about adding specific support for Docker and SSH in the past, but having a more generic feature to support these and more would be interesting and more flexible (and less stuff for us to maintain). 🙂

[ccxuy]

Any updates? Maybe a simplier solution for this is to copy whole project and task files to remote ssh machine and execute, and copy it back when it's all done.

[domenkozar]

Taskfile Integration Specification for devenv.sh

Overview

This specification outlines the proposed integration of Taskfile into devenv.sh, addressing the requirements discussed in GitHub issue #1362. The focus is on enhancing Taskfile v3 with SDK support and inputs/outputs functionality.

Key Features

1. SDK Support

Requirements

• Engineering teams often prefer specific languages for tooling.
• Existing utilities and task systems are typically language-specific.
• Easing transition and reducing cognitive load is crucial.

Implementation

• Tasks should be writable using SDKs.
• To avoid performance overhead, Taskfile will run an executable at startup:

includes:
  - exec: our-tasks

• Communication Protocol:
  • Taskfile creates a Unix socket, passing it as the first argument to the executable.
  • A defined protocol (e.g., using JSON Lines) facilitates communication about supported tasks and invocation methods.

2. Inputs / Outputs

Requirements

• Tasks often require parameter passing (e.g., commit message for a git commit task).
• Output specification from tasks is necessary (e.g., committed revision).

Implementation

• Input Example:

inputs:
   git-commit:
      message: "Hola!"

• Taskfile passes inputs as JSON to the task.
• JSON outputs of all completed tasks are also passed as an input.

[pbitty]

Taskfile Integration Specification for devenv.sh

...

• Communication Protocol:

  • Taskfile creates a Unix socket, passing it as the first argument to the executable.
  • A defined protocol (e.g., using JSON Lines) facilitates communication about supported tasks and invocation methods.

• Taskfile passes inputs as JSON to the task.

@domenkozar , this is an interesting idea. If I understand correctly, does this mean that an arbitrary tool would be able to communicate to task that it has certain tasks it can run, and their require variables, etc?

I wonder if the Taskfile.yml format itself would work for it. It already supports defining tasks and their required vars. If task can read something in the format of a Taskfile.yml it will know exactly how to execute those tasks. To pass parameters to the tasks, one could use the template string {{ . | toJson }}.

For example, if a hypothetical tool called my-tool generated this content:

# Taskfile.yml generated by my-tool
version: '3'

tasks:
  some-task:
    env:
      INPUT: '{{ . | toJson }}'
    # my-tool would be able to read all of `task`'s variable via the `INPUT` env var
    cmd: my-tool some-task 

  some-other-task:
    description: ... # description of task
    required: [ ... ] # list of required input variables
    env:
      INPUT: '{{ . | toJson }}'
    # alternatively, you could provide the input vars to my-tool via stdin
    cmd: echo $INPUT | my-tool some-other-task

I'm just brainstorming an idea here where we basically use the existing format and don't have to invent anything new, and we use stdin and/or environment variables to pass input data and don't need to rely on a unix socket. Would this help solve your use-case?

I'm not sure if this would solve your requirements for output though. Could you expand a bit more on that?

[domenkozar]

@pbitty Unfortunately for https://devenv.sh we've decided to ship our own tasks implementation, although Taskfile is really close to what we want!

We're still in the experimenting phase, but we're going to ship something like this: https://gist.github.com/domenkozar/f8fde21396762c0022b26afb1b6b986f

Would be great if taskfile also supported the same protocol one day :)