-
-
Notifications
You must be signed in to change notification settings - Fork 624
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remote Taskfiles experiment #1317
Comments
This issue has been marked as an experiment proposal! 🧪 It will now enter a period of consultation during which we encourage the community to provide feedback on the proposed design. Please see the experiment workflow documentation for more information on how we release experiments. |
This experiment has been marked as a draft! ✨ This means that an initial implementation has been added to the latest release of Task! You can find information about this experiment and how to enable it in our experiments documentation. Please see the experiment workflow documentation for more information on how we release experiments. |
If one wants to use remote tasks in CI how does one bypass the prompt when running a task for the first time. Is there a flag or environment variable that can be set to “force yes” to the prompt on whether or not to include the remote task file? BTW thanks for a great experiment! |
The documentation mentions the |
@wburningham Thanks for pointing this out. The |
For Running root remote Taskfiles, consider specifying the remote taskfile via the |
Any idea why the following doesn't work? # taskfile.yml
version: '3'
include:
my-remote-namespace: https://raw.githubusercontent.com/go-task/task/main/Taskfile.yml
tasks:
test:
cmds:
- echo test succesful $ task --version
Task version: 3.30.1
$ task test
task: [test] echo test succesful
test succesful
$ export TASK_X_REMOTE_TASKFILES=1
$ task my-remote-namespace:packages
task: No tasks with description available. Try --list-all to list all tasks
task: Task "my-remote-namespace:packages" does not exist |
@oliver-helix Good idea about the root remote taskfiles. As for why your example doesn't work. I think its just a simple typo: # taskfile.yml
version: '3'
- include:
+ includes:
my-remote-namespace: https://raw.githubusercontent.com/go-task/task/main/Taskfile.yml
tasks:
test:
cmds:
- echo test succesful
Edit: We should probably give a better error message for this... |
Doh! My bad. Thanks for the help! Consider updating the experiment docs to fix this typo. |
Ah whoops. Well spotted. I've updated the docs 👍 |
edit: I caught up on #1152 and I see there has already been talk about authentication. looking forward to whatever comes! Thanks. |
@pd93 my company is ready to try out taskfiles because the remote taskfile experiment moved to draft. Being able to run on CI with a When you say "the next iteration" are you wanting (or willing to accept) a PR? The Experiments Documentation didn't specify any general timelines for accepting feedback/comments. Are you planning on waiting a minimum amount of time before releasing changes to this experiment? |
@wburningham really nice to hear that this feature is making the difference for you. Are there any other blockers or features you'd need before being able to use this fully? (besides the obvious Git/SSH integration).
Absolutely! Contributions are always welcome, though we encourage people to discuss them first to make sure multiple people aren't doing the same thing and/or wasting their time. In this particular case, I actually already have a branch which will add this functionality and I intend to post a PR soon. As a potential future user of that functionality, your feedback on this would be very welcome. If there are other extensions/improvements you'd like to help contribute to, I'm always happy to have some help.
There is no minimum/fixed time period for changes. @andreynering and I both work on this in our spare time, so planning is hard and changes tend to happen in chunks when we find some bandwidth. A new draft of the experiment will be released alongside the next version of Task which tends to happen around once a month, but this is also subject to our availability and having a set of meaningful changes to release. I'm hoping to address the remaining TODOs in this issue (not the future extensions) in the next week or so and hopefully we'll get some good feedback from that which will allow us to move forward. Edit - See PR: |
I've also just created #1345 which changes the default behaviour of Remote Taskfiles to prefer remote files over cached ones as discussed in previous PRs. I've added a couple of thoughts to the end of the PR description. I would really appreciate feedback on this from @andreynering @ryancurrah @caphrim007 and anyone else with an opinion on the matter 🙏 |
Any roadmap (ETA) for when remote taskfiles with Git ssh for private repository acecss will be added? Thanks so much for your hard work on this awesome project! |
Greetings, First and foremost, I would like to extend my heartfelt congratulations and gratitude for the exemplary work you have undertaken. I am curious to know whether you have plans to support 'remote' Taskfiles that incorporate other Taskfiles. For instance, in this link: GitLab - Devcontainer Taskfiles, if a directory is segmented into multiple Taskfiles, will remotely including the principal Taskfile also entail the inclusion of the associated Taskfiles? I look forward to your insightful response and thank you in advance for your consideration. |
Is it possible to add completion capabilities on remote tasks? |
Hi! |
I wonder if adding something like a shasum check would be a good idea here |
Hi, I wonder if it is possible to have something like a provider pattern for including remote files? With knowing the provider it would be possible to parse required credentials from CI environment variables during the include. This would make it easier for the user to handle the secrets. Example: includes:
example:
provider: gitlab
url: gitlab.my-org.com # for self-hosted variants, should default to SaaS
project: my-group/project
file: Taskfile.yaml
ref: main It's just an idea from the user's perspective. I have not yet been able to take a look at the code to say how well this can be implemented. |
I've just taken my PR for cache TTL implementation out of draft, and it's ready for review now. |
This looks indeed promising. Since we have most of our tasks quite generic (not coupled to one repo) it would be interesting to define a remote namespace as default so that |
A PR is in progess to add a Flatten include (#1704) It'll be possible to combine remote with flatten. |
Hi all - firstly thanks for the significant progress here, I'm eager to make use of this feature when it reaches maturity! I've been building a few task utilities across multiple projects and am now beginning the work in normalizing usage now that some consistent patterns have emerged. The remote taskfile feature is exactly what I need to get this done. I can be certain that a developer will have the github cli installed and intend to use that to source the users authentication token, the intent being it can then be used in the url of the remote file. However, it would seem that the environment parameters configured in the task file are not resolved before the remote includes. For example, the following works: # yaml-language-server: $schema=https://taskfile.dev/schema.json
version: '3'
output: prefixed
env:
GH_TOKEN:
sh: gh auth token
tasks:
init-deps:
desc: Initialize and update the project dependencies
cmds:
- echo {{.GH_TOKEN}} The console displays the token returned from the github cli. However the following does not work: # yaml-language-server: $schema=https://taskfile.dev/schema.json
version: '3'
output: prefixed
env:
GH_TOKEN:
sh: |
gh auth token
includes:
remote-source: https://{{.GH_TOKEN}}@raw.githubusercontent.com/ORG/REPO/BRANCH/taskfile.yml
tasks:
init-deps:
desc: Initialize and update the project dependencies
cmds:
- task: remote-source:init-deps The output of which is I can see that the work from @vmaerten may well render this path obsolete (and I would be happy not to follow it) but I'm unsure if this is an issue in task itself, or the implementation of remote includes |
Additionally, a note on this warning message
The |
Hello, we have been experimenting with go-task at my workplace (Wattpad) and are looking into incorporating it into our CI/CD workflows. Thanks for creating such a useful tool! The features are really well thought out. I have a proposal: We are looking for the ability to pull supporting files along with our remote Taskfile. We could build a separate tool for this, but it seems so close to what “remote taskfiles” provides that it could be a part of the same feature. In essence, we treat the remote Taskfile as a “remote bundle” that includes the whole directory along with the Taskfile. This allows us to include shared scripts and configuration files, alongside the Taskfile that exposes them. I did a prototype where I reference a folder on github, and
This gives us the ability in the included file to reference other bundled files using paths based on The feature can remain optional and compatible with the single-remote-taskfile case by supporting both “folder” and “file” references. For example (using pseudo-URLs, not an actual format):
How do you feel about incorporating this feature? My proposal is that I put in the time to help design it, implement it, and submit necessary PRs (with others’ input of course). Related to this, the go-getter library (mentioned above) would lend to this very well, as it is designed to download files and directories (or archives), and it handles other requests from this thread (authentication, other providers beyond HTTP). I would explore this option and satisfy both goals at the same time. If you would like to see any prototype PRs first to get a sense for how it would look and feel, I could do this as well. Would you be open to this? /cc @vmaerten - this would supersede #1652, but doesn't have to block it being merged if the syntax is compatible. Thanks for doing this work! |
I've started a draft implementation of the above in #1774. I would love to discuss the design with someone, and if we get some consensus I can continue implementing. |
@vmaerten I've been testing your work from #1652 for remote Cannot use commit hash for
|
Case | Refspec |
---|---|
o.Mirror |
+refs/*:refs/* |
o.SingleBranch && o.ReferenceName == plumbing.HEAD |
+HEAD:refs/remotes/{remote}/HEAD |
o.SingleBranch |
+refs/heads/{ref}:refs/remotes/{remote}/{ref} |
default |
+refs/heads/*:refs/remote/{remote}/* |
The 3rd case o.SingleBranch
is where we land today.
I am stuck here.
Password is logged in clear text when prompting to trust the remote source
This also happens when using --verbose
When using basic auth with https, the {user}:{password}
is logged to the terminal in clear-text ...
includes:
task: https://git:{{.TOKEN}}@gitlab.com/group/project.git//Taskfile.task.yml
outputs
The task you are attempting to run depends on the remote Taskfile at "https://git:[email protected]/group/project.git//Taskfile.task.yml".
I've tracked this to GitNode.Location() returning rawUrl
Two possible alternatives (not sure of the implications)
1. Use URL.Redacted() but you lose the path
and/or ref
func (node *GitNode) Location() string {
return node.URL.Redacted()
}
will output
The task you are attempting to run depends on the remote Taskfile at "https://git:[email protected]/group/project.git".
2. Construct the URL from its parts
func (node *GitNode) Location() string {
l := fmt.Sprintf("%s://%s/%s//%s", node.URL.Scheme, node.URL.Host, node.URL.Path, node.path)
if node.ref != "" {
l += fmt.Sprintf("?ref=%s", node.ref)
}
return l
}
will output
The task you are attempting to run depends on the remote Taskfile at "https://gitlab.com/group/project.git//Taskfile.task.yml".
Task version: v3.39.3-0.20240924174559-e619bad4a9fb (h1:iJZzplUCicBgTdzcfKEIc4KrF/cp/XLquiJX/IoeNDA=)
Operating system: Linux 6f588dd-lcelt 6.8.0-45-generic #45~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 11 15:25:05 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Experiments enabled: ENV_PRECEDENCE GENTLE_FORCE REMOTE_TASKFILES
@mgbowman , when I try I wonder why in your case it didn't come out. |
I have two separate suggestions for this feature. I would love to hear people's thoughts on them. 1. Enable downloading directories from
|
I'm a big fan of number 2! |
Number 2 will be trivial with the next release as there's a You'll have to run |
Is this what you have to run to force it to download the first time? |
@mgbowman Thanks for your feedback! I'll take a look at it :) @pbitty Yes, @mgbowman is right. In the next version, you will always be able to use the cached taskfile instead of the remote one by setting If you want to force a fresh copy, you can either run Note: |
Regarding my suggestion above:
@andreynering @pd93 @vmaerten, are you open to the idea? If so, I propose the following:
What do you think? (There are details to iron out, like how files are stored and checksummed, but I think that can be better addressed in a draft PR.) |
edit(duplicate): my bad i just saw #1317 (comment) that already report this issue. Hello, The git remote include does not seem to work with tags and commits (at least GitHub HTTPS scheme). This works: includes:
lint:
taskfile: https://github.com/nikaro/meta.git//taskfiles/lint.yml?ref=main
internal: true This (using a tag) does not work: includes:
lint:
taskfile: https://github.com/nikaro/meta.git//taskfiles/lint.yml?ref=tasks-v1
internal: true This (using a commit hash) does not work: includes:
lint:
taskfile: https://github.com/nikaro/meta.git//taskfiles/lint.yml?ref=6383d5ce51cf21dea72aab6190e6e112e7249940
internal: true Is this intended or am i doing something wrong? |
@nikaro I got it working with tags, not sure if it is possible with a commit hash. Internally the implementation is using go-git, see https://pkg.go.dev/github.com/go-git/go-git/v5/plumbing#ReferenceName Meaning you can do branches like includes:
lint:
taskfile: https://github.com/nikaro/meta.git//taskfiles/lint.yml?ref=refs/heads/main
internal: true And tags like includes:
lint:
taskfile: https://github.com/nikaro/meta.git//taskfiles/lint.yml?ref=refs/tags/task-v1
internal: true |
The security aspects of using API tokens could be improved by #951: vars:
GITHOST_TOKEN:
sh: security find-generic-password -s "Git host token" -w
includes:
compose: https://git:{{.GITHOST_TOKEN}}@example.com/shared/taskfiles.git//Tasks.yaml?ref=main On macOS you can store tokens in the Keychain using the |
This is exactly the pattern I attempted here using the GitHub cli: #1317 (comment) |
Warning
All experimental features are subject to breaking changes and/or removal at any time. We strongly recommend that you do not use these features in a production environment. They are intended for testing and feedback only.
Note
You can view the Remote Taskfiles experiment documentation on our website, including instructions on how to enable/disable it.
Context
This experiment attempts to solve the problems originally described by #770.
Currently, Task only supports running Taskfiles located on the local filesystem. This makes it difficult to reuse Taskfiles across various repositories and systems without making copies of a file. It would be useful to have a way to run a Taskfile located on a remote system or run a local Taskfile that includes one on a remote system.
Proposal
This experiment would allow users to run a Taskfiles located elsewhere, such as via HTTP or in a remote Git repository. These remote Taskfiles can either be run directly from the CLI or included by another Taskfile.
We're looking to gather feedback on the current draft, so please feel free to leave your thoughts/comments here.
Decision log / TODOs:
--download
/--offline
flags or similar)--yes
flag on prompts feat: unify prompts #1344local
->remote
->remote:local
includes work correctly feat: enable root remote taskfiles #1347task test --taskfile https://raw.githubusercontent.com/go-task/task/main/Taskfile.yml
[email protected]/foo/bar?ref=main//Taskfile.yml
feat(remote): support include git remote #1652The text was updated successfully, but these errors were encountered: