-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
allow --shell-escape equivalent #152
Comments
Hi @3continents, I'm glad you find texd useful. Adding Do you require the POST /render HTTP/1.1
Content-Type: multipart/form-data; boundary=boundary
--boundary
Content-Disposition: form-data; name=input.tex; filename=input.tex
Content-Type: application/octet-stream
[content of input.tex omitted]
--boundary
Content-Disposition: form-data; name=image.tiff; filename=image.tiff
Content-Type: application/x.texd; url=1
https://somewhere.example.com/satimg/export.tiff
--boundary-- This could instruct (I'd prefer this approach, because then |
I am not really worried about the security issue -- although I understand
the concern. texd is running in a sandboxed container within a private
network of other containers and only the app server has access to it. The
app server makes http call which I control and everything sits behind a
nginx reverse proxy and a firewall where only ports 80 and 443 are open.
In any case, I can construct the request to download the images through the
API if this is a preferred option for you. I appreciate your quick response
and your willingness to add this functionality.
…On Tue, Nov 5, 2024 at 5:08 PM Dominik Menke ***@***.***> wrote:
Hi @3continents <https://github.com/3continents>,
I'm glad you find texd useful.
Adding -shell-escape support to texd, is a bit dangerous, because it
allows anyone with access to that server to run arbitrary commands. Some
danger might be mitigated when running the compile process in a Docker
container, but still, *I* wouldn't deploy this even to an internal-only
system.
Do you require the lualatex compiler to fetch the images, or would it
also helpful, if texd accepts URLs in addition to file attachments?
Something akin to:
POST /render HTTP/1.1Content-Type: multipart/form-data; boundary=boundary
--boundaryContent-Disposition: form-data; name=input.tex; filename=input.texContent-Type: application/octet-stream
[content of input.tex omitted]--boundaryContent-Disposition: form-data; name=image.tiff; filename=image.tiffContent-Type: application/x.texd; url=1
https://somewhere.example.com/satimg/export.tiff--boundary--
This could instruct texd to fetch
https://somewhere.example.com/satimg/export.tiff and store it as
./image.tiff in the project directory - before lualatex even starts.
(I'd prefer this approach, because then texd could also try to fetch
multiple images in parallel).
—
Reply to this email directly, view it on GitHub
<#152 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BERXO2G4SYOOSYUEDBZRXBDZ7DUPHAVCNFSM6AAAAABRGSCKWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJXGU4DSNZTGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I'm just worried that some users will simply ignore the red flags, enable this feature and place a server instance on the internet (and then complain about the consequences). I guess, the typical TeX user has a bit more understanding of the security implications. I might prioritize adding a |
This allows or prohibits shell command execution from within TeX files. TODO: Verify whether `latexmk -showextraoptions` mentions direct support for passing either flag. latexmk(1) does not, but has examples passing it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate the construction of the latexmk command. Fixes: #152
This allows or prohibits shell command execution from within TeX files. TODO: Verify whether `latexmk -showextraoptions` mentions direct support for passing either flag. latexmk(1) does not, but has examples passing it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate the construction of the latexmk command. Fixes: #152
This allows or prohibits shell command execution from within TeX files. TODO: Verify whether `latexmk -showextraoptions` mentions direct support for passing either flag. latexmk(1) does not, but has examples passing it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate the construction of the latexmk command. Fixes: #152
(I accidentally fat-fingered the git command while rebasing. The implementation is almost done, but not quite yet :)) |
This allows or prohibits shell command execution from within TeX files. TODO: Verify whether `latexmk -showextraoptions` mentions direct support for passing either flag. latexmk(1) does not, but has examples passing it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate the construction of the latexmk command. Fixes: #152
Hi. I hate to be a bother. I was just wondering if you were close to completing this. I build an ad-hoc / temporary solution with my own python based micro-service but it is very clunky. (I didn't get the --shell-escape in the docker containers -- no idea why as it worked locally -- and so I finally had to implement something with a two step rendering of my jinja templates so the microservice could use python and jinja to download the files. It's really ugly and I am about to implement a number of different use cases. If you are close, then I will wait. If not, I think I will test installing tex-live in the docker container for my application server so that I can simpy bypass the microservice and can call the command directly. (I have wanted to avoid this because I have many instances of application server and tex-live is so quite bloated. Please let me know. |
I'm relatively close, but didn't have the time to finish. There's one thing I wanted to verify before I merge the changes: Does The man-page does not mention it explicitly, but the source code has a list of flags it passes to the TeX compiler (with If you're willing to test, you could checkout the $ git clone https://github.com/digineo/texd -b feat/shell-escape ./texd
$ cd texd
$ docker build -t texd:experimental -f .github/Dockerfile.release . This would produce a $ docker run --rm --name ... -v ... -p ... texd:experimental --shell-escape ... |
I tried to test but I got an error at the docker build stage.
This command --
docker build -t texd:experimental -f .github/Dockerfile.release .
Gave the following error --
Status: Downloaded newer image for ghcr.io/digineo/texd:base
…---> c56f8b2a3f9b
Step 8/11 : COPY --from=builder /work/texd /bin/
COPY failed: stat work/texd: file does not exist
THanks!
Monte
On Mon, Nov 25, 2024 at 10:58 AM Dominik Menke ***@***.***> wrote:
I'm relatively close, but didn't have the time to finish. There's one
thing I wanted to verify before I merge the changes: Does latexmk
actually accept a -shell-escape flag?
The man-page does not mention it explicitly, but the source code has a
list of flags it passes to the TeX compiler (with -shell-escape being one
of them). If latexmk -shell-escape does not work as expected, I would
need to come up with something else...
If you're willing to test, you could checkout the feat/shell-escape
branch, and build a preview Docker image:
$ git clone https://github.com/digineo/texd -b feat/shell-escape ./texd
$ cd texd
$ docker build -t texd:experimental -f .github/Dockerfile.release .
This would produce a texd:experimental image, which accepts --shell-escape
(note the double-dash on the option for texd). You would run it as:
$ docker run --rm --name texd texd:experimental --shell-escape <other options...>
—
Reply to this email directly, view it on GitHub
<#152 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BERXO2FSKUUEG63GIML2KXT2CLYF5AVCNFSM6AAAAABRGSCKWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJXGQ4DAOJQGY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hm, interesting... not in a good way, though. This ought to work, as that's what the CI does regularly... 🤔 I assume you have the docker-buildx plugin installed? When running on my machine, I get roughly this output: Console outpout$ docker build -t texd:experimental -f .github/Dockerfile.release .
[+] Building 218.3s (18/18) FINISHED docker:default
=> [internal] load build definition from Dockerfile.release 0.0s
=> => transferring dockerfile: 859B 0.0s
=> resolve image config for docker-image://docker.io/docker/dockerfile:1.4 6.4s
=> [auth] docker/dockerfile:pull token for registry-1.docker.io 0.0s
=> docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531b...ef1a0dbc 1.2s
=> => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531b...ef1a0dbc 0.0s
=> => sha256:1328b32c...e7b08546 9.94MB / 9.94MB 1.1s
=> => sha256:9ba7531b...ef1a0dbc 2.00kB / 2.00kB 0.0s
=> => sha256:ad87fb03...66836f34 528B / 528B 0.0s
=> => sha256:1e8a1682...6efa47bd 2.37kB / 2.37kB 0.0s
=> => extracting sha256:1328b32c...e7b08546 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 46B 0.0s
=> [internal] load metadata for ghcr.io/digineo/texd:base 4.6s
=> [internal] load metadata for docker.io/library/golang:1.23-bookworm 6.7s
=> [auth] digineo/texd:pull token for ghcr.io 0.0s
=> [auth] library/golang:pull token for registry-1.docker.io 0.0s
=> [builder 1/4] FROM docker.io/library/golang:1.23-bookworm@sha256:3f3b9daa...b7c30ec5 37.8s
=> => resolve docker.io/library/golang:1.23-bookworm@sha256:3f3b9daa...b7c30ec5 0.0s
=> => sha256:0457bb69...1232930d 2.92kB / 2.92kB 0.0s
=> => sha256:f61a48f4...8afe855d 2.32kB / 2.32kB 0.0s
=> => sha256:b2b31b28...ef056f39 49.58MB / 49.58MB 16.7s
=> => sha256:c3cc7b6f...5161cd27 24.06MB / 24.06MB 10.9s
=> => sha256:2112e5e7...650bc13d 64.39MB / 64.39MB 9.3s
=> => sha256:3f3b9daa...b7c30ec5 9.10kB / 9.10kB 0.0s
=> => sha256:b5958539...e73e0517 92.29MB / 92.29MB 35.3s
=> => sha256:c79bddf3...9365dc80 74.04MB / 74.04MB 28.4s
=> => extracting sha256:b2b31b28...ef056f39 0.6s
=> => sha256:fe407d04...794759ea 124B / 124B 17.1s
=> => sha256:4f4fb700...38e8acc1 32B / 32B 17.3s
=> => extracting sha256:c3cc7b6f...5161cd27 0.2s
=> => extracting sha256:2112e5e7...650bc13d 0.7s
=> => extracting sha256:b5958539...e73e0517 0.7s
=> => extracting sha256:c79bddf3...9365dc80 1.3s
=> => extracting sha256:fe407d04...794759ea 0.0s
=> => extracting sha256:4f4fb700...38e8acc1 0.0s
=> [stage-1 1/3] FROM ghcr.io/digineo/texd:base@sha256:ddce2e36...f64c5280 197.9s
=> => resolve ghcr.io/digineo/texd:base@sha256:ddce2e36...f64c5280 0.0s
=> => sha256:ddce2e36...f64c5280 1.61kB / 1.61kB 0.0s
=> => sha256:6096f893...ea6c8577 677B / 677B 0.0s
=> => sha256:90a0a12e...7c5ddc98 2.21kB / 2.21kB 0.0s
=> => sha256:f2965f2d...0b03c864 1.60GB / 1.60GB 181.1s
=> => extracting sha256:f2965f2d...0b03c864 16.0s
=> [internal] load build context 0.2s
=> => transferring context: 22.39MB 0.1s
=> [builder 2/4] WORKDIR /work 0.5s
=> [builder 3/4] ADD . /work/ 0.2s
=> [builder 4/4] RUN <<-eot (set -ex...) 24.2s
=> [stage-1 2/3] COPY --from=builder /work/texd /bin/ 5.7s
=> [stage-1 3/3] WORKDIR /texd 0.0s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:d9a381b1...eec04ac6 0.0s
=> => naming to docker.io/library/texd:experimental 0.0s I would need to investigate this a little further, maybe I'll find some time when I'm back home. |
Hi. I tried on my preprod server which is a clean debian install rather
than my local machine which is a Mac and it worked. I will continue the
testing.
…On Tue, Nov 26, 2024 at 1:37 PM Dominik Menke ***@***.***> wrote:
Hm, interesting... not in a good way, though. This ought to work, as
that's what the CI does regularly... 🤔 I assume you have the docker-buildx
plugin installed?
When running on my machine, I get roughly this output:
Console outpout
$ docker build -t texd:experimental -f .github/Dockerfile.release .[+] Building 218.3s (18/18) FINISHED docker:default => [internal] load build definition from Dockerfile.release 0.0s => => transferring dockerfile: 859B 0.0s => resolve image config for docker-image://docker.io/docker/dockerfile:1.4 6.4s => [auth] docker/dockerfile:pull token for registry-1.docker.io 0.0s => ***@***.***:9ba7531b...ef1a0dbc 1.2s => => resolve ***@***.***:9ba7531b...ef1a0dbc 0.0s => => sha256:1328b32c...e7b08546 9.94MB / 9.94MB 1.1s => => sha256:9ba7531b...ef1a0dbc 2.00kB / 2.00kB 0.0s => => sha256:ad87fb03...66836f34 528B / 528B 0.0s => => sha256:1e8a1682...6efa47bd 2.37kB / 2.37kB 0.0s => => extracting sha256:1328b32c...e7b08546 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 46B 0.0s => [internal] load metadata for ghcr.io/digineo/texd:base 4.6s => [internal] load metadata for docker.io/library/golang:1.23-bookworm 6.7s => [auth] digineo/texd:pull token for ghcr.io 0.0s => [auth] library/golang:pull token for registry-1.docker.io 0.0s => [builder 1/4] FROM ***@***.***:3f3b9daa...b7c30ec5 37.8s => => resolve ***@***.***:3f3b9daa...b7c30ec5 0.0s => => sha256:0457bb69...1232930d 2.92kB / 2.92kB 0.0s => => sha256:f61a48f4...8afe855d 2.32kB / 2.32kB 0.0s => => sha256:b2b31b28...ef056f39 49.58MB / 49.58MB 16.7s => => sha256:c3cc7b6f...5161cd27 24.06MB / 24.06MB 10.9s => => sha256:2112e5e7...650bc13d 64.39MB / 64.39MB 9.3s => => sha256:3f3b9daa...b7c30ec5 9.10kB / 9.10kB 0.0s => => sha256:b5958539...e73e0517 92.29MB / 92.29MB 35.3s => => sha256:c79bddf3...9365dc80 74.04MB / 74.04MB 28.4s => => extracting sha256:b2b31b28...ef056f39 0.6s => => sha256:fe407d04...794759ea 124B / 124B 17.1s => => sha256:4f4fb700...38e8acc1 32B / 32B 17.3s => => extracting sha256:c3cc7b6f...5161cd27 0.2s => => extracting sha256:2112e5e7...650bc13d 0.7s => => extracting sha256:b5958539...e73e0517 0.7s => => extracting sha256:c79bddf3...9365dc80 1.3s => => extracting sha256:fe407d04...794759ea 0.0s => => extracting sha256:4f4fb700...38e8acc1 0.0s => [stage-1 1/3] FROM ***@***.***:ddce2e36...f64c5280 197.9s => => resolve ***@***.***:ddce2e36...f64c5280 0.0s => => sha256:ddce2e36...f64c5280 1.61kB / 1.61kB 0.0s => => sha256:6096f893...ea6c8577 677B / 677B 0.0s => => sha256:90a0a12e...7c5ddc98 2.21kB / 2.21kB 0.0s => => sha256:f2965f2d...0b03c864 1.60GB / 1.60GB 181.1s => => extracting sha256:f2965f2d...0b03c864 16.0s => [internal] load build context 0.2s => => transferring context: 22.39MB 0.1s => [builder 2/4] WORKDIR /work 0.5s => [builder 3/4] ADD . /work/ 0.2s => [builder 4/4] RUN <<-eot (set -ex...) 24.2s => [stage-1 2/3] COPY --from=builder /work/texd /bin/ 5.7s => [stage-1 3/3] WORKDIR /texd 0.0s => exporting to image 0.1s => => exporting layers 0.1s => => writing image sha256:d9a381b1...eec04ac6 0.0s => => naming to docker.io/library/texd:experimental 0.0s
I would need to investigate this a little further, maybe I'll find some
time when I'm back home.
—
Reply to this email directly, view it on GitHub
<#152 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BERXO2HQC4LMWT2AFMAYJXD2CRTQBAVCNFSM6AAAAABRGSCKWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBQGY4DMMJSGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
I have tested and it is not working.
I have a the following in my latex document.
\adjustbox{margin=-0.75cm 0cm 0cm 0cm}{ \begin{luacode*}
local url = "
https://scipioholding.com/media/images/scipio-logo.original.png"
local output = "8f340a0a-44ff-4fcc-a1c1-8588755e488e.png"
local command = string.format("curl -o %s %s", output, url)
os.execute(command)
\end{luacode*}
\adjustbox{max size={ 5cm }{ 20cm }}{\includegraphics{
8f340a0a-44ff-4fcc-a1c1-8588755e488e.png }} }
On when compiling with lualatex interpreter in texd, I get --
Compilation failed
Logs
Package luatex.def Error: File `8f340a0a-44ff-4fcc-a1c1-8588755e488e.png' not
This is exactly what happens when shell escape is not enabled as an option.
The file is not downloaded and therefore it isn't found.
Thanks a ton and sorry to share the bad news.
Monte
…On Mon, Nov 25, 2024 at 10:58 AM Dominik Menke ***@***.***> wrote:
I'm relatively close, but didn't have the time to finish. There's one
thing I wanted to verify before I merge the changes: Does latexmk
actually accept a -shell-escape flag?
The man-page does not mention it explicitly, but the source code has a
list of flags it passes to the TeX compiler (with -shell-escape being one
of them). If latexmk -shell-escape does not work as expected, I would
need to come up with something else...
If you're willing to test, you could checkout the feat/shell-escape
branch, and build a preview Docker image:
$ git clone https://github.com/digineo/texd -b feat/shell-escape ./texd
$ cd texd
$ docker build -t texd:experimental -f .github/Dockerfile.release .
This would produce a texd:experimental image, which accepts --shell-escape
(note the double-dash on the option for texd). You would run it as:
$ docker run --rm --name texd texd:experimental --shell-escape <other options...>
—
Reply to this email directly, view it on GitHub
<#152 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BERXO2FSKUUEG63GIML2KXT2CLYF5AVCNFSM6AAAAABRGSCKWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJXGQ4DAOJQGY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Can you share the |
docker run --rm -t -p 0.0.0.0:2201:2201 --name texd texd:experimental
--shell-escape
…On Tue, Nov 26, 2024 at 2:35 PM Dominik Menke ***@***.***> wrote:
This is exactly what happens when shell escape is not enabled as an option.
Can you share the docker run command you've used? (If it contains some
settings you don't want to share publicly, feel free to drop me an email at 'github+texd-152'
.. '@' .. 'dmke.org').
—
Reply to this email directly, view it on GitHub
<#152 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/BERXO2E27ZSHFTAU7GZQGTL2CR2KJAVCNFSM6AAAAABRGSCKWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBQHAZDANZWHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Thanks, I'll try to reproduce and fix this later. |
This allows or prohibits shell command execution from within TeX files. TODO: Verify whether `latexmk -showextraoptions` mentions direct support for passing either flag. latexmk(1) does not, but has examples passing it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate the construction of the latexmk command. Fixes: #152
Using your example curl image\documentclass{article}
\usepackage{luacode}
\usepackage{graphicx}
\begin{document}
\begin{luacode*}
local url = "https://scipioholding.com/media/images/scipio-logo.original.png"
local output = "8f340a0a-44ff-4fcc-a1c1-8588755e488e.png"
local command = string.format("curl -o %s %s", output, url)
os.execute(command)
\end{luacode*}
\includegraphics{8f340a0a-44ff-4fcc-a1c1-8588755e488e.png}
\end{document} I indeed get an error. However the error is not that command execution failed, but that The problem here is, that You can inspect the list of installed packages using input.tex\documentclass[10pt]{article}
\usepackage[landscape, margin=1mm]{geometry}
\usepackage{luacode}
\usepackage{verbatim}
\begin{document}
\begin{luacode*}
os.execute("dpkg -l > packages.txt")
\end{luacode*}
\verbatiminput{packages.txt}
\end{document} |
This allows or prohibits shell command execution from within TeX files. The use case here is to allow users to download images from within the build process, e.g. compiling something like this with `lualatex`: \documentclass{article} \usepackage{luacode} \usepackage{graphicx} \begin{document} \begin{luacode*} os.execute("curl -o apod.jpg https://apod.nasa.gov/apod/") \end{luacode*} \includegraphics{apod.jpeg} \end{document} To this end, I've included `curl` into the base image. Fixes: #152
This allows or prohibits shell command execution from within TeX files. The use case here is to allow users to download images from within the build process, e.g. compiling something like this with `lualatex`: \documentclass{article} \usepackage{luacode} \usepackage{graphicx} \begin{document} \begin{luacode*} os.execute("curl -o apod.jpg https://apod.nasa.gov/apod/") \end{luacode*} \includegraphics{apod.jpeg} \end{document} To this end, I've included `curl` into the base image. Fixes: #152
This allows or prohibits shell command execution from within TeX files. The use case here is to allow users to download images from within the build process, e.g. compiling something like this with `lualatex`: \documentclass{article} \usepackage{luacode} \usepackage{graphicx} \begin{document} \begin{luacode*} os.execute("curl -o apod.jpg https://apod.nasa.gov/apod/") \end{luacode*} \includegraphics{apod.jpeg} \end{document} To this end, I've included `curl` into the base image. Fixes: #152
I'm preparing a new base image (with Building a new base image takes a bit on Github, so I'm postponing a new feature release til tomorrow. |
Hi. Thanks for this product. It is truly great. It has allowed me to deploy latex generation to our internal web applications without having to download and manage a tex distribution locally on the app-server with each update.
I have, unfortunately, ran into a specific. In my latex generation, I need to download images to include them in the documents. This is helpful locally but truly required when using your docker container with is separate from the rest of the containers. To achieve this locally, I have to run lualatex using the --shell-escape directive or I get image not found errors.
I am getting this error when posting to your web interface. Would it be possible to add the --shell-escape option in the URL parameters? That would solve everything for me.
The text was updated successfully, but these errors were encountered: