Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow --shell-escape equivalent #152

Open
3continents opened this issue Nov 5, 2024 · 15 comments · May be fixed by #153
Open

allow --shell-escape equivalent #152

3continents opened this issue Nov 5, 2024 · 15 comments · May be fixed by #153

Comments

@3continents
Copy link

Hi. Thanks for this product. It is truly great. It has allowed me to deploy latex generation to our internal web applications without having to download and manage a tex distribution locally on the app-server with each update.

I have, unfortunately, ran into a specific. In my latex generation, I need to download images to include them in the documents. This is helpful locally but truly required when using your docker container with is separate from the rest of the containers. To achieve this locally, I have to run lualatex using the --shell-escape directive or I get image not found errors.

I am getting this error when posting to your web interface. Would it be possible to add the --shell-escape option in the URL parameters? That would solve everything for me.

@dmke
Copy link
Member

dmke commented Nov 5, 2024

Hi @3continents,

I'm glad you find texd useful.

Adding -shell-escape support to texd, is a bit dangerous, because it allows anyone with access to that server to run arbitrary commands. Some danger might be mitigated when running the compile process in a Docker container, but still, I wouldn't deploy this even to an internal-only system.

Do you require the lualatex compiler to fetch the images, or would it also helpful, if texd accepts URLs in addition to file attachments? Something akin to:

POST /render HTTP/1.1
Content-Type: multipart/form-data; boundary=boundary

--boundary
Content-Disposition: form-data; name=input.tex; filename=input.tex
Content-Type: application/octet-stream

[content of input.tex omitted]
--boundary
Content-Disposition: form-data; name=image.tiff; filename=image.tiff
Content-Type: application/x.texd; url=1

https://somewhere.example.com/satimg/export.tiff
--boundary--

This could instruct texd to fetch https://somewhere.example.com/satimg/export.tiff and store it as ./image.tiff in the project directory - before lualatex even starts.

(I'd prefer this approach, because then texd could also try to fetch multiple images in parallel).

@3continents
Copy link
Author

3continents commented Nov 5, 2024 via email

@dmke
Copy link
Member

dmke commented Nov 5, 2024

I'm just worried that some users will simply ignore the red flags, enable this feature and place a server instance on the internet (and then complain about the consequences). I guess, the typical TeX user has a bit more understanding of the security implications.

I might prioritize adding a --shell-escape CLI argument to texd over a custom pipeline for fetching and storing external files (this is waaaay easier to implement).

@dmke dmke linked a pull request Nov 5, 2024 that will close this issue
dmke added a commit that referenced this issue Nov 5, 2024
This allows or prohibits shell command execution from within
TeX files.

TODO: Verify whether `latexmk -showextraoptions` mentions direct support
for passing either flag. latexmk(1) does not, but has examples passing
it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate
the construction of the latexmk command.

Fixes: #152
@dmke dmke closed this as completed in d9e5b50 Nov 8, 2024
dmke added a commit that referenced this issue Nov 8, 2024
This allows or prohibits shell command execution from within
TeX files.

TODO: Verify whether `latexmk -showextraoptions` mentions direct support
for passing either flag. latexmk(1) does not, but has examples passing
it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate
the construction of the latexmk command.

Fixes: #152
dmke added a commit that referenced this issue Nov 8, 2024
This allows or prohibits shell command execution from within
TeX files.

TODO: Verify whether `latexmk -showextraoptions` mentions direct support
for passing either flag. latexmk(1) does not, but has examples passing
it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate
the construction of the latexmk command.

Fixes: #152
@dmke dmke reopened this Nov 8, 2024
@dmke
Copy link
Member

dmke commented Nov 8, 2024

(I accidentally fat-fingered the git command while rebasing. The implementation is almost done, but not quite yet :))

dmke added a commit that referenced this issue Nov 22, 2024
This allows or prohibits shell command execution from within
TeX files.

TODO: Verify whether `latexmk -showextraoptions` mentions direct support
for passing either flag. latexmk(1) does not, but has examples passing
it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate
the construction of the latexmk command.

Fixes: #152
@3continents
Copy link
Author

Hi. I hate to be a bother. I was just wondering if you were close to completing this. I build an ad-hoc / temporary solution with my own python based micro-service but it is very clunky. (I didn't get the --shell-escape in the docker containers -- no idea why as it worked locally -- and so I finally had to implement something with a two step rendering of my jinja templates so the microservice could use python and jinja to download the files. It's really ugly and I am about to implement a number of different use cases.

If you are close, then I will wait. If not, I think I will test installing tex-live in the docker container for my application server so that I can simpy bypass the microservice and can call the command directly. (I have wanted to avoid this because I have many instances of application server and tex-live is so quite bloated.

Please let me know.

@dmke
Copy link
Member

dmke commented Nov 25, 2024

I'm relatively close, but didn't have the time to finish. There's one thing I wanted to verify before I merge the changes: Does latexmk actually accept a -shell-escape flag?

The man-page does not mention it explicitly, but the source code has a list of flags it passes to the TeX compiler (with -shell-escape being one of them). If latexmk -shell-escape does not work as expected, I would need to come up with something else...

If you're willing to test, you could checkout the feat/shell-escape branch, and build a preview Docker image:

$ git clone https://github.com/digineo/texd -b feat/shell-escape ./texd
$ cd texd
$ docker build -t texd:experimental -f .github/Dockerfile.release .

This would produce a texd:experimental image, which accepts --shell-escape (note the double-dash on the option for texd). You would run it as:

$ docker run --rm --name ... -v ... -p ... texd:experimental --shell-escape ...

@3continents
Copy link
Author

3continents commented Nov 26, 2024 via email

@dmke
Copy link
Member

dmke commented Nov 26, 2024

Hm, interesting... not in a good way, though. This ought to work, as that's what the CI does regularly... 🤔 I assume you have the docker-buildx plugin installed?

When running on my machine, I get roughly this output:

Console outpout
$ docker build -t texd:experimental -f .github/Dockerfile.release .
[+] Building 218.3s (18/18) FINISHED                                             docker:default
 => [internal] load build definition from Dockerfile.release                               0.0s
 => => transferring dockerfile: 859B                                                       0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1.4                6.4s
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                           0.0s
 => docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531b...ef1a0dbc              1.2s
 => => resolve docker.io/docker/dockerfile:1.4@sha256:9ba7531b...ef1a0dbc                  0.0s
 => => sha256:1328b32c...e7b08546 9.94MB / 9.94MB                                          1.1s
 => => sha256:9ba7531b...ef1a0dbc 2.00kB / 2.00kB                                          0.0s
 => => sha256:ad87fb03...66836f34 528B / 528B                                              0.0s
 => => sha256:1e8a1682...6efa47bd 2.37kB / 2.37kB                                          0.0s
 => => extracting sha256:1328b32c...e7b08546                                               0.0s
 => [internal] load .dockerignore                                                          0.0s
 => => transferring context: 46B                                                           0.0s
 => [internal] load metadata for ghcr.io/digineo/texd:base                                 4.6s
 => [internal] load metadata for docker.io/library/golang:1.23-bookworm                    6.7s
 => [auth] digineo/texd:pull token for ghcr.io                                             0.0s
 => [auth] library/golang:pull token for registry-1.docker.io                              0.0s
 => [builder 1/4] FROM docker.io/library/golang:1.23-bookworm@sha256:3f3b9daa...b7c30ec5  37.8s
 => => resolve docker.io/library/golang:1.23-bookworm@sha256:3f3b9daa...b7c30ec5           0.0s
 => => sha256:0457bb69...1232930d 2.92kB / 2.92kB                                          0.0s
 => => sha256:f61a48f4...8afe855d 2.32kB / 2.32kB                                          0.0s
 => => sha256:b2b31b28...ef056f39 49.58MB / 49.58MB                                       16.7s
 => => sha256:c3cc7b6f...5161cd27 24.06MB / 24.06MB                                       10.9s
 => => sha256:2112e5e7...650bc13d 64.39MB / 64.39MB                                        9.3s
 => => sha256:3f3b9daa...b7c30ec5 9.10kB / 9.10kB                                          0.0s
 => => sha256:b5958539...e73e0517 92.29MB / 92.29MB                                       35.3s
 => => sha256:c79bddf3...9365dc80 74.04MB / 74.04MB                                       28.4s
 => => extracting sha256:b2b31b28...ef056f39                                               0.6s
 => => sha256:fe407d04...794759ea 124B / 124B                                             17.1s
 => => sha256:4f4fb700...38e8acc1 32B / 32B                                               17.3s
 => => extracting sha256:c3cc7b6f...5161cd27                                               0.2s
 => => extracting sha256:2112e5e7...650bc13d                                               0.7s
 => => extracting sha256:b5958539...e73e0517                                               0.7s
 => => extracting sha256:c79bddf3...9365dc80                                               1.3s
 => => extracting sha256:fe407d04...794759ea                                               0.0s
 => => extracting sha256:4f4fb700...38e8acc1                                               0.0s
 => [stage-1 1/3] FROM ghcr.io/digineo/texd:base@sha256:ddce2e36...f64c5280              197.9s
 => => resolve ghcr.io/digineo/texd:base@sha256:ddce2e36...f64c5280                        0.0s
 => => sha256:ddce2e36...f64c5280 1.61kB / 1.61kB                                          0.0s
 => => sha256:6096f893...ea6c8577 677B / 677B                                              0.0s
 => => sha256:90a0a12e...7c5ddc98 2.21kB / 2.21kB                                          0.0s
 => => sha256:f2965f2d...0b03c864 1.60GB / 1.60GB                                        181.1s
 => => extracting sha256:f2965f2d...0b03c864                                              16.0s
 => [internal] load build context                                                          0.2s
 => => transferring context: 22.39MB                                                       0.1s
 => [builder 2/4] WORKDIR /work                                                            0.5s
 => [builder 3/4] ADD . /work/                                                             0.2s
 => [builder 4/4] RUN <<-eot (set -ex...)                                                 24.2s 
 => [stage-1 2/3] COPY --from=builder /work/texd /bin/                                     5.7s 
 => [stage-1 3/3] WORKDIR /texd                                                            0.0s 
 => exporting to image                                                                     0.1s 
 => => exporting layers                                                                    0.1s 
 => => writing image sha256:d9a381b1...eec04ac6                                            0.0s 
 => => naming to docker.io/library/texd:experimental                                       0.0s

I would need to investigate this a little further, maybe I'll find some time when I'm back home.

@3continents
Copy link
Author

3continents commented Nov 26, 2024 via email

@3continents
Copy link
Author

3continents commented Nov 26, 2024 via email

@dmke
Copy link
Member

dmke commented Nov 26, 2024

This is exactly what happens when shell escape is not enabled as an option.

Can you share the docker run command you've used? (If it contains some settings you don't want to share publicly, feel free to drop me an email at 'github+texd-152' .. '@' .. 'dmke.org').

@3continents
Copy link
Author

3continents commented Nov 26, 2024 via email

@dmke
Copy link
Member

dmke commented Nov 26, 2024

Thanks, I'll try to reproduce and fix this later.

dmke added a commit that referenced this issue Nov 26, 2024
This allows or prohibits shell command execution from within
TeX files.

TODO: Verify whether `latexmk -showextraoptions` mentions direct support
for passing either flag. latexmk(1) does not, but has examples passing
it via the `-pdf{lua,xe,}latex="COMMAND"` options. This would complicate
the construction of the latexmk command.

Fixes: #152
@dmke
Copy link
Member

dmke commented Nov 26, 2024

Using your example

curl image
\documentclass{article}
\usepackage{luacode}
\usepackage{graphicx}
\begin{document}

\begin{luacode*}
  local url = "https://scipioholding.com/media/images/scipio-logo.original.png"
  local output = "8f340a0a-44ff-4fcc-a1c1-8588755e488e.png"
  local command = string.format("curl -o %s %s", output, url)
  os.execute(command)
\end{luacode*}
\includegraphics{8f340a0a-44ff-4fcc-a1c1-8588755e488e.png}

\end{document}

I indeed get an error.

However the error is not that command execution failed, but that \includegraphics could not find the file:

image

The problem here is, that curl is not installed :)

You can inspect the list of installed packages using dpkg -l (and also verify that -shell-escape works as expected):

input.tex
\documentclass[10pt]{article}
\usepackage[landscape, margin=1mm]{geometry}
\usepackage{luacode}
\usepackage{verbatim}
\begin{document}

\begin{luacode*}
  os.execute("dpkg -l > packages.txt")
\end{luacode*}
\verbatiminput{packages.txt}

\end{document}

image

dmke added a commit that referenced this issue Nov 26, 2024
This allows or prohibits shell command execution from within TeX files.

The use case here is to allow users to download images from within the
build process, e.g. compiling something like this with `lualatex`:

    \documentclass{article}
    \usepackage{luacode}
    \usepackage{graphicx}

    \begin{document}
    \begin{luacode*}
      os.execute("curl -o apod.jpg https://apod.nasa.gov/apod/")
    \end{luacode*}
    \includegraphics{apod.jpeg}
    \end{document}

To this end, I've included `curl` into the base image.

Fixes: #152
dmke added a commit that referenced this issue Nov 26, 2024
This allows or prohibits shell command execution from within TeX files.

The use case here is to allow users to download images from within the
build process, e.g. compiling something like this with `lualatex`:

    \documentclass{article}
    \usepackage{luacode}
    \usepackage{graphicx}

    \begin{document}
    \begin{luacode*}
      os.execute("curl -o apod.jpg https://apod.nasa.gov/apod/")
    \end{luacode*}
    \includegraphics{apod.jpeg}
    \end{document}

To this end, I've included `curl` into the base image.

Fixes: #152
dmke added a commit that referenced this issue Nov 26, 2024
This allows or prohibits shell command execution from within TeX files.

The use case here is to allow users to download images from within the
build process, e.g. compiling something like this with `lualatex`:

    \documentclass{article}
    \usepackage{luacode}
    \usepackage{graphicx}

    \begin{document}
    \begin{luacode*}
      os.execute("curl -o apod.jpg https://apod.nasa.gov/apod/")
    \end{luacode*}
    \includegraphics{apod.jpeg}
    \end{document}

To this end, I've included `curl` into the base image.

Fixes: #152
@dmke
Copy link
Member

dmke commented Nov 26, 2024

I'm preparing a new base image (with curl), and have put finishing touches on the feat/shell-escape branch.

Building a new base image takes a bit on Github, so I'm postponing a new feature release til tomorrow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants