merge cog-trt-llm into this repo #38
Conversation
yorickvP
commented
May 14, 2024
yorickvP
commented
- subtree merge cog-trt-llm
- ignore cog-trt-llm subdir in runners
- use cog-trt-llm as root dir in builder
…ATH to fix libnvinfer.so.9 import
Update README.md
Co-authored-by: Nathan Raw <[email protected]>
Make instructions more noob friendly & fix papercuts
Read TRTLLM_DIR from env with fallback
| # copy cog-trt-llm source into /src | ||
| cognix.postCopyCommands = '' | ||
| cp ${config.deps.cog-trt-llm}/{*.py,cog-trt-llm-config.yaml} $out/src/ | ||
| ''; | ||
| cognix.rootPath = lib.mkForce "${./cog-trt-llm}"; |
There was a problem hiding this comment.
doesn't this result in a pretty different image?
There was a problem hiding this comment.
/src will be different, just containing the data from cog-trt-llm instead of a weird merge. But that should be it.
There was a problem hiding this comment.
I feel like I'd prefer to keep the weird merge as it is, I think
There was a problem hiding this comment.
@technillogue, could you tldr the costs/benefits as you see them? I don't think I understand your preference, but I'd like to :)
There was a problem hiding this comment.
I was just thinking of keeping the output images the way things are now, but I don't actually care that much
|
This seems good to me. Presumably you've tested the builds? |
| [submodule "tensorrtllm_backend"] | ||
| path = tensorrtllm_backend | ||
| url = https://github.com/triton-inference-server/tensorrtllm_backend.git |
There was a problem hiding this comment.
we should probably drop this
There was a problem hiding this comment.
haven't dropped it because that would break the 'legacy' dockerfile cog-trt-llm build
| *.bin | ||
| *.safetensors | ||
| *.cog | ||
| *.hypothesis | ||
| *.pytest_cache | ||
| __pycache__/ | ||
| *.pyc | ||
| models/* |
There was a problem hiding this comment.
this doesn't do anything
