Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server : add "tokens" output #10853

Merged
merged 5 commits into from
Dec 18, 2024
Merged

server : add "tokens" output #10853

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
79a8176
server : add "tokens" output
ggerganov Dec 16, 2024
d58f8a1
server : update readme
ggerganov Dec 16, 2024
8bcfc55
server : return tokens ids only if requested
ggerganov Dec 17, 2024
5bf29af
tests : improve "tokens" type check
ggerganov Dec 18, 2024
99cb6be
server : remove "tokens" from the OAI endpoint
ggerganov Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions examples/server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,13 +444,14 @@ These words will not be included in the completion, so make sure to add them to

**Response format**

- Note: In streaming mode (`stream`), only `content` and `stop` will be returned until end of completion. Responses are sent using the [Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html) standard. Note: the browser's `EventSource` interface cannot be used due to its lack of `POST` request support.
- Note: In streaming mode (`stream`), only `content`, `tokens` and `stop` will be returned until end of completion. Responses are sent using the [Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html) standard. Note: the browser's `EventSource` interface cannot be used due to its lack of `POST` request support.

- `completion_probabilities`: An array of token probabilities for each completion. The array's length is `n_predict`. Each item in the array has the following structure:

```json
{
"content": "<the token selected by the model>",
"content": "<the token generated by the model>",
"tokens": [ generated token ids ],
"probs": [
{
"prob": float,
Expand All @@ -468,6 +469,7 @@ These words will not be included in the completion, so make sure to add them to
Notice that each `probs` is an array of length `n_probs`.

- `content`: Completion result as a string (excluding `stopping_word` if any). In case of streaming mode, will contain the next token as a string.
- `tokens`: Same as `content` but represented as raw token ids.
- `stop`: Boolean for use with `stream` to check whether the generation has stopped (Note: This is not related to stopping words array `stop` from input options)
- `generation_settings`: The provided options above excluding `prompt` but including `n_ctx`, `model`. These options may differ from the original ones in some way (e.g. bad values filtered out, strings converted to tokens, etc.).
- `model`: The path to the model loaded with `-m`
Expand Down
27 changes: 22 additions & 5 deletions examples/server/server.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,7 +468,10 @@ struct completion_token_output {

struct server_task_result_cmpl_final : server_task_result {
int index = 0;
std::string content;

std::string content;
llama_tokens tokens;

bool stream;
result_timings timings;
std::string prompt;
Expand Down Expand Up @@ -510,6 +513,7 @@ struct server_task_result_cmpl_final : server_task_result {
json res = json {
{"index", index},
{"content", stream ? "" : content}, // in stream mode, content is already in last partial chunk
{"tokens", stream ? llama_tokens {} : tokens},
{"id_slot", id_slot},
{"stop", true},
{"model", oaicompat_model},
Expand Down Expand Up @@ -541,7 +545,8 @@ struct server_task_result_cmpl_final : server_task_result {
{"index", 0},
{"message", json{
{"content", content},
{"role", "assistant"}
{"tokens", tokens},
{"role", "assistant"}
}
}}});

Expand Down Expand Up @@ -605,7 +610,9 @@ struct server_task_result_cmpl_final : server_task_result {

struct server_task_result_cmpl_partial : server_task_result {
int index = 0;
std::string content;

std::string content;
llama_tokens tokens;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can also replace these 2 fields with completion_token_output. Then inside send_partial_response, we can std::move it to res

Copy link
Collaborator

@ngxson ngxson Dec 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P/s: we cannot std::move it, because inside process_token, result is still being used after send_partial_response

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure that the "return_tokens" logic is necessary. The tokens array should be similar in JSON length to the content string, though I am not sure performance wise how much slower it is to serialize an array of integers compared to a string. Anyway, I've added the flag and added tests.

Note that with "stream": true we always return the tokens field in the partial responses (i.e. this is not affected by the "return_tokens" flag).

Copy link
Collaborator

@ngxson ngxson Dec 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I'm thinking is that this should not degrade the performance of JSON serializing/parsing. But I'm just thinking about the bandwidth, because it seems like in most cases, we're now using double the bandwidth.

For stream, I don't think it's a problem because time to serialize/send/receive/parse is minor compared to the time a token is generated.

But I think for now we can keep it this way. The non-OAI /completion is a playground anw so it's fine to expose everything. The OAI compat /v1/completions that I'm planning to do next will be more prod-ready, thus it won't have these data in the response.

Edit: I didn't notice that you implemented return_tokens, that's good then, let's keep it 👍

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I think for now we can keep it this way. The non-OAI /completion is a playground anw so it's fine to expose everything. The OAI compat /v1/completions that I'm planning to do next will be more prod-ready, thus it won't have these data in the response.

Yes, I agree we can keep /v1/completions strongly OAI-compat (i.e. not even have extra fields like tokens) and only have these in the non-OAI endpoints like /completions.

Copy link
Contributor

@isaac-mcfadyen isaac-mcfadyen Dec 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you implemented return_tokens, that's good then, let's keep it 👍

This is great to see, thank you.

I sometimes use the /completions API on a bandwidth-constrained network (Wireguard over a bad WAN connection) so having an option to disable tokens if I don't need them is perfect.


int32_t n_decoded;
int32_t n_prompt_tokens;
Expand Down Expand Up @@ -637,6 +644,7 @@ struct server_task_result_cmpl_partial : server_task_result {
json res = json {
{"index", index},
{"content", content},
{"tokens", tokens},
{"stop", false},
{"id_slot", id_slot},
{"tokens_predicted", n_decoded},
Expand Down Expand Up @@ -679,7 +687,8 @@ struct server_task_result_cmpl_partial : server_task_result {
{"choices", json::array({json{{"finish_reason", nullptr},
{"index", 0},
{"delta", json{
{"content", content}}}
{"content", content},
{"tokens", tokens}}}
}})},
{"created", t},
{"id", oaicompat_cmpl_id},
Expand All @@ -695,6 +704,7 @@ struct server_task_result_cmpl_partial : server_task_result {
{"delta",
json{
{"content", content},
{"tokens", tokens}
}},
}});
}
Expand Down Expand Up @@ -949,8 +959,11 @@ struct server_slot {

size_t last_nl_pos = 0;

std::string generated_text;
std::string generated_text;
llama_tokens generated_tokens;

llama_tokens cache_tokens;

std::vector<completion_token_output> generated_token_probs;

bool has_next_token = true;
Expand Down Expand Up @@ -985,6 +998,7 @@ struct server_slot {
n_prompt_tokens = 0;
last_nl_pos = 0;
generated_text = "";
generated_tokens = {};
has_new_line = false;
truncated = false;
stop = STOP_TYPE_NONE;
Expand Down Expand Up @@ -1736,6 +1750,7 @@ struct server_context {

// search stop word and delete it
slot.generated_text += token_str;
slot.generated_tokens.push_back(result.tok);
slot.has_next_token = true;

// check if there is incomplete UTF-8 character at the end
Expand Down Expand Up @@ -1912,6 +1927,7 @@ struct server_context {
res->id = slot.id_task;
res->index = slot.index;
res->content = tkn.text_to_send;
res->tokens = { tkn.tok };

res->n_decoded = slot.n_decoded;
res->n_prompt_tokens = slot.n_prompt_tokens;
Expand Down Expand Up @@ -1952,6 +1968,7 @@ struct server_context {

res->index = slot.index;
res->content = slot.generated_text;
res->tokens = slot.generated_tokens;
res->timings = slot.get_timings();
res->prompt = common_detokenize(ctx, slot.prompt_tokens, true);

Expand Down
Loading