Skip to content

Large Language Model (LLM)

This module contains functions to manipulate Large Language Model (LLM) prompts and perform LLM completions against the builtin OpenAPI integration and other LLM providers. See https://developer.dialox.ai/platform/nlp/llm/ for more information on how to work with LLM completions in general.

LLM.compact()

Compact full_transcript for later LLM.complete calls.

Runs the conversation so far through the overridable compact_full_transcript platform prompt (or an explicit prompt) and stores a compaction annotation. Later full_transcript builds keep pre-compaction system messages, place the compaction message next, and drop other pre-compaction turns.

LLM.compact()
_result = LLM.complete(@prompts.agent)

Override the platform prompt by defining a prompt with the same id, or pass a prompt explicitly:

LLM.compact(@prompts.my_compact)

LLM.complete(prompt, bindings, scope)

Perform a LLM completion request

The full result of the LLM.complete call is a map array which contains the following:

  • text - The output text that LLM produced
  • json - A JSON deserialized version of the text; the runtime detects whether JSON is available in the result and, if so, parses it. The JSON message itself can be padded with arbitrary other texts.
  • usage - The total tokens that were used for this API call
  • request_time - The nr of milliseconds this request took
  • raw - The raw OpenAPI response

Speaking while generating

On a voice call, a prompt with speak_streaming: true is spoken to the caller while the model generates it; speak_streaming: "say" speaks only the say field of a json_schema response. The result then also has:

  • spoken - whether the caller heard (part of) the response
  • spoken_text - what was sent to the caller
  • interrupted - whether the caller barged in and cut the response off

Say the result as usual: it is written to the conversation but not spoken again. Once spoken is true, do not retry the prompt or say a fallback, the caller would hear both. Elsewhere (chat, a translated conversation) nothing is streamed and spoken is false.

LLM.count_tokens(prompt_or_model, text)

Count the number of tokens given a prompt or model and an input text.

Returns nil in case of error

LLM.decode_tokens(prompt_or_model, tokens)

Decode the given list of (integer) tokens into a binary string.

Returns nil in case of error.

LLM.encode_tokens(prompt_or_model, text)

Encode the given text into a list of tokens.

Returns nil in case of error

LLM.prepare_agentloop_prompt(prompt, scope)

Prepare a prompt for the platform agent loop.

Renders the prompt with the current interpreter scope and returns a map containing the prompt configuration, rendered messages, and tool schemas.

LLM.prompt(orig \\ %Prompt{}, args)

Construct a new prompt to be used in the LLM completion functions.

This function can be used to override certain properties of other prompts, or create new prompts altogether. However, prefer to use the builtins Prompts YAML files to define prompts, as this is more efficient and less error prone.

To set for instance an extra parameter on a prompt you could do the following:

myprompt = LLM.prompt(@prompts.original, endpoint_params: %{temperature: 1.3})

LLM.render_prompt(prompt, scope)

Render a prompt into its final form, an array of chat messages.

This function is mostly used for debugging and should not be needed in normal use cases.