llm.bend relies on unsafe/foreign

raw source on the hub · import bend-kit-llm@0.3.0.0/llm.bend as Llm

LLM client for the Anthropic Messages and OpenAI Chat Completions APIs, with SSE streaming. Source: https://github.com/paymog/bend-kit/tree/main/llm

8 imports

import Base
import ./sse.bend as Sse
import bend-kit-hairpin@0.3.0.0/hairpin.bend as Hairpin
import bend-kit-hairpin@0.3.0.0/retry.bend as Retry
import bend-kit-http@0.32.0.0/http.bend as Http
import 0x584fc27920487ceab242392391418d7f/json.bend as Json
import 0x49814d83de8f70993a43e1002be29ecd/bytes.bend as Bytes
import 0xcfc8be7b076f41f95c8e118383892d55/encoding.bend as Enc

Types

type Api source · line 13 · raw

Data

Anthropic: POST {base}/v1/messages. OpenAI: POST {base}/chat/completions, which many other providers serve too.

Anthropic
```
Api
```
OpenAI
```
Api
```

type Role source · line 17 · raw

Data

User
```
Role
```
Assistant
```
Role
```

type Msg source · line 21 · raw

Data

Msg
```
@role:Role -> @text:String -> Msg
```

type Req source · line 25 · raw

Data

system: "" for none. max: the most tokens to write.

Req

@model:String -> @system:String -> @msgs:List<&2, Msg> -> @max:U32 -> Req

type Reply source · line 29 · raw

Data

stop: the API's reason, as "end_turn" or "stop". input and output count tokens.

Reply

@id:String -> @model:String -> @text:String -> @stop:String -> @input:U32 -> @output:U32 -> Reply

type Err source · line 34 · raw

Data

ErrNet: the request did not complete, or Hairpin's breaker admitted no attempt. ErrApi: the API refused, with its status and error text; an error event in a stream has status 200. ErrReply: a reply that is not the API's JSON, with its text.

ErrNet

@e:0x5f997a9351e7133e43ba9611e734f8a6/hairpin.Err -> Err

ErrApi
```
@status:U32 -> @body:String -> Err
```
ErrReply
```
@body:String -> Err
```

type Client source · line 40 · raw

Type

headers go on every request. ms: the step timeout. retries: extra tries after a retryable failure.

Client

@api:Api -> @base:String -> @headers:Map<&2, List<&2, String>> -> @ms:U32 -> @retries:U32 -> @http:0x5f997a9351e7133e43ba9611e734f8a6/hairpin.Client -> Client

type Leaf source · line 103 · raw

Data

A JSON leaf: its path, as "content.0.text", and its text.

Leaf
```
@p:String -> @v:String -> Leaf
```

type Ev source · line 217 · raw

Data

A text piece, the end, an error, or an event with no text.

EText
```
@s:String -> Ev
```
EEnd
```
Ev
```
EFail
```
@e:Err -> Ev
```
ESkip
```
Ev
```

type Stream source · line 412 · raw

Type

Stream

@api:Api -> @h:0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream -> @rd:0xce2a0a9d4864ec82024ab27469071032/sse.Reader -> Stream

type Rn source · line 541 · raw

Type

The reader after the next event, or the stream's result.

RnNext

@h:0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream -> @r:Pair(0xce2a0a9d4864ec82024ab27469071032/sse.Reader, 0xce2a0a9d4864ec82024ab27469071032/sse.Next) -> Rn

RnOut

@h:0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream -> @rd:0xce2a0a9d4864ec82024ab27469071032/sse.Reader -> @r:Result<&1, &1, Err, Maybe<&1, 0xce2a0a9d4864ec82024ab27469071032/sse.Message>> -> Rn

Definitions

def str source · line 45 · raw

@s:String -> 0x584fc27920487ceab242392391418d7f/json.Val

def kv source · line 48 · raw

@k:String -> @v:0x584fc27920487ceab242392391418d7f/json.Val -> Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x584fc27920487ceab242392391418d7f/json.Val)

def num source · line 51 · raw

@+n:U32 -> 0x584fc27920487ceab242392391418d7f/json.Val

def obj source · line 54 · raw

@kvs:List<&1, Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x584fc27920487ceab242392391418d7f/json.Val)> -> 0x584fc27920487ceab242392391418d7f/json.Val

def when source · line 57 · raw

@on:Bool -> @x:Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x584fc27920487ceab242392391418d7f/json.Val) -> @xs:List<&1, Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x584fc27920487ceab242392391418d7f/json.Val)> -> List<&1, Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x584fc27920487ceab242392391418d7f/json.Val)>

def role source · line 64 · raw

@r:Role -> String

def msg source · line 71 · raw

@+r:String -> @t:String -> 0x584fc27920487ceab242392391418d7f/json.Val

def msgs source · line 74 · raw

@xs:List<&2, Msg> -> List<&1, 0x584fc27920487ceab242392391418d7f/json.Val>

def sys source · line 81 · raw

@on:Bool -> @s:String -> @xs:List<&1, 0x584fc27920487ceab242392391418d7f/json.Val> -> List<&1, 0x584fc27920487ceab242392391418d7f/json.Val>

def body source · line 89 · raw

@api:Api -> @req:Req -> @stream:Bool -> 0x584fc27920487ceab242392391418d7f/json.Val

The request body for api. stream asks for SSE.

def path source · line 106 · raw

@+p:String -> @+k:String -> String

def push.arr source · line 109 · raw

@xs:List<&1, 0x584fc27920487ceab242392391418d7f/json.Val> -> @+p:String -> @+i:U32 -> @t:List<&1, Pair(String, 0x584fc27920487ceab242392391418d7f/json.Val)> -> List<&1, Pair(String, 0x584fc27920487ceab242392391418d7f/json.Val)>

def push.obj source · line 116 · raw

@kvs:List<&1, Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x584fc27920487ceab242392391418d7f/json.Val)> -> @+p:String -> @t:List<&1, Pair(String, 0x584fc27920487ceab242392391418d7f/json.Val)> -> List<&1, Pair(String, 0x584fc27920487ceab242392391418d7f/json.Val)>

def flat source · line 124 · raw

@f:Nat -> @stack:List<&1, Pair(String, 0x584fc27920487ceab242392391418d7f/json.Val)> -> @acc:List<&2, Leaf> -> List<&2, Leaf>

The leaves under the stack's values, last first. f: at least the values left.

def doc.of source · line 147 · raw

@+n:U32 -> @m:Maybe<&1, 0x584fc27920487ceab242392391418d7f/json.Val> -> Maybe<&2, List<&2, Leaf>>

def doc source · line 156 · raw

@b:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> Maybe<&2, List<&2, Leaf>>

Every non-null leaf of a JSON text in document order, keyed by its path, as ("content.0.text", "Hi"). Strings are decoded; numbers keep their text. None when b is not JSON. Each value takes a byte, so n + 1 steps suffice.

def look source · line 161 · raw

@d:List<&2, Leaf> -> @+k:String -> String

The first value at path k, or "".

def count source · line 168 · raw

@s:String -> U32

def dots source · line 171 · raw

@s:String -> @+n:U32 -> U32

def block source · line 179 · raw

@+p:String -> Bool

Is p the text of a content block, "content.<i>.text"?

def blocks source · line 183 · raw

@d:List<&2, Leaf> -> String

The text blocks of an Anthropic message, joined.

def reply.of source · line 190 · raw

@api:Api -> @+d:List<&2, Leaf> -> Reply

def reply.got source · line 199 · raw

@api:Api -> @b:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> @m:Maybe<&2, List<&2, Leaf>> -> Result<&1, &1, Err, Reply>

def reply.copy source · line 206 · raw

@api:Api -> @r:Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes) -> Result<&1, &1, Err, Reply>

def reply source · line 211 · raw

@api:Api -> @b:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> Result<&1, &1, Err, Reply>

A 2xx response body as a Reply.

def ev.text source · line 223 · raw

@+s:String -> Ev

def ev.doc source · line 226 · raw

@m:Maybe<&2, List<&2, Leaf>> -> List<&2, Leaf>

def ev.anthropic source · line 229 · raw

@+name:String -> @data:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> Ev

def ev.openai source · line 235 · raw

@+s:String -> Ev

def ev source · line 242 · raw

@api:Api -> @e:0xce2a0a9d4864ec82024ab27469071032/sse.Message -> Ev

What one SSE event of api means.

def ok source · line 253 · raw

@+s:U32 -> Bool

def again.status source · line 257 · raw

@+s:U32 -> Bool

Statuses a later try may clear: 408, 409, 429, and 5xx, which holds Anthropic's 529.

def again.err source · line 261 · raw

@+e:0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err -> Maybe<&2, String>

A refused connect or a timeout may clear.

def again.of source · line 270 · raw

@+status:U32 -> @+h:Map<&2, List<&2, String>> -> Maybe<&2, String>

def again source · line 274 · raw

@r:Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Res> -> Pair(Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Res>, Maybe<&2, String>)

The result, with Some{Retry-After} when another try may help. Hairpin's judge for every call.

def body.ok source · line 282 · raw

@good:Bool -> @+status:U32 -> @b:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> Result<&1, &1, Err, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes>

def body.of source · line 290 · raw

@r:Result<&1, &1, 0x5f997a9351e7133e43ba9611e734f8a6/hairpin.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Res> -> Result<&1, &1, Err, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes>

The 2xx body, or why there is none.

def url source · line 298 · raw

@api:Api -> @+base:String -> String

def new source · line 307 · raw

@+api:Api -> @base:String -> @headers:Map<&2, List<&2, String>> -> Client

def json source · line 310 · raw

@h:Map<&2, List<&2, String>> -> Map<&2, List<&2, String>>

def anthropic source · line 314 · raw

@key:String -> Client

base https://api.anthropic.com, 10 min steps, 2 retries.

def openai source · line 318 · raw

@key:String -> Client

base https://api.openai.com/v1, 10 min steps, 2 retries.

def base source · line 322 · raw

@c:Client -> @url:String -> Client

Another base URL, as for a provider that serves the same API.

def header source · line 327 · raw

@c:Client -> @k:String -> @v:String -> Client

A header on every request, as anthropic-beta. It replaces one of the same name.

def timeout source · line 331 · raw

@c:Client -> @+ms:U32 -> Client

def retry source · line 335 · raw

@c:Client -> @n:U32 -> Client

def close source · line 339 · raw

@c:Client -> IO(Unit)

def call.done source · line 343 · raw

@+api:Api -> @base:String -> @headers:Map<&2, List<&2, String>> -> @+ms:U32 -> @+n:U32 -> @x:Pair(0x5f997a9351e7133e43ba9611e734f8a6/hairpin.Client, Pair(0x5f997a9351e7133e43ba9611e734f8a6/retry.Budget, Result<&1, &1, 0x5f997a9351e7133e43ba9611e734f8a6/hairpin.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Res>)) -> Pair(Client, Result<&1, &1, Err, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes>)

def call.budget source · line 348 · raw

@+api:Api -> @+base:String -> @+headers:Map<&2, List<&2, String>> -> @body:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> @y:Pair(0x5f997a9351e7133e43ba9611e734f8a6/hairpin.Client, 0x5f997a9351e7133e43ba9611e734f8a6/retry.Budget) -> IO(Pair(0x5f997a9351e7133e43ba9611e734f8a6/hairpin.Client, Pair(0x5f997a9351e7133e43ba9611e734f8a6/retry.Budget, Result<&1, &1, 0x5f997a9351e7133e43ba9611e734f8a6/hairpin.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Res>)))

def call source · line 357 · raw

@c:Client -> @body:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> IO(Pair(Client, Result<&1, &1, Err, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes>))

POST body to the API with n retries on a budget of n + 1; the 2xx body comes back. These POSTs are safe to repeat, so Hairpin retries them as again judges.

def raw.got source · line 364 · raw

@b:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> @m:Maybe<&1, 0x584fc27920487ceab242392391418d7f/json.Val> -> Result<&1, &1, Err, 0x584fc27920487ceab242392391418d7f/json.Val>

def raw.copy source · line 371 · raw

@r:Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes) -> Result<&1, &1, Err, 0x584fc27920487ceab242392391418d7f/json.Val>

def raw.of source · line 375 · raw

@r:Result<&1, &1, Err, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes> -> Result<&1, &1, Err, 0x584fc27920487ceab242392391418d7f/json.Val>

def raw.done source · line 382 · raw

@x:Pair(Client, Result<&1, &1, Err, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes>) -> Pair(Client, Result<&1, &1, Err, 0x584fc27920487ceab242392391418d7f/json.Val>)

def send.raw source · line 387 · raw

@c:Client -> @v:0x584fc27920487ceab242392391418d7f/json.Val -> IO(Pair(Client, Result<&1, &1, Err, 0x584fc27920487ceab242392391418d7f/json.Val>))

Any request body, as for tools or images, and the reply as JSON.

def send.of source · line 392 · raw

@r:Result<&1, &1, Err, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes> -> @+api:Api -> Result<&1, &1, Err, Reply>

def send.done source · line 399 · raw

@+api:Api -> @x:Pair(Client, Result<&1, &1, Err, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes>) -> Pair(Client, Result<&1, &1, Err, Reply>)

def send source · line 404 · raw

@c:Client -> @req:Req -> IO(Pair(Client, Result<&1, &1, Err, Reply>))

One whole reply.

def open.res source · line 415 · raw

@x:Pair(0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Res) -> Pair(Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>, Maybe<&2, String>)

def open.judge source · line 420 · raw

@r:Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream> -> Pair(Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>, Maybe<&2, String>)

def open.kept source · line 427 · raw

@body:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> @j:Pair(Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>, Maybe<&2, String>) -> Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, Pair(Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>, Maybe<&2, String>))

def open.once source · line 431 · raw

@+url:String -> @+headers:Map<&2, List<&2, String>> -> @+ms:U32 -> @r:Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes) -> IO(Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, Pair(Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>, Maybe<&2, String>)))

def open.drop source · line 438 · raw

@r:Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream> -> IO(Unit)

def open.go source · line 446 · raw

@left:Nat -> @+n:Nat -> @+url:String -> @+headers:Map<&2, List<&2, String>> -> @+ms:U32 -> @t:Pair(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes, Pair(Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>, Maybe<&2, String>)) -> IO(Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>)

A refused stream closes before the next try.

def drain source · line 465 · raw

@f:Nat -> @x:Pair(0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream, Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, Maybe<&1, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes>>) -> @acc:0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes -> IO(0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes)

A refused stream's error body, at most f more pieces, then the stream closed.

def start.refused source · line 490 · raw

@+status:U32 -> @h:0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream -> IO(Result<&1, &1, Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>)

def start.ok source · line 496 · raw

@good:Bool -> @+status:U32 -> @h:0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream -> IO(Result<&1, &1, Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>)

def start.res source · line 503 · raw

@x:Pair(0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Res) -> IO(Result<&1, &1, Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>)

def start source · line 508 · raw

@r:Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream> -> IO(Result<&1, &1, Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream>)

def stream.of source · line 515 · raw

@r:Result<&1, &1, Err, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream> -> @+api:Api -> Result<&1, &1, Err, Stream>

def stream.raw source · line 523 · raw

@c:Client -> @v:0x584fc27920487ceab242392391418d7f/json.Val -> IO(Pair(Client, Result<&1, &1, Err, Stream>))

Any request body that asks for SSE, as for tools. A stream opens its own connection, outside the pool.

def stream source · line 532 · raw

@c:Client -> @req:Req -> IO(Pair(Client, Result<&1, &1, Err, Stream>))

The reply as it is written: read it with next.

def stream.close source · line 536 · raw

@s:Stream -> IO(Unit)

def event.read source · line 545 · raw

@rd:0xce2a0a9d4864ec82024ab27469071032/sse.Reader -> @x:Pair(0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream, Result<&1, &1, 0x6a946b742ab6f76f0ba5b90b14da52ad/http.Err, Maybe<&1, 0x49814d83de8f70993a43e1002be29ecd/bytes.Bytes>>) -> Rn

def event.loop source · line 558 · raw

@f:Nat -> @st:Rn -> IO(Pair(0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream, Pair(0xce2a0a9d4864ec82024ab27469071032/sse.Reader, Result<&1, &1, Err, Maybe<&1, 0xce2a0a9d4864ec82024ab27469071032/sse.Message>>)))

f: pieces left to read before the stream counts as stalled.

def event.done source · line 581 · raw

@+api:Api -> @x:Pair(0x6a946b742ab6f76f0ba5b90b14da52ad/http.Stream, Pair(0xce2a0a9d4864ec82024ab27469071032/sse.Reader, Result<&1, &1, Err, Maybe<&1, 0xce2a0a9d4864ec82024ab27469071032/sse.Message>>)) -> Pair(Stream, Result<&1, &1, Err, Maybe<&1, 0xce2a0a9d4864ec82024ab27469071032/sse.Message>>)

def event source · line 586 · raw

@s:Stream -> IO(Pair(Stream, Result<&1, &1, Err, Maybe<&1, 0xce2a0a9d4864ec82024ab27469071032/sse.Message>>))

The next SSE event, or None when the stream ends. An event the end cuts off is dropped.

def next.ev source · line 592 · raw

@r:Result<&1, &1, Err, Maybe<&1, 0xce2a0a9d4864ec82024ab27469071032/sse.Message>> -> @+api:Api -> Ev

def next.of source · line 603 · raw

@x:Pair(Stream, Result<&1, &1, Err, Maybe<&1, 0xce2a0a9d4864ec82024ab27469071032/sse.Message>>) -> Pair(Stream, Ev)

def next.loop source · line 609 · raw

@f:Nat -> @x:Pair(Stream, Ev) -> IO(Pair(Stream, Result<&1, &1, Err, Maybe<&2, String>>))

f: events left to skip.

def next source · line 629 · raw

@s:Stream -> IO(Pair(Stream, Result<&1, &1, Err, Maybe<&2, String>>))

The next piece of reply text, or None at the end.