utf8/utf8.bend checks

raw source on the hub · import 0x64e1b9e0466cf913fa57e70aeb11c176/utf8/utf8.bend as Utf8

1 import

import Base

Types

type U8 source · line 16 · raw

Data

a UTF-8 decoder that rejects what the runtime's text reader would turn into U+FFFD: overlong forms, surrogates, code points past U+10FFFF, and cut-off sequences. need is how many continuation bytes are still to come, and lo and hi bound the next one; at counts the chars so far

U8

@need:U32 -> @code:U32 -> @lo:U32 -> @hi:U32 -> @at:U32 -> @out:String -> U8

U8Bad
```
@at:U32 -> @out:String -> U8
```

type Text source · line 73 · raw

Data

the decoded text, or where the first bad sequence starts and the chars before it (reversed)

TOk
```
@text:String -> Text
```
TBad
```
@at:U32 -> @before:String -> Text
```

Definitions

def in_range source · line 9 · raw

@+x:U32 -> @lo:U32 -> @hi:U32 -> Bool

def u8.char source · line 20 · raw

@+c:U32 -> @+at:U32 -> @out:String -> U8

def u8.lead source · line 23 · raw

@ok:Bool -> @+b:U32 -> @+at:U32 -> @out:String -> U8

def u8.start source · line 36 · raw

@ascii:Bool -> @+b:U32 -> @+at:U32 -> @out:String -> U8

def u8.more source · line 43 · raw

@last:Bool -> @+c:U32 -> @+need:U32 -> @+at:U32 -> @out:String -> U8

def u8.cont source · line 50 · raw

@ok:Bool -> @+b:U32 -> @+need:U32 -> @+c:U32 -> @+at:U32 -> @out:String -> U8

def u8.go source · line 57 · raw

@start:Bool -> @+b:U32 -> @+need:U32 -> @+c:U32 -> @+lo:U32 -> @+hi:U32 -> @+at:U32 -> @out:String -> U8

def u8.step source · line 64 · raw

@u:U8 -> @+b:U32 -> U8

def u8.end source · line 77 · raw

@done:Bool -> @+at:U32 -> @out:String -> Text

def decode.go source · line 84 · raw

@bs:List<&2, U32> -> @u:U8 -> Text

def decode source · line 93 · raw

@bs:List<&2, U32> -> Text

def back_col source · line 97 · raw

@before:String -> @+col:U32 -> U32

the column of the char after before (reversed), counting from col

def back_line source · line 106 · raw

@before:String -> @+line:U32 -> U32

def six source · line 120 · raw

@+x:U32 -> @n:Nat -> U32

the six bits a continuation byte carries, from bit n up

def top source · line 123 · raw

@+x:U32 -> @n:Nat -> @+mark:U32 -> U32

def enc.width source · line 126 · raw

@one:Bool -> @two:Bool -> @three:Bool -> @+x:U32 -> @out:List<&2, U32> -> List<&2, U32>

def enc.push source · line 138 · raw

@+x:U32 -> @out:List<&2, U32> -> List<&2, U32>

a code point's bytes, pushed onto a reversed list

def enc.go source · line 141 · raw

@s:String -> @out:List<&2, U32> -> List<&2, U32>

def encode source · line 149 · raw

@s:String -> List<&2, U32>

the UTF-8 bytes of a text