lex.bend checks

raw source on the hub · import 0xa3c2445eb44c5d8406e6229be518fccb/lex.bend as Lex

ezjson/lex: text to tokens, one char per step: a tail-recursive state machine over the char list, so it needs no fuel and runs as a loop.

1 import

import Base

Types

type Tok source · line 7 · raw

Data

the tokens: brackets, :, ,, a string (decoded), or a bare word (a number, true, false, null)

TOpenArr
```
Tok
```
TCloseArr
```
Tok
```
TOpenObj
```
Tok
```
TCloseObj
```
Tok
```
TColon
```
Tok
```
TComma
```
Tok
```
TStr
```
@s:String -> Tok
```
TWord
```
@raw:String -> Tok
```
TBad
```
Tok
```

type Class source · line 20 · raw

Data

what a char is: punctuation (with its token), a quote, whitespace, or a word char

CPunct
```
@t:Tok -> Class
```
CQuote
```
Class
```
CBack
```
Class
```
CSpace
```
Class
```
COther
```
Class
```

type Mode source · line 29 · raw

Data

what the machine is inside of: nothing, a bare word, a string, an escape in a string, or a \\u escape with hex digits still owed

Idle
```
Mode
```
Word
```
Mode
```
Str
```
Mode
```
Esc
```
Mode
```
Uni
```
@left:Nat -> @acc:U32 -> Mode
```

type Lex source · line 37 · raw

Data

the lexer's state: buf and toks are both reversed

Lex

@mode:Mode -> @buf:List<&2, Char> -> @toks:List<&2, Tok> -> Lex

Definitions

def classify.go source · line 43 · raw

@+x:U32 -> Class

a char's class, by code point. Written as comparisons rather than as case '[': arms: a character literal in a pattern is a U32 literal inside a constructor pattern, and bend's C backend pays about 90 MB for each one.

def classify source · line 59 · raw

@c:Char -> Class

a char's class

def text source · line 63 · raw

@buf:List<&2, Char> -> String

the buffered chars (reversed) as a string

def unescape source · line 67 · raw

@+c:Char -> Char

the char an escape stands for; an unknown escape stands for itself

def hex source · line 76 · raw

@c:Char -> U32

a hex digit's value

def escape source · line 82 · raw

@c:Char -> @buf:List<&2, Char> -> @toks:List<&2, Tok> -> Lex

the char after a backslash in a string

def unicode source · line 90 · raw

@left:Nat -> @acc:U32 -> @c:Char -> @buf:List<&2, Char> -> @toks:List<&2, Tok> -> Lex

one hex digit of a \\uXXXX escape; the code point once four are in

def step source · line 100 · raw

@mode:Mode -> @cls:Class -> @c:Char -> @buf:List<&2, Char> -> @toks:List<&2, Tok> -> Lex

one char, by the mode and the char's class

def feed source · line 134 · raw

@+c:Char -> @st:Lex -> Lex

one char into the machine

def run source · line 139 · raw

@cs:List<&2, Char> -> @st:Lex -> Lex

every char, in order

def finish source · line 147 · raw

@st:Lex -> List<&2, Tok>

the tokens, in order; text that ends inside a string ends in TBad

def tokens source · line 158 · raw

@s:String -> List<&2, Tok>

a text as tokens