src/syntax/lex.bend checks

raw source on the hub · import 0xde9bb08f7de298b03207fb5797ede9a5/src/syntax/lex.bend as Lex

src/syntax/lex: a source as tokens with positions. Lossless: every char lands in exactly one token, so the texts concatenate back to the source, whatever the source is. A string runs to its closing quote across newlines, as in bend, so an unclosed one runs to the end of the file (bend rejects that file). A char literal and a comment end at their line's end. One char per step, a tail-recursive state machine.

A name is letters, digits, _ and . (List.map, M.go). Operator chars group into runs (->, <-, =>, ==, ++), except :, which always stands alone: List<U32>: must not read >:.

Kinds tell apart what binds from what does not, so a walk over the tokens can branch by constructor: a keyword (TKey), a dotted name (TDotted: never a binder), a capitalized one (TUpper: a constructor before {, a type, or a binder), _ (TWild), and the operators that bind: : = <- -> => @ &.

2 imports

import Base
import ../lazy/lazy.bend as Lazy

Types

type TokKind source · line 22 · raw

Data

the kinds; TName is a plain name (lowercase, undotted): what a pattern can bind

TName
```
TokKind
```
TUpper
```
TokKind
```
TDotted
```
TokKind
```
TWild
```
TokKind
```
TKey
```
TokKind
```
TNum
```
TokKind
```
TStr
```
TokKind
```
TChar
```
TokKind
```
TComment
```
TokKind
```
TSpace
```
TokKind
```
TNewline
```
TokKind
```
TOp
```
TokKind
```
TColon
```
TokKind
```
TEq
```
TokKind
```
TBind
```
TokKind
```
TArrow
```
TokKind
```
TLam
```
TokKind
```
TAll
```
TokKind
```
TAmp
```
TokKind
```
TOpen
```
TokKind
```
TClose
```
TokKind
```
TComma
```
TokKind
```

type Tok source · line 47 · raw

Data

line and col are 0-based, of the token's first char

Tok

@kind:TokKind -> @text:String -> @line:U32 -> @col:U32 -> Tok

type Class source · line 51 · raw

Data

what a char is, for the machine

CAlpha
```
Class
```
CDigit
```
Class
```
CDot
```
Class
```
CQuote
```
Class
```
CTick
```
Class
```
CBack
```
Class
```
CHash
```
Class
```
CSpace
```
Class
```
CNewline
```
Class
```
COp
```
Class
```
CColon
```
Class
```
COpen
```
Class
```
CClose
```
Class
```
CComma
```
Class
```

type Mode source · line 68 · raw

Data

what the machine is inside of; Fresh: the next char starts a token

Fresh
```
Mode
```
InName
```
Mode
```
InNum
```
Mode
```
InStr
```
Mode
```
InStrEsc
```
Mode
```
InChar
```
Mode
```
InCharEsc
```
Mode
```
InComment
```
Mode
```
InSpace
```
Mode
```
InOp
```
Mode
```

type St source · line 81 · raw

Data

buf and toks are reversed; (sl, sc) is where the current token started

St

@mode:Mode -> @kind:TokKind -> @line:U32 -> @col:U32 -> @sl:U32 -> @sc:U32 -> @buf:List<&2, Char> -> @toks:List<&2, Tok> -> St

Definitions

def classify.go source · line 89 · raw

@+xx:U32 -> @other:Class -> Class

the class of a code point, given the class any other character would get. Written as comparisons rather than as case '.': arms: a character literal in a pattern is a U32 literal inside a constructor pattern, and bend's C backend pays about 90 MB for each one -- these eighteen cost 1.57 GB as literal arms and 0.10 GB written this way.

def classify source · line 111 · raw

@+cc:Char -> Class

a char's class

def continues source · line 117 · raw

@mm:Mode -> @cls:Class -> Bool

does a char of this class continue the token the machine is inside of?

def after source · line 159 · raw

@mm:Mode -> @cls:Class -> Mode

the mode after a char that continued the token

def begin.mode source · line 177 · raw

@cls:Class -> Mode

the mode and the kind of the token a char starts

def begin.kind source · line 200 · raw

@cls:Class -> TokKind

def word source · line 226 · raw

@buf:List<&2, Char> -> String

the buffered chars as a string, in order

def is_keyword source · line 230 · raw

@+tt:String -> Bool

is the text one of Bend's keywords?

def upper_first source · line 236 · raw

@cs:List<&2, Char> -> Bool

does the text start with a capital?

def refine_name source · line 244 · raw

@+tt:String -> TokKind

a whole name's kind

def refine_op source · line 251 · raw

@+tt:String -> TokKind

a whole operator's kind

def refine source · line 261 · raw

@kind:TokKind -> @tt:String -> TokKind

a name's or an operator's kind, once its text is whole

def flush source · line 271 · raw

@kind:TokKind -> @sl:U32 -> @sc:U32 -> @+buf:List<&2, Char> -> @+toks:List<&2, Tok> -> List<&2, Tok>

the token so far joins the others

def is_newline source · line 276 · raw

@cc:Char -> Bool

is the char a line break, '\n'? The one char whose class is CNewline

def step source · line 282 · raw

@+cc:Char -> @st:St -> St

one char: it continues the current token, or starts a new one. Only the branch taken is built: the token is flushed (its text reversed, its kind refined) once, where it ends, not on every char of it

def run source · line 293 · raw

@cs:List<&2, Char> -> @st:St -> St

every char, in order

def finish source · line 301 · raw

@st:St -> List<&2, Tok>

the tokens once the chars are over, the last one flushed

def tokens source · line 306 · raw

@source:String -> List<&2, Tok>

a source as tokens

def text.go source · line 310 · raw

@toks:List<&2, Tok> -> @acc:List<&2, String> -> List<&2, String>

the tokens' texts, back to back: the source

def text source · line 318 · raw

@toks:List<&2, Tok> -> String

the tokens' texts, back to back: the source

def is_open source · line 322 · raw

@kk:TokKind -> Bool

an opening bracket?

def is_comma source · line 330 · raw

@kk:TokKind -> Bool

a comma?

def is_name source · line 338 · raw

@kk:TokKind -> Bool

any identifier: a name, a keyword, a dotted or a capitalized one, _

def is_nl source · line 354 · raw

@kk:TokKind -> Bool

a newline?

def significant source · line 362 · raw

@kk:TokKind -> Bool

not space, a newline or a comment

def is_str source · line 374 · raw

@kk:TokKind -> Bool

a string literal?