syntax/lex.bend checks

raw source on the hub · import 0x729eecea86ea5a2cdba3a2856a313bca/syntax/lex.bend as Lex

syntax/lex: a source as tokens with positions. Lossless: every char lands in exactly one token, so the texts concatenate back to the source, whatever the source is -- an unclosed string ends at its line's end rather than swallowing the file. One char per step, a tail-recursive state machine.

A name is letters, digits, _ and . (List.map, M.go). Operator chars group into runs (->, <-, =>, ==, ++), except :, which always stands alone: List<U32>: must not read >:.

Kinds tell apart what binds from what does not, so a walk over the tokens can branch by constructor: a keyword (TKey), a dotted name (TDotted: never a binder), a capitalized one (TUpper: a constructor before {, a type, or a binder), _ (TWild), and the operators that bind: : = <- -> => @ &.

1 import

import Base

Types

type TokKind source · line 19 · raw

Data

the kinds; TName is a plain name (lowercase, undotted): what a pattern can bind

TName
```
TokKind
```
TUpper
```
TokKind
```
TDotted
```
TokKind
```
TWild
```
TokKind
```
TKey
```
TokKind
```
TNum
```
TokKind
```
TStr
```
TokKind
```
TChar
```
TokKind
```
TComment
```
TokKind
```
TSpace
```
TokKind
```
TNewline
```
TokKind
```
TOp
```
TokKind
```
TColon
```
TokKind
```
TEq
```
TokKind
```
TBind
```
TokKind
```
TArrow
```
TokKind
```
TLam
```
TokKind
```
TAll
```
TokKind
```
TAmp
```
TokKind
```
TOpen
```
TokKind
```
TClose
```
TokKind
```
TComma
```
TokKind
```

type Tok source · line 44 · raw

Data

line and col are 0-based, of the token's first char

Tok

@kind:TokKind -> @text:String -> @line:U32 -> @col:U32 -> Tok

type Class source · line 48 · raw

Data

what a char is, for the machine

CAlpha
```
Class
```
CDigit
```
Class
```
CDot
```
Class
```
CQuote
```
Class
```
CTick
```
Class
```
CBack
```
Class
```
CHash
```
Class
```
CSpace
```
Class
```
CNewline
```
Class
```
COp
```
Class
```
CColon
```
Class
```
COpen
```
Class
```
CClose
```
Class
```
CComma
```
Class
```

type Mode source · line 65 · raw

Data

what the machine is inside of; Fresh: the next char starts a token

Fresh
```
Mode
```
InName
```
Mode
```
InNum
```
Mode
```
InStr
```
Mode
```
InStrEsc
```
Mode
```
InChar
```
Mode
```
InCharEsc
```
Mode
```
InComment
```
Mode
```
InSpace
```
Mode
```
InOp
```
Mode
```

type St source · line 78 · raw

Data

buf and toks are reversed; (sl, sc) is where the current token started

St

@mode:Mode -> @kind:TokKind -> @line:U32 -> @col:U32 -> @sl:U32 -> @sc:U32 -> @buf:List<&2, Char> -> @toks:List<&2, Tok> -> St

Definitions

def classify.go source · line 86 · raw

@+x:U32 -> @other:Class -> Class

the class of a code point, given the class any other character would get. Written as comparisons rather than as case '.': arms: a character literal in a pattern is a U32 literal inside a constructor pattern, and bend's C backend pays about 90 MB for each one -- these eighteen cost 1.57 GB as literal arms and 0.10 GB written this way.

def classify source · line 108 · raw

@+c:Char -> Class

a char's class

def continues source · line 114 · raw

@m:Mode -> @cls:Class -> Bool

does a char of this class continue the token the machine is inside of?

def after source · line 160 · raw

@m:Mode -> @cls:Class -> Mode

the mode after a char that continued the token

def begin.mode source · line 178 · raw

@cls:Class -> Mode

the mode and the kind of the token a char starts

def begin.kind source · line 201 · raw

@cls:Class -> TokKind

def word source · line 227 · raw

@buf:List<&2, Char> -> String

the buffered chars as a string, in order

def is_keyword source · line 231 · raw

@+t:String -> Bool

is the text one of Bend's keywords?

def upper_first source · line 237 · raw

@cs:List<&2, Char> -> Bool

does the text start with a capital?

def refine_name source · line 245 · raw

@+t:String -> TokKind

a whole name's kind

def refine_op source · line 252 · raw

@+t:String -> TokKind

a whole operator's kind

def refine source · line 262 · raw

@kind:TokKind -> @t:String -> TokKind

a name's or an operator's kind, once its text is whole

def flush source · line 272 · raw

@kind:TokKind -> @sl:U32 -> @sc:U32 -> @+buf:List<&2, Char> -> @+toks:List<&2, Tok> -> List<&2, Tok>

the token so far joins the others

def is_newline source · line 277 · raw

@cls:Class -> Bool

is the class a newline's?

def step source · line 285 · raw

@+c:Char -> @st:St -> St

one char: it continues the current token, or starts a new one

def run source · line 296 · raw

@cs:List<&2, Char> -> @st:St -> St

every char, in order

def finish source · line 304 · raw

@st:St -> List<&2, Tok>

the tokens once the chars are over, the last one flushed

def tokens source · line 309 · raw

@source:String -> List<&2, Tok>

a source as tokens

def text.go source · line 313 · raw

@toks:List<&2, Tok> -> @acc:List<&2, String> -> List<&2, String>

the tokens' texts, back to back: the source

def text source · line 321 · raw

@toks:List<&2, Tok> -> String

the tokens' texts, back to back: the source

def is_open source · line 325 · raw

@k:TokKind -> Bool

an opening bracket?

def is_close source · line 333 · raw

@k:TokKind -> Bool

a closing bracket?

def is_comma source · line 341 · raw

@k:TokKind -> Bool

a comma?

def is_name source · line 349 · raw

@k:TokKind -> Bool

any identifier: a name, a keyword, a dotted or a capitalized one, _

def is_op source · line 365 · raw

@k:TokKind -> Bool

an operator of any kind

def is_nl source · line 387 · raw

@k:TokKind -> Bool

a newline?

def significant source · line 395 · raw

@k:TokKind -> Bool

not space, a newline or a comment