~/bend-docscommunity

unicode.bend checks

raw source on the hub · import bend-kit-unicode@0.1.0.0/unicode.bend as Unicode

Unicode 17.0 general category, NFC/NFD, full case folding, and grapheme clusters. Source: https://github.com/paymog/bend-kit/tree/main/unicode

2 imports
import Base
import ./tables.bend as T

Types

type Nfc source · line 103 · raw

Data

Composition state: is there a starter, the starter, the class of the last pending mark (0 when none), the marks after the starter reversed, and the output reversed.

type Gr source · line 205 · raw

Data

Segmentation state: the previous character's T.gb, ri, ep, incb, the current cluster reversed, and the finished clusters reversed.

Definitions

def category source · line 8 · raw

@c:Char -> String

General category as its two-letter UCD alias: "Lu", "Nd", "Zs", ... Unassigned is "Cn".

def ccc source · line 13 · raw

@c:Char -> U32

Canonical combining class: 0 for a starter.

def hangul.dec source · line 18 · raw

@+s:U32 -> String

Hangul syllables decompose and compose by formula (Unicode §3.12).

def decomp.if source · line 23 · raw

@hangul:Bool -> @+s:U32 -> @+c:U32 -> String

def decomp source · line 31 · raw

@+c:U32 -> String

The full canonical decomposition of c; c itself when it has none.

def order.gt source · line 36 · raw

@acc:String -> @+k:U32 -> Bool

Does the reversed output start with a mark of class above k?

def order.put source · line 44 · raw

@acc:String -> @go:Bool -> @+c:U32 -> @+k:U32 -> String

Canonical ordering: c, of class k > 0, goes past each mark of a higher class. go says whether acc's head is one.

def nfd.put source · line 55 · raw

@starter:Bool -> @+acc:String -> @+c:U32 -> @+k:U32 -> String

def nfd.push source · line 62 · raw

@d:String -> @acc:String -> String

def nfd.go source · line 70 · raw

@s:String -> @acc:String -> String

def nfd source · line 78 · raw

@s:String -> String

Normalization Form D (UAX #15): full canonical decomposition, then canonical order.

def hangul.lvt source · line 81 · raw

@ok:Bool -> @+a:U32 -> @+t:U32 -> Maybe<&2, U32>

def hangul.lv source · line 88 · raw

@ok:Bool -> @+a:U32 -> @+b:U32 -> Maybe<&2, U32>

def compose source · line 98 · raw

@+a:U32 -> @+b:U32 -> Maybe<&2, U32>

The primary composite of a and b, if there is one.

def nfc.flush source · line 106 · raw

@has:Bool -> @+s:U32 -> @pend:String -> @out:String -> String

def nfc.miss source · line 113 · raw

@starter:Bool -> @+has:Bool -> @+s:U32 -> @+last:U32 -> @pend:String -> @out:String -> @+c:U32 -> @+k:U32 -> Nfc

def nfc.hit source · line 120 · raw

@m:Maybe<&2, U32> -> @+has:Bool -> @+s:U32 -> @+last:U32 -> @pend:String -> @out:String -> @+c:U32 -> @+k:U32 -> Nfc

def nfc.try source · line 127 · raw

@free:Bool -> @+has:Bool -> @+s:U32 -> @+last:U32 -> @pend:String -> @out:String -> @+c:U32 -> @+k:U32 -> Nfc

def nfc.step source · line 135 · raw

@z:Nfc -> @+c:U32 -> Nfc

c joins the starter unless a mark between them has class 0 or at least c's class.

def nfc.go source · line 140 · raw

@s:String -> @z:Nfc -> String

def nfc source · line 149 · raw

@s:String -> String

Normalization Form C (UAX #15): NFD, then canonical composition.

def fold.go source · line 152 · raw

@s:String -> @acc:String -> String

def fold source · line 160 · raw

@s:String -> String

Full case folding (CaseFolding.txt, statuses C and F): "Straße" folds to "strasse".

def gcb source · line 166 · raw

@+q:U32 -> U32

T.gb packs Grapheme_Cluster_Break in bits 0-3 (Other 0, CR 1, LF 2, Control 3, Extend 4, ZWJ 5, Regional_Indicator 6, Prepend 7, SpacingMark 8, L 9, V 10, T 11, LV 12, LVT 13), Extended_Pictographic in bit 4, and InCB in bits 5-6 (Consonant 1, Extend 2, Linker 3).

def eq source · line 169 · raw

@+g:U32 -> @+v:U32 -> Bool

def hangul.join source · line 172 · raw

@+p:U32 -> @+q:U32 -> Bool

def brk source · line 180 · raw

@+p:U32 -> @+q:U32 -> @+ri:Bool -> @+ep:U32 -> @+incb:U32 -> Bool

UAX #29 GB3-GB13. ri: the regional indicators before q are odd in number. ep: 1 after ExtPict Extend*, 2 after ExtPict Extend* ZWJ. incb: 1 after an InCB consonant and extenders, 2 once a linker follows it.

def next.ri source · line 190 · raw

@+p:U32 -> @+q:U32 -> @+ri:Bool -> Bool

def next.ep source · line 193 · raw

@+q:U32 -> @+ep:U32 -> U32

def next.incb source · line 197 · raw

@+q:U32 -> @+incb:U32 -> U32

def gr.cut source · line 208 · raw

@b:Bool -> @+q:U32 -> @+ri:Bool -> @+ep:U32 -> @+incb:U32 -> @+c:U32 -> @cur:String -> @out:List<&2, String> -> Gr

def gr.step source · line 215 · raw

@z:Gr -> @+c:U32 -> Gr

def gr.go source · line 220 · raw

@s:String -> @z:Gr -> List<&2, String>

def graphemes source · line 229 · raw

@s:String -> List<&2, String>

Extended grapheme clusters (UAX #29): "e\u{301}x" is ["e\u{301}", "x"].