tensor_file.bend checks

raw source on the hub · import stelliferous@0.0.2.0/tensor_file.bend as Tensor_file

Raw little-endian FP32 files and NumPy .npy files through the official File effects.

4 imports

import Base
import ./storage_buffer.bend as Buffers
import ./tensor.bend as Tensors
import ./tensor_layout.bend as Layout

Types

type Words source · line 33 · raw

Data

Words

@values:List<&2, F32> -> @count:U32 -> @word:U32 -> @weight:U32 -> Words

type Target source · line 37 · raw

Type

The buffer of the tensor, the values found so far, and a word in progress.

Target

@buffer:0x9be3b13bf249759bc81e1958bcd1a4c0/storage_buffer.Buffer<F32> -> @found:U32 -> @word:U32 -> @weight:U32 -> Target

type Reading source · line 40 · raw

Type

More
```
@file:File -> @target:Target -> Reading
```

End

@file:File -> @target:Target -> @complete:Bool -> Reading

type Outcome source · line 44 · raw

Type

Outcome

@target:Target -> @complete:Bool -> Outcome

type Header source · line 168 · raw

Data

Header

@shape:List<&2, U32> -> @data:List<&2, U32> -> Header

type Writing source · line 299 · raw

Type

Pending

@file:File -> @storage:Array<F32> -> @offset:U32 -> @remaining:Nat -> Writing

Finished
```
@file:File -> @complete:Bool -> Writing
```

Definitions

def Tensor source · line 7 · raw

Type

def Read source · line 10 · raw

Type

def Written source · line 13 · raw

Type

def chunk source · line 17 · raw

U32

Values per File effect.

def from_bits source · line 22 · raw

@word:U32 -> F32

def to_bits source · line 26 · raw

@value:F32 -> U32

def gather source · line 47 · raw

@full:Bool -> @values:List<&2, F32> -> @+count:U32 -> @+word:U32 -> @+weight:U32 -> Words

def push source · line 52 · raw

@+byte:U32 -> @words:Words -> Words

def trickle source · line 56 · raw

@bytes:List<&2, U32> -> @words:Words -> Words

def quads source · line 62 · raw

@bytes:List<&2, U32> -> @values:List<&2, F32> -> @+count:U32 -> Words

Four bytes at a time while the bytes are whole words; the values are newest first.

def words source · line 68 · raw

@whole:Bool -> @bytes:List<&2, U32> -> @+word:U32 -> @+weight:U32 -> Words

def placed source · line 74 · raw

@read:Words -> @buffer:0x9be3b13bf249759bc81e1958bcd1a4c0/storage_buffer.Buffer<F32> -> @+found:U32 -> @+count:U32 -> Target

Values beyond the count of the shape are counted and not written.

def fill source · line 79 · raw

@bytes:List<&2, U32> -> @target:Target -> @+count:U32 -> Target

def absorbed source · line 83 · raw

@bytes:List<&2, U32> -> @file:File -> @target:Target -> @+count:U32 -> Reading

def received source · line 88 · raw

@result:Result<&1, &1, Pair(U32, String), List<&2, U32>> -> @file:File -> @target:Target -> @+count:U32 -> Reading

def taken source · line 93 · raw

@read:Read -> @target:Target -> @+count:U32 -> Reading

def closed source · line 97 · raw

@file:File -> @target:Target -> @complete:Bool -> IO(Outcome)

def read_all source · line 103 · raw

@fuel:Nat -> @reading:Reading -> @+count:U32 -> IO(Outcome)

A file longer than its shape stops at the fuel and fails the count.

def fresh source · line 113 · raw

@+count:U32 -> Target

def fuel source · line 116 · raw

@+count:U32 -> Nat

def fault source · line 119 · raw

@operation:String -> @path:String -> @detail:String -> @shape:List<&2, U32> -> Tensor

def sized source · line 122 · raw

@matches:Bool -> @operation:String -> @path:String -> @+shape:List<&2, U32> -> @buffer:0x9be3b13bf249759bc81e1958bcd1a4c0/storage_buffer.Buffer<F32> -> Tensor

def whole source · line 127 · raw

@aligned:Bool -> @operation:String -> @path:String -> @+shape:List<&2, U32> -> @buffer:0x9be3b13bf249759bc81e1958bcd1a4c0/storage_buffer.Buffer<F32> -> @+found:U32 -> @+count:U32 -> Tensor

def aligned source · line 132 · raw

@operation:String -> @path:String -> @+shape:List<&2, U32> -> @target:Target -> @+count:U32 -> Tensor

def checked source · line 136 · raw

@complete:Bool -> @operation:String -> @path:String -> @+shape:List<&2, U32> -> @target:Target -> @+count:U32 -> Tensor

def loaded source · line 141 · raw

@operation:String -> @path:String -> @+shape:List<&2, U32> -> @outcome:Outcome -> @+count:U32 -> Tensor

def opened source · line 145 · raw

@result:Result<&1, &1, Pair(U32, String), File> -> @path:String -> @+shape:List<&2, U32> -> @+count:U32 -> IO(Tensor)

def counted source · line 153 · raw

@count:Maybe<&2, U32> -> @+path:String -> @+shape:List<&2, U32> -> IO(Tensor)

def load source · line 162 · raw

@+path:String -> @+shape:List<&2, U32> -> IO(Tensor)

The file holds the values of the shape in row-major order and nothing else.

def codes source · line 171 · raw

@text:String -> List<&2, U32>

def magic source · line 176 · raw

List<&2, U32>

def starts source · line 179 · raw

@pattern:List<&2, U32> -> @bytes:List<&2, U32> -> Bool

def at source · line 185 · raw

@bytes:List<&2, U32> -> @index:Nat -> U32

def contains source · line 191 · raw

@text:List<&2, U32> -> @+pattern:List<&2, U32> -> Bool

def after source · line 197 · raw

@text:List<&2, U32> -> @+pattern:List<&2, U32> -> Maybe<&2, List<&2, U32>>

The text after the first occurrence of the pattern.

def pushed source · line 203 · raw

@digits:Bool -> @current:U32 -> @shape:List<&2, U32> -> List<&2, U32>

def closed_shape source · line 208 · raw

@valid:Bool -> @shape:List<&2, U32> -> Maybe<&2, List<&2, U32>>

def extents source · line 214 · raw

@text:List<&2, U32> -> @+current:U32 -> @digits:Bool -> @valid:Bool -> @shape:List<&2, U32> -> Maybe<&2, List<&2, U32>>

Extents up to the closing parenthesis: digits, commas and spaces.

def shaped source · line 224 · raw

@shape:Maybe<&2, List<&2, U32>> -> @data:List<&2, U32> -> Maybe<&2, Header>

def typed source · line 229 · raw

@supported:Bool -> @text:Maybe<&2, List<&2, U32>> -> @data:List<&2, U32> -> Maybe<&2, Header>

def described source · line 235 · raw

@+text:List<&2, U32> -> @+length:Nat -> @data:List<&2, U32> -> Maybe<&2, Header>

def region source · line 239 · raw

@+bytes:List<&2, U32> -> @+length:Nat -> Maybe<&2, Header>

def versioned source · line 243 · raw

@major:U32 -> @+bytes:List<&2, U32> -> Maybe<&2, Header>

Version 1 has a two-byte header length; later versions have four bytes.

def marked source · line 248 · raw

@valid:Bool -> @+bytes:List<&2, U32> -> Maybe<&2, Header>

def header source · line 253 · raw

@+bytes:List<&2, U32> -> Maybe<&2, Header>

def refused source · line 256 · raw

@file:File -> @tensor:Tensor -> IO(Tensor)

def npy_counted source · line 261 · raw

@count:Maybe<&2, U32> -> @+shape:List<&2, U32> -> @data:List<&2, U32> -> @file:File -> @+path:String -> IO(Tensor)

def npy_parsed source · line 269 · raw

@parsed:Maybe<&2, Header> -> @file:File -> @+path:String -> IO(Tensor)

def npy_received source · line 274 · raw

@result:Result<&1, &1, Pair(U32, String), List<&2, U32>> -> @file:File -> @+path:String -> IO(Tensor)

def npy_headed source · line 279 · raw

@read:Read -> @+path:String -> IO(Tensor)

def npy_opened source · line 283 · raw

@result:Result<&1, &1, Pair(U32, String), File> -> @+path:String -> IO(Tensor)

def load_npy source · line 292 · raw

@+path:String -> IO(Tensor)

An FP32 array saved by NumPy, with its shape.

def bytes source · line 303 · raw

@+word:U32 -> @rest:List<&2, U32> -> List<&2, U32>

def encode source · line 306 · raw

@count:Nat -> @values:List<&2, F32> -> List<&2, U32>

def read_chunk source · line 313 · raw

@count:Nat -> @+offset:U32 -> @storage:Array<F32> -> @values:List<&2, F32> -> Pair(Array<F32>, List<&2, F32>)

The cursor retains the array; only the current chunk becomes a list.

def pending source · line 320 · raw

@remaining:Nat -> @file:File -> @storage:Array<F32> -> @offset:U32 -> Writing

def continued source · line 325 · raw

@result:Result<&1, &1, Pair(U32, String), Unit> -> @file:File -> @storage:Array<F32> -> @remaining:Nat -> @offset:U32 -> Writing

def after_write source · line 330 · raw

@written:Written -> @storage:Array<F32> -> @remaining:Nat -> @offset:U32 -> Writing

def write_chunk source · line 334 · raw

@file:File -> @read:Pair(Array<F32>, List<&2, F32>) -> @+count:Nat -> @remaining:Nat -> @offset:U32 -> IO(Writing)

def finished source · line 340 · raw

@file:File -> @complete:Bool -> IO(Bool)

def write_all source · line 345 · raw

@fuel:Nat -> @writing:Writing -> IO(Bool)

def unsaved source · line 356 · raw

@pair:Pair(Tensor, List<&2, U32>) -> @operation:String -> @path:String -> @detail:String -> Tensor

def written source · line 360 · raw

@complete:Bool -> @operation:String -> @path:String -> @tensor:Tensor -> Tensor

def created source · line 366 · raw

@result:Result<&1, &1, Pair(U32, String), File> -> @operation:String -> @+path:String -> @tensor:Tensor -> @prefix:List<&2, U32> -> @storage:Array<F32> -> @+length:Nat -> IO(Tensor)

The prefix is written first; a raw file has none.

def saving source · line 375 · raw

@operation:String -> @+path:String -> @tensor:Tensor -> @prefix:List<&2, U32> -> @buffer:Result<&2, &1, 0x9be3b13bf249759bc81e1958bcd1a4c0/tensor.Problem, 0x9be3b13bf249759bc81e1958bcd1a4c0/storage_buffer.Buffer<F32>> -> IO(Tensor)

def save source · line 385 · raw

@+path:String -> @tensor:Tensor -> IO(Tensor)

Writes the values in row-major order and returns the tensor. A failed tensor writes nothing and keeps its problem.

def joined source · line 388 · raw

@shape:List<&2, U32> -> String

def tuple source · line 395 · raw

@shape:List<&2, U32> -> String

NumPy writes (), (3,) and (2, 3).

def spaces source · line 401 · raw

@count:Nat -> List<&2, U32>

def framed source · line 407 · raw

@+text:List<&2, U32> -> @+length:U32 -> List<&2, U32>

Version 1: the header ends in a newline and the data starts at a multiple of 64 bytes.

def npy_header source · line 412 · raw

@shape:List<&2, U32> -> List<&2, U32>

def npy_saving source · line 416 · raw

@+path:String -> @pair:Pair(Tensor, Tensor) -> @+shape:List<&2, U32> -> IO(Tensor)

def save_npy source · line 421 · raw

@+path:String -> @tensor:Tensor -> IO(Tensor)

Writes a .npy file that NumPy loads with the tensor's shape.