math_activation.bend checks

raw source on the hub · import stelliferous@0.0.2.0/math_activation.bend as Math_activation

FP32 activations, imported by callers as Act: Act.silu() is a value for F.conv2d or F.activate, Act.silu_value the scalar function. Each built-in states its operations per element for the planners, counted from the emitted C and assembly: arithmetic, comparisons, selects and word operations; addressing, moves and loop control are excluded. The sigmoid, SiLU, tanh and GELU run eight lanes at a time, so their counts are the instructions per eight lanes divided by eight; the others are scalar counts. The sigmoid and tanh use ordered FP32 arithmetic and word operations (lib/math_fp32.bend). Sigmoid and SiLU scale their negative tails before the final underflow; accuracy is measured against MPFR rather than a global rounding guarantee.

2 imports

import Base
import ./math_fp32.bend as Fp32

Types

type Activation source · line 15 · raw

Data

Custom is a caller-supplied function; only its operation count lives here.

Identity
```
Activation
```
ReLU
```
Activation
```
ReLU6
```
Activation
```
LeakyReLU
```
@slope:F32 -> Activation
```
Sigmoid
```
Activation
```
SiLU
```
Activation
```
Tanh
```
Activation
```
HardSigmoid
```
Activation
```
HardSwish
```
Activation
```
GELUTanh
```
Activation
```
Custom
```
@operations:F32 -> Activation
```

Definitions

def identity_value source · line 28 · raw

@value:F32 -> F32

def relu_value source · line 31 · raw

@value:F32 -> F32

def relu6_value source · line 34 · raw

@value:F32 -> F32

def leaky source · line 37 · raw

@slope:F32 -> @+value:F32 -> F32

def leaky_relu_value source · line 41 · raw

@value:F32 -> @slope:F32 -> F32

x for x >= 0, slope * x otherwise.

def sigmoid_value source · line 44 · raw

@value:F32 -> F32

def silu_value source · line 47 · raw

@value:F32 -> F32

def tanh_value source · line 50 · raw

@value:F32 -> F32

def hard_sigmoid_value source · line 54 · raw

@value:F32 -> F32

PyTorch's order: relu6(x + 3) / 6 and x * relu6(x + 3) / 6.

def swished source · line 57 · raw

@+value:F32 -> F32

def hard_swish_value source · line 60 · raw

@value:F32 -> F32

def gelu_tanh_of source · line 64 · raw

@+value:F32 -> F32

PyTorch's gelu(approximate='tanh'): 0.5 x (1 + tanh(sqrt(2/pi) (x + 0.044715 x^3))).

def gelu_tanh_value source · line 68 · raw

@value:F32 -> F32

def identity source · line 71 · raw

Activation

def relu source · line 74 · raw

Activation

def relu6 source · line 77 · raw

Activation

def leaky_relu source · line 80 · raw

@slope:F32 -> Activation

def sigmoid source · line 83 · raw

Activation

def silu source · line 86 · raw

Activation

def tanh source · line 89 · raw

Activation

def hard_sigmoid source · line 92 · raw

Activation

def hard_swish source · line 95 · raw

Activation

def gelu_tanh source · line 98 · raw

Activation

def is_identity source · line 102 · raw

@activation:Activation -> Bool

The identity needs no pass at all.

def operations source · line 108 · raw

@activation:Activation -> F32

Operations per element, excluding the pass that applies them.

def unknown_operations source · line 134 · raw

F32

A custom function (F.conv2d_custom, F.apply) states its scalar operations per element: one per arithmetic, comparison or select; F32.exp about 37 and F32.tanh about 82 with their conversions. The count only steers parallel planning; results never depend on it. When unsure, use this conservative count: a call into the C math library and a few operations around it.