Mamba Operators¶

本页暂无中文版。以下为英文原文。

Every op on this page is used the same way: construct it once, then call it. The constructor takes what the kernel is compiled with; the call takes the tensors. Both are documented under each op — __init__ and forward, where forward is what runs when you call op(...).

Full forward¶

tileops.ops.mamba.mamba2_fwd.Mamba2FwdOp ¶

Mamba-2 State-Space Dual (SSD) full forward pass operator.

Combines DaCumsum → SSDChunkState → SSDStatePassing → SSDChunkScan into a single callable whose interface matches mamba_chunk_scan_combined from the official mamba_ssm library.

init ¶

__init__(
    chunk_size=256,
    dt_softplus=True,
    tune=False,
    kernel_map=None,
)

Build the op. Shapes and dtype are taken from the first call.

Parameters:

chunk_size (int, default: 256 ) –

Tokens per chunk (default 256).
dt_softplus (bool, default: True ) –

Apply softplus to (dt + dt_bias) before use.
tune (bool, default: False ) –

Whether to autotune tile configs on construction.

forward ¶

forward(
    x,
    dt,
    A,
    B,
    C,
    dt_bias=None,
    initial_states=None,
)

Run the full Mamba-2 SSD forward pass.

Parameters:

x (Tensor) –

(batch, seqlen, n_heads, d_head) dtype
dt (Tensor) –

(batch, seqlen, n_heads) float32
A (Tensor) –

(n_heads,) float32 (log-space, ≤ 0)
B (Tensor) –

(batch, seqlen, n_groups, d_state) dtype
C (Tensor) –

(batch, seqlen, n_groups, d_state) dtype
dt_bias (Optional[Tensor], default: None ) –

(n_heads,) float32, optional
initial_states (Optional[Tensor], default: None ) –

(batch, n_heads, d_head, d_state) float32, optional

Returns:

y ( Tensor ) –

(batch, seqlen, n_heads, d_head) float32
final_states ( Optional[Tensor] ) –

(batch, n_heads, d_head, d_state) float32, or None

SSD stages¶

tileops.ops.mamba.da_cumsum.DaCumsumFwdOp ¶

Mamba-2 dA_cumsum forward operator.

Applies optional per-head bias, optional softplus activation, and clamping to raw dt values, then computes the chunk-local inclusive prefix sum of dA = dt * A.

Note: dt_out is cast to the target dtype for storage efficiency, but dA_cumsum is computed from the fp32 dt values before casting, ensuring numerical precision.

init ¶

__init__(
    chunk_len,
    dtype=torch.float32,
    dt_softplus=False,
    dt_min=0.0,
    dt_max=float(
        "inf"
    ),
    tune=False,
    kernel_map=None,
)

Build the op. Shapes and dtype are taken from the first call.

Parameters:

chunk_len (int) –

Tokens per chunk.
dt_softplus (bool, default: False ) –

Whether to apply softplus (with bypass for dt > 20) to dt.
dt_min (float, default: 0.0 ) –

Lower clamp bound applied after bias and softplus.
dt_max (float, default: float('inf') ) –

Upper clamp bound applied after bias and softplus.
tune (bool, default: False ) –

Whether to autotune tile config on construction.

forward ¶

forward(
    dt,
    A,
    dt_bias=None,
)

Run the dA_cumsum forward pass.

Parameters:

dt (Tensor) –

(batch, seq_len, n_heads) float32 — raw dt values.
A (Tensor) –

(n_heads,) float32 — SSM decay parameters.
dt_bias (Optional[Tensor], default: None ) –

(n_heads,) float32, optional — per-head dt bias.

Returns:

dt_out ( Tensor ) –

(batch, n_heads, num_chunks, chunk_len) dtype — processed dt in target dtype.
dA_cumsum ( Tensor ) –

(batch, n_heads, num_chunks, chunk_len) float32 — inclusive prefix sum of dA = dt_val * A, computed from fp32 dt_val before casting dt_out.

tileops.ops.mamba.cb_producer.CBProducerFwdOp ¶

CB (C@B) matrix producer operator.

Computes cb[b,c,g,l,s] = sum_n C[b,c,g,l,n] * B[b,c,g,s,n] with causal masking (cb[l,s] = 0 if s > l).

init ¶

__init__(
    batch,
    num_chunks,
    n_groups,
    chunk_len,
    d_state,
    tune=False,
    kernel_map=None,
)

Build the op. Shapes and dtype are taken from the first call.

Parameters:

batch (int) –

Batch size
num_chunks (int) –

Number of chunks
n_groups (int) –

Number of groups
chunk_len (int) –

Chunk length (Q)
d_state (int) –

State dimension (N)
tune (bool, default: False ) –

Whether to autotune
kernel_map (Optional[Dict[str, Kernel]], default: None ) –

Optional pre-initialized kernels

forward ¶

forward(
    C_mat,
    B_mat,
)

Parameters:

C_mat (Tensor) –

[B, S, G, N] dtype (contiguous)
B_mat (Tensor) –

[B, S, G, N] dtype (contiguous)

Returns:

cb ( Tensor ) –

[B, C, G, Q, Q] dtype

tileops.ops.mamba.ssd_chunk_state.SSDChunkStateFwdOp ¶

Mamba-2 State-Space Dual (SSD) chunk state forward operator.

Computes the chunk-end State Space Model (SSM) state for each chunk:

out[b, c, h, p, n] = sum_{l=0}^{Q-1} x[b, cQ+l, h, p] * B[b, cQ+l, g(h), n] * exp(dA_cumsum[b,h,c,Q-1] - dA_cumsum[b,h,c,l]) * dt[b, h, c, l] * (1 if seq_idx is None else (seq_idx[b,cQ+Q-1] >= 0 and seq_idx[b,cQ+l] == seq_idx[b,c*Q+Q-1]))

init ¶

__init__(
    tune=False,
    kernel_map=None,
)

Build the op. Shapes and dtype are taken from the first call.

Parameters:

tune (bool, default: False ) –

Whether to autotune tile config on construction.

forward ¶

forward(
    x,
    Bmat,
    dt,
    dA_cumsum,
    seq_idx=None,
)

Run the SSD chunk state forward pass.

Parameters:

x (Tensor) –

(batch, seq_len, n_heads, d_head)
Bmat (Tensor) –

(batch, seq_len, n_groups, d_state)
dt (Tensor) –

(batch, n_heads, num_chunks, chunk_len) float32
dA_cumsum (Tensor) –

(batch, n_heads, num_chunks, chunk_len) float32
seq_idx (Optional[Tensor], default: None ) –

(batch, seq_len) int32, optional

Returns:

out ( Tensor ) –

(batch, num_chunks, n_heads, d_head, d_state) float32

tileops.ops.mamba.ssd_state_passing.SSDStatePassingFwdOp ¶

Mamba-2 State-Space Dual (SSD) state passing forward operator.

Performs the inter-chunk recurrent scan:

s_c[m] = exp(dA_chunk_cumsum[b, h, c]) * s_{c-1}[m] + states[b, c, h, m]

with s_{-1} = initial_states, or 0 when it is not passed.

init ¶

__init__(
    tune=False,
    kernel_map=None,
)

Build the op. Shapes and dtype are taken from the first call.

Parameters:

tune (bool, default: False ) –

Whether to autotune tile config on construction.

forward ¶

forward(
    states,
    dA_chunk_cumsum,
    initial_states=None,
)

Run the SSD state passing forward pass.

Parameters:

states (Tensor) –

(batch, num_chunks, n_heads, d_state)
dA_chunk_cumsum (Tensor) –

(batch, n_heads, num_chunks) float32
initial_states (Optional[Tensor], default: None ) –

(batch, n_heads, d_state) float32

Returns:

out ( Tensor ) –

(batch, num_chunks, n_heads, d_state) float32
final_states ( Tensor ) –

(batch, n_heads, d_state) float32

tileops.ops.mamba.ssd_chunk_scan.SSDChunkScanFwdOp ¶

Mamba-2 State-Space Dual (SSD) fused chunk output operator.

Fuses the history (prev_states) contribution and intra-chunk causal decay into a single pass, computing:

out[l, p] = exp(dA_cumsum[l]) * (C[l] @ prev_states) + sum_{s <= l} cb[l, s] * exp(dA_cumsum[l] - dA_cumsum[s]) * dt[s] * x[s, p]

init ¶

__init__(
    tune=False,
    kernel_map=None,
)

Build the op. Shapes and dtype are taken from the first call.

Parameters:

tune (bool, default: False ) –

Whether to autotune tile config on construction.

forward ¶

forward(
    x,
    cb,
    dA_cumsum,
    C,
    prev_states,
    dt,
)

Run the fused SSD chunk scan.

Parameters:

x (Tensor) –

(batch, seqlen, n_heads, d_head) dtype
cb (Tensor) –

(batch, num_chunks, n_groups, chunk_len, chunk_len) dtype
dA_cumsum (Tensor) –

(batch, n_heads, num_chunks, chunk_len) float32
C (Tensor) –

(batch, seqlen, n_groups, d_state) dtype
prev_states (Tensor) –

(batch, num_chunks, n_heads, d_head, d_state) dtype
dt (Tensor) –

(batch, n_heads, num_chunks, chunk_len) dtype

Returns:

out ( Tensor ) –

(batch, seqlen, n_heads, d_head) float32

Decode¶

tileops.ops.mamba.ssd_decode.SSDDecodeFwdOp ¶

Mamba-2 State-Space Dual (SSD) recurrent decode (step) operator.

Performs a single decode step of the Mamba-2 State Space Model (SSM) core: updates the recurrent state in-place and returns the output y for the current token:

g = h // (n_heads // n_groups) dA[b, h, p, n] = exp(dt[b, h, p] * A[h, p, n]) state[b,h,p,n] <- dA[b,h,p,n] * state[b,h,p,n] + dt[b,h,p] * B_in[b,g,n] * x[b,h,p] y_out[b, h, p] = sum_n state[b, h, p, n] * C_in[b, g, n]

The skip connection (D * x) and output gate (z * silu) are not fused here and must be applied by the caller if needed.

init ¶

__init__(
    tune=False,
    kernel_map=None,
)

Build the op. Shapes and dtype are taken from the first call.

Parameters:

tune (bool, default: False ) –

Whether to autotune tile config on construction.

forward ¶

forward(
    A,
    dt,
    x,
    B_in,
    C_in,
    state,
)

Run a single Mamba-2 decode step.

Parameters:

A (Tensor) –

(n_heads, d_head, d_state) float32 -- SSM decay parameter (A <= 0)
dt (Tensor) –

(batch, n_heads, d_head) float32 -- discretization step (post-softplus)
x (Tensor) –

(batch, n_heads, d_head) dtype -- input features per head
B_in (Tensor) –

(batch, n_groups, d_state) dtype -- SSM B matrix (per group)
C_in (Tensor) –

(batch, n_groups, d_state) dtype -- SSM C matrix (per group)
state (Tensor) –

(batch, n_heads, d_head, d_state) float32 -- recurrent state (mutated in-place)

Returns:

y_out ( Tensor ) –

(batch, n_heads, d_head) float32

Mamba Operators¶

Full forward¶

tileops.ops.mamba.mamba2_fwd.Mamba2FwdOp ¶

__init__ ¶

forward ¶

SSD stages¶

tileops.ops.mamba.da_cumsum.DaCumsumFwdOp ¶

__init__ ¶

forward ¶

tileops.ops.mamba.cb_producer.CBProducerFwdOp ¶

__init__ ¶

forward ¶

tileops.ops.mamba.ssd_chunk_state.SSDChunkStateFwdOp ¶

__init__ ¶

forward ¶

tileops.ops.mamba.ssd_state_passing.SSDStatePassingFwdOp ¶

__init__ ¶

forward ¶

tileops.ops.mamba.ssd_chunk_scan.SSDChunkScanFwdOp ¶

__init__ ¶

forward ¶

Decode¶

tileops.ops.mamba.ssd_decode.SSDDecodeFwdOp ¶

__init__ ¶

forward ¶

init ¶

init ¶

init ¶

init ¶

init ¶

init ¶

init ¶