Search Results for author:
Found 5 papers, 5 papers with code
Softmax Attention with Constant Cost per Token
We propose a simple modification to the conventional attention mechanism applied by Transformers: Instead of quantifying pairwise query-key similarity with scaled dot-products, we quantify it with the logarithms of scaled dot-products of exponentials.
Efficient Parallelization of a Ubiquitous Sequential Computation
We find a succinct expression for computing the sequence $x_t = a_t x_{t-1} + b_t$ in parallel with two prefix sums, given $t = (1, 2, \dots, n)$, $a_t \in \mathbb{R}^n$, $b_t \in \mathbb{R}^n$, and initial value $x_0 \in \mathbb{R}$.
An Algorithm for Routing Vectors in Sequences
We propose a routing algorithm that takes a sequence of vectors and computes a new sequence with specified length and vector size.
Tree Methods for Hierarchical Classification in Parallel
We propose methods that enable efficient hierarchical classification in parallel.
An Algorithm for Routing Capsules in All Domains
Building on recent work on capsule networks, we propose a new, general-purpose form of "routing by agreement" that activates output capsules in a layer as a function of their net benefit to use and net cost to ignore input capsules from earlier layers.
Ranked #1 on
Image Classification
on smallNORB
