PROJECT 02 / MODEL ARCHITECTURE
A sequence model must decide
what to carry forward.
EILER (Eigen-Integrated Linear Evolution Recurrence) is an experimental sequence-model architecture exploring alternating complex-valued state-space modules and decoupled delta-memory blocks. The hypothesis is that evolving state and selective memory updates may serve different roles. Whether this combination improves the efficiency or quality of sequence modeling remains to be tested against controlled baselines.
A sequence model needs a mechanism for turning past inputs into information that can influence later outputs. The design trade-off is between preserving detail, maintaining useful long-range signals, and keeping computation and memory manageable.
Attention mechanisms can directly compare positions across a context, while recurrent and state-space approaches propagate a compact state as inputs arrive. Compact state can make continued processing attractive, but compression may discard a detail that becomes important later. No single approach is best for every task or budget.
EILER explores a hybrid design: a complex-valued state-space path for evolving dynamics, alongside a separate delta-memory path for selective writes or revisions. The architecture adds complexity, so it earns its place only if ablations and baseline comparisons show a meaningful benefit.
Separating evolving state from explicit memory updates may make retention, compute cost, and information routing easier to inspect and compare.
This diagram is a conceptual overview, not a full mathematical specification or evidence that the components outperform an alternative.
The state-space path is intended to capture sequence dynamics in a compact evolving representation. Complex-valued parameterizations may be useful for some dynamics, but expressiveness in notation does not guarantee better learning. The parameterization, stability, optimization, and task all matter.
The delta-memory path provides a separate update route for a stored memory representation. The design questions include when writes occur, how new information interacts with existing contents, what gets overwritten, and how memory and runtime scale with the sequence and hidden dimensions.
Alternating components changes how information moves through the model and adds potential failure points: gradient flow, initialization, normalization, routing, and extra compute. The combination should be retained only if controlled ablations justify the added complexity.
Architecture research requires matched conditions. EILER should be compared with credible reference models using the same data, tokenization, training budget, and evaluation protocol. Quality must be reported alongside parameters, memory, throughput, latency, convergence, and implementation cost.
This page does not claim a performance advantage. Results should be published only with enough detail to reproduce the training run, reconstruct the baseline, and understand both positive and negative outcomes.
Document tensor shapes, state transitions, write rules, initialization, normalization, gradient flow, and computational complexity.
Compare to well-tuned baselines; record loss, throughput, peak memory, convergence, generalization, and seed-to-seed variance.
Use tasks with controlled dependencies and distractors to test whether remembered information changes the right output, not just whether loss decreases.
RESEARCH IN PROGRESS
EILER is an experimental line of work. As implementations and results mature, this page should grow to include reproducible methods and findings.