AI’s Stealth Surges

Disruption can be deceptive

From a consumer perspective, AI progress can look incremental. Beneath the surface, the shifts are anything but. The real advances have been accumulating in specialised domains, largely out of public view, and they are about to surface.

AI models now handle PhD-level questions and are accelerating research in fields like materials science. AI’s ability to diagnose and fix software bugs has jumped from around 5% to over 70% on standard benchmarks. Google now attributes over 25% of its newly generated code to AI systems.

AI systems keep outperforming human experts in specialised tasks while remaining cheaper. New research suggests this will continue. Models fine-tuned on weaker, less expensive synthetic data often surpass those trained on costlier, higher-quality datasets. Simpler models that generate a broader range of solutions frequently perform better overall, because they cover more unique scenarios. Multiple valid solutions to the same problem also let models develop richer reasoning abilities.

Simpler models may initially produce “false positives” (correct outcomes despite flawed intermediate steps), but downstream models still learn robust reasoning from them and end up matching the logic consistency of models trained on expensive, high-quality data.

The upshot: modest models, given the right scaffolding and aggregation, can leverage synthetic data to punch well above their weight.

Google’s “Titans” architecture is another step-change: it supports radically larger models with far better long-term contextual retention.

Traditional Transformer models, which attend to every token in a fixed-length “context window,” slow down and lose focus as they handle more extensive inputs. Titans addresses this limitation with a hybrid approach: immediate context processing handled by traditional attention mechanisms (short-term memory), complemented by a novel “neural memory” module for retaining and dynamically updating historical context (long-term memory).

The neural memory module adapts continuously during inference through gradient-based updates, allowing it to respond dynamically to new inputs. An adaptive update mechanism selectively incorporates new data, guided by a “forget gate” that discards obsolete information. This system employs a sophisticated model of “momentary surprise” (the degree of deviation from the current model’s understanding) and “past surprise,” representing a decaying record of past unexpected events. Inspired by human memory, this approach prioritises retention of unexpected or novel inputs.

By cleanly separating short-term attention from long-term memory, Titans makes it feasible to scale efficiently beyond millions of tokens. Expect a much larger generation of models to adopt similar techniques before long.

These stealthy surges are like bubbles rippling on the ocean before a whale blasts through the surface.