deazy

I was exploring this conceptual architecture for neural networks (transformers/attention) based long-context models, its conceptual but grounded in sound existing research and architecture implementations on specialized hardware like gpu's and tpu's.

Can a we scale up independent shards of (mini) contexts, i.e Sub-global attention blocks or "sub-context experts" that can operate somewhat independently with global composition into a larger global attention as a paradigm for handling extremely long contexts. Context shared, distributed and sharded across chips, that can act as Independent shards of (mini) Contexts.

This could possibly (speculating here) make attention based context sub-quadratic. Its possible (again speculating here) google might have used something like this for having such long context windows.

Evidence points to this: Google's pioneering MoE research (Shazeer, GShard, Switch), advanced TPUs (v4/v5p/Ironwood) with massive HBM & high-bandwidth 3D Torus/OCS Inter-Chip Interconnect (ICI) enabling essential distribution (MoE experts, sequence parallelism like Ring Attention), and TPU pod VRAM capacities aligning with 10M token context needs. Google's Pathways & system optimizations further support possibility of such a distributed, concurrent model.

Share your thoughts on this if its possible, feasible or why it might not work.

Gemini's Long Context MoE Architecture (Hypothesized):

Sharing how I think (hypothesis) Gemini models achieve their 1-10 Million long context window. With details to clues to support the same.

Ensemble of Expert (EoE) or Mesh of Expert (MeoE) with common/shared long (1-10M) context window

Gemini's 1M+ token MoE likely uses "instances" (active expert sets/TPU shards) sharing a common distributed context; individual active expert groups then use relevant "parts" of this vast context for generation. This allows concurrent, independent requests via distinct system "partitions."

The context is sharded and managed across numerous interconnected TPUs within a pod.

For any given input, only a sparse set of specialized "expert" subnetworks (a "dynamic pathway") within the total model are activated, based on complexity and context required.

The overall MoE model can handle multiple, concurrent user requests simultaneously.

Each request, with its specific input and context, will trigger its own distinct and isolated pathway of active experts.

Shared context that can act as independent shards of (mini) contexts.

The massively distributed Mixture of Experts (MoE) architecture, across TPUs in a single pod, have its the long context sharded and managed via parallelism, and with ability to handle concurrent requests by part of that context window and independent expert pathways across a large TPU pod, also it can use the entire context window for a single request if required.

Evidence points to this: Google's pioneering MoE research (Shazeer, GShard, Switch), advanced TPUs (v4/v5p/Ironwood) with massive HBM & high-bandwidth 3D Torus/OCS Inter-Chip Interconnect (ICI) enabling essential distribution (MoE experts, sequence parallelism like Ring Attention), and TPU pod VRAM capacities aligning with 10M token context needs. Google's Pathways & system optimizations further support this distributed, concurrent model.

og x thread: https://x.com/ditpoo/status/1923966380854157434