Today’s AI systems consist of huge arrays of power-hungry processors and a myriad of connections to external memories and switches used to route the data around the system. The speed and efficiency at which the data can be moved is now a critical factor to determine how fast and efficient the entire system can run. 

To address this, innovators across the ecosystem are racing to develop new processors, accelerators, memory tiers, interconnects and specialist computing capabilities. No singular company, or singular technology, will define the future of AI: it will be heterogenous – combining all of these components and placing each part of the workload in the optimal location for the greatest system-level efficiency. 

This is where photonic Compute-in-Transit becomes a compelling proposition: introducing processing into the optical data path, enhancing part of the architecture that has conventionally been solely concerned with moving information. 

Why location matters in inference – prefill v decode 

There are two main stages to large language model (LLM) inference: prefill and decode. 

During prefill, the model processes the input prompt and builds up the intermediate information it will need to generate a response. This includes the key-value, or KV, cache. 

During decode, the answer is generated token-by-token. For each successive token, relevant information must be retrieved from the KV cache, operated on and updated before the process repeats. 

Decode is therefore highly dependent on how efficiently vast swathes of data can be supplied to the processor —-a driver in interconnect and memory architectures becoming such vital parts of AI infrastructure: processor capability alone is redundant if the wider system cannot keep it supplied. 

Copper connectivity is increasingly being supplemented or replaced by photonics because light provides a route to far greater bandwidth between memory, processors and other parts of the system. The industry is already investing heavily in making those optical paths faster, denser, and more energy efficient. 

If data is already moving optically between memory and a processor, or between various parts of a distributed compute system, can we perform useful operations on that data while it is moving, rather than waiting until it has reached another conventional compute stage? 

Computing between memory and processor 

Compute-in-Transit places programmable computation at the boundary between communication and processing: at the receive side of optical links, between memory and accelerators, between processing and potentially increasingly close to advanced packages. 

Today, transport and compute are largely treated as separate functions. Information is moved, received and reconstructed before the useful operation happens. 

With Compute-in-Transit, selected operations can become part of the movement event itself. 

This doesn’t mean replacing conventional XPUs with photonics systems. While photonics is naturally suited to particular classes of linear mathematical operations, electronics remains ideal for general-purpose control, nonlinear processing and many other workloads GPUs perform exceptionally well. 

Heterogenous architectures offer us the opportunity to explore better division of labor. 

Matrix-vector multiplication and Compute-in-Transit 

One area we are exploring closely is general matrix-vector multiplication, or GEMV, which is common in AI inference. Conceptually, a large matrix of information is multiplied by a much smaller vector to produce another vector. 

That structure is particularly interesting in a streaming architecture. 

Imagine that a large matrix is being brought from memory towards a processor: instead of transmitting the complete matrix and then asking the processor to multiply it by the vector, the vector can potentially be introduced at the interface and the multiplication carried out progressively as the matrix streams through the link. 

By the time the information reaches the other end, the system no longer needs to deliver the entire original matrix to the processor. It needs to deliver the result. 

For example, multiplying a 7,000 × 7,000 matrix by a 1 x 7,000-element vector produces 1 x 7,000-element vector. Moving the operation into the data path therefore has the potential to significantly change the quantity of information that ultimately needs to arrive at the accelerator, addressing latency and data-movement bottlenecks. 

We are currently modeling how those advantages translate into complete AI architectures and real workloads, creating value and efficiency gains once integrated into the wider system.  

Heterogeneity is changing how we develop systems 

Our work in Compute-in-Transit connects to a much wider change in AI architectures. 

Development in GPUs, in or near-memory computation, interconnects and specialist accelerators is booming. AI models and infrastructure are evolving too quickly to optimize a fundamental technology around one particular component, and future systems will be varying combinations of them all. 

But heterogenous architectures create another challenge: if computation can happen in several places, those components can’t be designed entirely in isolation. 

Where operations happen will impact the representation and precision of the data travelling through the system. It will affect memory placement, interfaces, protocols, buffering and orchestration, and how software decides which resource should perform each operation. 

Photonic compute capability can’t achieve its full potential in isolation from the accelerator it feeds, the memory supplying the data, the optical fabric carrying it or the software managing it. 

The same applies in reverse: an accelerator, memory architecture or interconnect can create greater system-level value when the rest of the machine is designed to exploit what it does best. 

Designing around where compute happens 

This makes collaboration as important as ever. 

We’re introducing Compute-in-Transit into existing AI architectures and identifying workloads where it creates meaningful benefits today. 

But new computing capabilities have historically had their biggest impact when system architects stop treating them as an add-on and begin designing around what they make possible. 

How could future systems benefit if different forms of computation were available through the architecture, and workloads could place each operation where it made most sense?  

Photonics has already changed how we think about moving information through large computing systems. 

Compute-in-Transit is about taking the next step and making that movement computationally useful. Making it – and many of the emerging solutions to AI’s energy, data-movement and memory bottlenecks – valuable at a system level requires collaboration across the stack to begin designing those systems now. 

---

At Optalysys we’re pioneering the architectural revolution enabling photonic Compute-in-Transit. Get in touch with us to find out how we can bring efficiency gains to your use case →