Following many recent conferences and conversations, I came away with one idea reinforced over any other: the future of AI performance is as much about what happens between processers as what happens on them. 

Memory has to store and supply an ever-increasing, enormous amount of data while interconnects have to move it between ever-more distributed resources. XPUs and accelerators spend precious time preparing, transforming and coordinating that data before the next useful operation can take place.

Rethinking where compute happens

Across the ecosystem, architectures that don’t just deliver more compute, but rethink where that compute happens are gaining traction. Innovators and designers at the forefront of AI are now exploring whether we can turn the network itself into active compute infrastructure. 

From the vantage point of memory, processing-in-memory is targeting decode heavy premium workloads to reduce data movement in pursuit of speed, energy and cost gains. While memory pooling, sharing and dynamic allocation make capacity more available. That capacity can follow workloads and be more effective in serving larger models and supporting longer contexts. 

Disaggregated memory can improve utilization and enable larger models or contexts, but physically separating memory from the accelerator introduces a transport path with different latency and bandwidth characteristics from local HBM. 

If memory can become part of compute, why not the data path? 

Moving data faster and smarter

The direct answer to overcoming the inefficiencies of data movement-bound workloads has been faster interconnects, and optics has become fundamental in this. 

But if an accelerator requests data from another memory tier, receives it, then uses valuable processing resources to convert, reshape, or aggregate that data before it can do the work we actually need the accelerator to perform, there is still an architectural cost. 

What if the data could arrive more useful than when it left?  

Photonic Compute-in-Transit introduces programmable mathematical operations into the optical path and power use that data is already taking between parts of the system.  

We aren’t putting the entire workload into the network, nor seeking to replace GPUs, CPUs or memory-side computation, but creating clever divisions of labor for ultimate system efficiency.  

Our work is currently evaluating these operation classes: 

• Precision conversion allows information to be stored or transported efficiently at low precision and converted towards the format required by the destination as it moves 

• Data shaping can prepare information for different processors in increasingly heterogeneous architectures 

• Compression and selection reduces the amount of information that ultimately has to enter memory or reach an XPU 

• Streaming linear operations create the possibility of performing useful mathematics on large datasets before the complete dataset reaches the accelerator 

With this, two potential efficiency gains come into focus: one is reducing the number of bytes that need to move, while the other is reducing the amount of work left for the XPU, even when the number of bytes does not change.  

For example, accumulation, scaling or statistics performed during transport may remove operations that would otherwise consume accelerator resources. The goal being to free that resource to improve latency, but it could equally improve concurrency or reduce energy per completed request. 

KV cache as a practical starting point

We’ll be exploring one iteration of Compute-in-Transit at the upcoming PECC Summit.  

Our demonstration uses an LLM KV cache workload – a particularly interesting example because modern inference increasingly depends on retaining and retrieving previously computed context. As conversations get longer and agentic workflows become more complex, the capacity and movement requirements of that cache grows too.  

In our use of an early photonic core, LMCache manages reusable cache objects outside the model-serving process, comparing two paths:  

• In the first, data is transported and a Hadamard transformation used within the cache codec runs in host software  

• In the photonic Compute-in-Transit path, supported Hadamard butterfly operations instead pass through the Optalysys link backend 

The rest of the workload remains where it belongs: inference, quantization, packing, normalization and metadata handling remain on the host.  

The isn’t that a Hadamard operation solves the KV cache problem on its own. It doesn’t, and in this codec the transform itself is not what creates the compressed representation. What’s interesting is what changes based on where that operation takes place. 

A task that would otherwise be executed by the host can become part of the movement of the cache data itself, which is particularly compelling when we consider what the rest of the industry is doing with memory.

A smarter division of labor

The future of AI infrastructure won’t hinge on finding one new technology that does everything, but on putting each operation in the place where it can be executed most efficiently.  

Sometimes that will be the XPU, increasingly it could be inside or close to memory and for certain operations, it will be within the data path itself.   

If we get that right, smarter data movement can do more than improve the network: it can help memory go further, free expensive XPUs to spend more of their time and power on the computation they are best at and it can give system architects much greater freedom over where memory and compute need to physically reside.  

That’s the opportunity I’m looking forward to exploring at PECC. 

---

Are you also heading to the PECC Summit? Book some time to connect with our team here →