How AI Inferencing is Driving New Networking Requirements

BrainBlog for Arrcus by Jason Bloomberg

At the center of the explosion of interest in AI are the models – in particular, the large language models (LLMs) that typically drive generative AI but also specialized language models (SLMs) of various sizes.

Models are like empty vessels until someone trains them – a time-consuming, resource-intensive process that takes place before putting them into use.

Training alone, however, doesn’t make a model useful. AI models deliver value via inferencing.

Inferencing refers to the process of applying a fully trained model to new data – data that drive whatever output the business requires, including decisions, predictions, or agentic behavior.

While training takes place beforehand, inferencing occurs and wherever people require results – often in real-time.

As a result, the infrastructure requirements for training and inferencing are quite different, especially for the network that must support the real-time nature of inferencing at scale.

The inferencing networking challenge

Inferencing use cases are numerous and varied across industries for both businesses and consumers. Here are some examples:

• For both consumers and business users, inferencing provides facial recognition, smart assistants, and translation on smart phones
• In retail, inferencing provides dynamic pricing and real-time customer behavior analytics, either online or at the point of sale
• In financial services, inferencing is at the heart of fraud detection, risk scoring, and credit scoring, often in real-time at the point of use
• Healthcare is also a primary user of AI inferencing for interpreting x-rays and other scans, patient monitoring, and point-of-care diagnostics – once again, at the point of care
• Public safety leverages AI inferencing as well, across emergency response, law enforcement, urban safety and transportation
• In service provider networks, it enables real-time decisions to optimize network performance, predict congestion, and outage minimization via predictive maintenance.

This variety and diversity of inferencing use cases raises the bar for AI’s supporting infrastructure – especially for the network.

In fact, inferencing is taking place across all industries for a wide range of use cases – where the inferencing itself runs on edge devices, cloud endpoints, and many other locations in corporate data centers, clouds, and around the world.

Understanding the networking needs of inferencing

While it’s certainly possible to perform inferencing in the cloud or a corporate data center, many organizations run their inferencing routines close to the point of use to reduce latency, bandwidth requirements, and cloud costs.

In fact, the primary networking considerations for inferencing are scale and latency as well as high availability and reliability, especially in edge scenarios.

Take, for example, video inferencing, either for surveillance or manufacturing quality purposes. Such inferencing may take place within the camera itself or in a nearby edge gateway to reduce latency, thus supporting real-time requirements for the technology.

Unsurprisingly, the infrastructure requirements for inferencing are different than for model training. While training has massive scale requirements for high-bandwidth, high-throughput networks in data centers, inferencing relies more heavily on horizontal scaling across edge and other endpoint devices.

Such scaling depends more on intelligent distribution of AI workloads than simple throughput, requiring orchestration and deployment pipelines to support frequent model updates.

While latency is a minor concern for model training, it is of utmost importance for inferencing across a range of common applications – not only for real-time video processing, but also for voice recognition and generation as well as all manner of autonomous systems from factory robots to self-driving cars.

In all of these situations, the network must minimize round-trip delays and support real-time data transmission whenever possible.

How Arrcus ACE-AI tackles inferencing networking challenges

Arrcus designed its ACE-AI networking platform to serve as a high-performance fabric for distributed AI workloads across data centers, clouds, and the edge.

At the core of ACE-AI is ArcOS, Arrcus’s software-based networking operating system. ArcOS offers hardware-agnostic programmable networking that provides lossless, low-latency Ethernet connectivity for both model training and inferencing tasks.

In addition, ArcOS’s Open APIs enable integration with various orchestration platforms for automation and simplification of operations.

For inferencing workloads, ACE-AI provides advanced capabilities that enable it to offer lossless and congestion free connectivity, including:

• Remote Direct Memory Access over Converged Ethernet v2 (RoCEv2), which lowers latency by offloading network packet processing from the CPU
• Priority Flow Control (PFC), which selectively pauses traffic to avoid packet loss during periods of potential network congestion
• Data Processing Unit (DPU) and SmartNIC integration that offloads networking for high performance, low latency, and secure connectivity at the Edge
• An open, software-defined architecture that provides flexibility and programmability for inferencing networks
• Additional technologies that provide visibility into network congestion as well as adaptive routing and dynamic load balancing that work together to reroute traffic around congestion to avoid any traffic loss

All these capabilities enable ACE-AI to provide extraordinarily low latency and high bandwidth that support even the most rigorous inferencing scenarios.

Complementing ACE-AI is Arrcus’s ArcIQ platform, which provides real-time observability into network health as well as performance and other metrics that enable operators to manage their networks with proactive incident management and troubleshooting from data centers to the edge.

The Intellyx take

Arrcus’s vendor-agnostic approach avoids lock-in while supporting cost-effective scaling for various inferencing requirements.

By tailoring its lossless, high-throughput Ethernet fabric for AI, Arrcus supports rapid deployments of AI inferencing at scale across both edge and cloud environments.

ACE-AI’s modular programmability and full-stack visibility round out the platform’s capabilities for supporting the wide range of AI inferencing use cases across industries.

The result is an AI-ready network that is fast, high performance, resilient, flexible, and manageable – everything today’s AI-forward enterprises require of their networks.

Copyright © Intellyx BV. Arrcus is an Intellyx customer. Intellyx retains final editorial control of this article. No AI was used to write this article.