Edge Inferencing: Bringing Intelligence to the Edge

Contributors: Vinaykumar Kadari (IIT Hyderabad), Somya Jain (IIT Hyderabad), Jarupula Sai Kumar (IIT Hyderabad)

1. Executive Summary

Edge inferencing is fundamentally reshaping the landscape of artificial intelligence, embedding real-time decision-making capabilities directly into the myriad devices that populate our environment – from smartphones and autonomous vehicles to industrial sensors and network infrastructure. This post explores the trajectory of edge inferencing, detailing its compelling advantages such as low latency, enhanced privacy, offline operational capability, and improved efficiency. We'll examine the diverse applications unlocked by this paradigm shift across various industries. Critically, we'll delve into the sophisticated optimization techniques being developed by researchers to surmount the inherent challenges of deploying complex AI models onto resource-constrained edge hardware. Specific research contributions are highlighted, outlining the problems addressed, the innovative solutions proposed, and the results achieved, thereby painting a picture of a rapidly evolving and impactful field. As the proliferation of edge devices continues unabated and AI models grow in complexity, mastering these optimization strategies becomes paramount to unlocking the full, transformative potential of edge AI.

2. Introduction – Why This Topic Now?

Artificial intelligence isn't just living in distant data centers anymore; it's right here in our pockets, cars, and homes. Think about the instant responses from your voice assistant, the lightning-fast reflexes of an autonomous vehicle, or the smart alerts from your security camera. The magic behind these is edge inferencing – running AI models directly on the device itself or on nearby edge servers.

Why the shift away from relying solely on the powerful cloud? Several critical needs are driving this change:

• Immediacy: Need decisions now? Edge AI cuts out the round-trip to the cloud, delivering the ultra-low latency essential for real-time control and interaction.

• Privacy: Concerned about sending sensitive data like camera feeds or voice commands off your device? Edge processing keeps it local, significantly boosting privacy and security.

• Reliability: What happens when the internet connection drops? Edge applications can keep functioning, crucial for reliability in remote or mobile scenarios.

• Efficiency: Sending massive amounts of data back and forth costs bandwidth and energy. Edge AI can reduce network strain and often be more power-efficient.

But here's the catch: squeezing complex, state-of-the-art AI models onto devices with limited computing power, memory, and battery life is a huge technical hurdle. How do we shrink these digital brains? How do we cleverly divide the workload between the device and the cloud? How can we tailor AI for specific jobs like spotting anomalies or understanding wireless signals?

This post dives into the landscape of solutions, exploring the evolution of techniques, specific research breakthroughs tackling these problems, and the ongoing quest for smarter, faster, and more efficient AI right where we need it. Let's explore how we're making AI work at the edge.

Interactive Prompt: How might the relative importance of latency vs. privacy vs. reliability shift depending on the specific edge application (e.g., autonomous driving vs. smart home assistant vs. remote environmental sensor)? What design trade-offs does this imply for developers choosing optimization strategies?

3. Landscape & Literature Overview: From Cloud Dependence to Edge Collaboration

The journey to today's sophisticated edge AI wasn't instantaneous. It evolved through distinct phases:

Phase 1: The Cloud-Centric Era (Before ~2017)

Initially, AI, especially deep learning, lived almost exclusively in the cloud. Our devices acted like messengers, sending data away for processing and waiting for the answers. Think early voice assistants that felt a bit sluggish. This model worked, but the delays (latency), data transmission costs, need for constant connectivity, and privacy concerns were significant drawbacks for many potential applications.

Phase 2: Early Edge Models (~2017) - Taking the First Step On-Device

Recognizing the cloud's limitations, researchers started designing lightweight AI models specifically for efficiency. Models like MobileNet emerged, proving it was possible to run meaningful AI directly on devices with limited resources.

This was the dawn of practical on-device inference, but often came with a trade-off in accuracy compared to their larger cloud cousins.

Phase 3: Hybrid and Distributed Intelligence (~2017-Present) - Finding the Sweet Spot

As AI on devices evolved, a crucial question emerged:

Why settle for either edge or cloud when we can have both?

The cloud, with its immense computational power, excels at running complex models but struggles with latency, privacy, and connectivity. The edge, on the other hand, is great for local processing — fast, private, and resilient — but can’t always handle the heavyweight tasks. So, why not combine their strengths?

This was the dawn of hybrid and distributed intelligence, where edge-cloud collaboration allows us to leverage both sides’ capabilities.

Task Offloading became one of the first strategies. Devices now make real-time decisions:

"Should I process this locally, or send it to the cloud for more power?"

Depending on network conditions, battery life, or the urgency of the task, edge devices can offload heavy processing to the cloud, balancing efficiency and performance. This approach maximizes both privacy and computational capability, depending on the situation.

Then came Split Learning, a game-changer in model distribution. But why not take this one step further? Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge (Shi et al., 2017) explored the idea of splitting AI models into multiple segments and distributing them between the cloud and edge devices. This technique is about splitting the model itself, dividing it between the cloud’s powerful servers and the edge’s limited resources. The edge handles the early stages of data processing, while the cloud picks up the heavy lifting of deeper layers.

In a similar vein, Shadow Puppets (Venugopal et al., 2018) demonstrated how splitting models optimizes collaboration between edge and cloud. The edge focuses on extracting basic features, and the cloud processes the more complex tasks. This reduces the computational load on both devices, allowing each to focus on what it does best.

Another related technique, BottleNet++ (Kang et al., 2017), takes a different approach by optimizing feature compression. By compressing data at the edge before sending it to the cloud, this technique reduces the communication overhead and ensures faster, more efficient edge-cloud collaboration. It also helps maintain system efficiency by improving both the edge and cloud’s responsiveness.

At the same time, Federated Learning emerged as a strong technique in the hybrid AI landscape. Instead of uploading raw data to the cloud, Federated Learning allows devices to locally train models and share only the learned updates. This approach preserves privacy while improving global model accuracy. While Federated Learning benefits from collaborative training, it faces challenges such as communication overhead, where devices must repeatedly send updates to the central server, and model convergence, where the diversity of data on different devices can slow down training efficiency.

Together, these innovations — Task Offloading, Split Learning, Neurosurgeon, Shadow Puppets, BottleNet++, and Federated Learning — represent a new era of collaborative intelligence. Instead of forcing a choice between the edge and the cloud, these techniques split tasks and models intelligently across both, improving performance, privacy, and efficiency.

However, despite the incredible potential, challenges remain:

These persistent challenges highlight the need for optimization techniques, which we’ll explore next. How can we refine these collaborative models to make them even more adaptive, responsive, and scalable?

Phase 4: Maturation – Tools, Hardware & Infrastructure (2019–Present)

As edge-cloud systems moved from labs to real-world deployments, the focus shifted from clever models to robust infrastructure. The question wasn’t “Can it run?” but “Can it scale reliably?”

To meet this demand, a new generation of hardware emerged:

• AI Accelerators like TPUs, NPUs, and custom edge chips offered high-throughput, low-power inference — essential for phones, drones, and sensors.

• FPGAs brought reconfigurable pipelines for efficient edge workloads.

• GPUs remained dominant in the cloud, especially for training and heavier inference tasks.

• High-speed storage and edge caching allowed data to be processed locally, minimizing latency and reducing cloud dependency.

• And of course, 5G made its mark — unlocking real-time collaboration between edge and cloud with faster, more reliable connectivity.

As discussed in Cisco’s AI Inference Infrastructure Summit 2025, the shift toward distributed co-inference required more than just raw speed — it needed intelligent coordination across networks, silicon, and workloads.

Meanwhile, the paper “AI Inference Infrastructure: Scaling the Cloud-Edge Co-Inference Model” (arXiv:2502.15712) explores how to architect infrastructure that balances energy efficiency, latency, and model size — especially for applications like AR, real-time video, and wearables.

On the software side, frameworks like TensorFlow Lite, ONNX Runtime, and PyTorch Mobile matured to support diverse deployment scenarios, while orchestration tools started automating decisions like when and where to run parts of a model.

Yet, even with all this progress, bottlenecks like heterogeneous device coordination, model portability, and security risks still persist.

That’s where optimization techniques come in — the final piece to make edge-cloud AI fast, private, and scalable. And that’s what we’ll tackle next.

4. Applications of Edge Inferencing

Edge AI isn’t just for phones and sensors — it’s increasingly at the heart of next-gen network infrastructure, enabling real-time analytics, anomaly detection, surveillance, and self-optimizing systems in 5G/6G and beyond.

Let’s begin with a high-level mapping of common network-side applications, the AI models they typically rely on, and why edge inference is key.

🔹 Network-Centric Applications of Edge AI & Typical Models

Edge inference in networks allows intelligence to happen where the data originates — in base stations, routers, or gateway devices — leading to faster decisions, less backhaul traffic, and stronger privacy.

Now, let’s enrich this with real examples from the papers you shared, showcasing these concepts in action.

---

1. Wireless RF-Based Sensing and Network Intelligence

Paper: Edge Perception: Intelligent Wireless Sensing at Network Edge (Arxiv:2410.21017)

• Use Case: Use WiFi or RF signal reflections to detect human activities (e.g., walking, falling, gestures).

• Application Area: Smart surveillance, indoor monitoring, wireless security.

• Models Used: Lightweight CNNs + Transformer-based temporal encoders.

• Edge Benefit:

  • No need for cameras — inherently privacy-preserving.

  • Deployed at WiFi routers/APs — no extra hardware.

• Challenges:

  • Accuracy drops in highly dynamic radio environments.

• Improvements:

  • Attention mechanisms help adapt to temporal variations.

  • Edge-optimized quantized models reduce latency on embedded devices.

---

2. AI for IoT & Networked Devices

📄 Paper: AI on the Edge: Specialized Architectures for AI in IoT Systems (Arxiv:2003.12488)

• Use Case: Resource-aware inference for devices like smart meters, gateways, and low-power nodes.

• Application Area: IoT edge in network slices, remote monitoring, telco control systems.

• Models Used: TinyML, compressed CNNs, model distillation.

• Edge Benefit:

  • Enables localized learning without transmitting sensitive telemetry data.

  • Operates even with intermittent connectivity.

• Challenges:

  • Severe compute and memory constraints (1–2 MB max).

• Improvements:

  • ASIC design tailored for edge inference.

  • Compiler toolchains that co-optimize model and hardware.

---

3. Edge-Assisted Detection in Real-Time Networks

📄 Paper: Neural Compression Techniques for Edge-Assisted Real-Time Object Detection (Arxiv:2007.15818)

• Use Case: Compress intermediate neural features and offload part of inference to the cloud.

• Application Area: Real-time surveillance over constrained links (e.g., mobile surveillance drones, city cameras).

• Models Used: YOLOv3 backbone + learnable bottlenecks (neural compression layers).

• Edge Benefit:

  • Drastically reduces bandwidth load while preserving performance.

  • Enables scalable edge-cloud collaboration.

• Challenges:

  • Still needs occasional uplink.

• Improvements:

  • Adaptive compression rates based on link status.

  • Joint end-to-end training minimizes accuracy drop.

Network AI at the Edge — Comparison Summary

---

Takeaway

Edge inference is rapidly becoming essential infrastructure in smart networking environments — from detecting abnormal behavior in real time, to optimizing how 6G networks allocate resources on the fly, to enabling secure, always-on surveillance without breaching privacy.

Next, we’ll explore how these edge workloads can be further optimized — through quantization, pruning, knowledge distillation, and NAS — unlocking new levels of performance even under strict network or device constraints.

5. Methodology: Optimizing AI Models for Edge Constraints

Edge inferencing is fundamentally reshaping the landscape of artificial intelligence, embedding real-time decision-making capabilities directly into the myriad devices that populate our environment — from smartphones and autonomous vehicles to surveillance cameras and wearable tech. But with this shift comes a critical challenge: how do we deploy large, compute-hungry deep neural networks (DNNs) on devices that are inherently resource-constrained?

To meet this challenge, researchers have developed a variety of optimization strategies that aim to strike a balance between efficiency and accuracy.

This section explores the evolution of these techniques — starting from model splitting, to early exiting, to dynamic model splitting, and moving toward the exciting future of adaptive Mixture of Experts.

---

5.1 Model Splitting: Sharing the Load Between Edge and Cloud

One of the earliest approaches to optimize edge inference was model splitting, dividing a DNN across the device and the cloud.

The seminal Split Neural Network (SplitNN) framework (Gupta et al., 2018) formalized this: the edge device computes initial layers, sends intermediate activations to the cloud, which then completes the computation.

Thought Prompt:

"If two inputs — say, an image in good lighting versus a noisy, low-light image — have different complexities, should the device offload at the same layer for both?"

Applications:

• Telemedicine: Small wearable health monitors sending partial features to hospital servers.

• Smart surveillance: On-camera quick processing before detailed cloud-based analysis.

EdgeRL (Mounesan et al., 2024): builds on this by applying reinforcement learning (A2C) to select not just the split point, but also the best model version (e.g., lightweight vs. heavyweight) — considering device energy, latency, and accuracy as a three-way tradeoff. Unlike static SplitNN, EdgeRL dynamically learns optimal execution profiles tailored to heterogeneous edge environments.

Pros:

• SplitNN: Simple, effective for bandwidth-constrained scenarios.

• EdgeRL: More adaptive, energy-aware, but requires online training and controller overhead.

---

5.2 Early Exiting: Knowing When to Stop

Another innovation was early exiting, where the model attaches intermediate classifiers that can terminate computation early if the model is sufficiently confident (BranchyNet, Teerapittayanon et al., 2016).

Thought Prompt:

"If your model decides it is confident after seeing just part of an input, would you always trust it? How could overconfident mistakes affect safety-critical applications?"

Applications:

• Autonomous vehicles: Quickly recognize easy cases like open roads without fully processing.

• Voice assistants: Instantly recognize short, clear commands without full network traversal.

BranchyNet introduces entropy-based confidence thresholds at exit branches, allowing early termination for "easy" inputs. This significantly reduces runtime — with experiments showing 2x–5x speedups on models like LeNet and ResNet.

Pros:

• BranchyNet: Low-latency and energy-saving, but can misclassify hard inputs early.

• Can be used in tandem with compression/pruning.

---

5.3 SplitEE: Merging Split Inference and Early Exit

SplitEE (Jangda et al., 2023) cleverly combines splitting and early exit.

The device processes initial layers, sends activations to the server only if early exits are not confident enough.

SplitEE introduced confidence intervals and confident classes:

• If few classes dominate the softmax probability distribution (low entropy), the model exits early.

• Otherwise, the features are offloaded.

Thought Prompt:

"This makes sense for classification tasks where you can define 'confident classes' easily. How might you design an early exit for dense tasks like object detection or image segmentation?"

Applications:

• Smart home automation: Simple device decisions made locally (e.g., "light is on") and complex tasks escalated to servers.

Pros:

• SplitEE improves both latency and energy, but is still fixed in split locations.

---

5.4 Dynamic Splitting: Toward True Adaptivity

While SplitEE significantly improves efficiency, it still relies on fixed split points.

Researchers have explored dynamic model splitting — choosing where to split per input dynamically.

Key techniques include:

• Input-aware splitting: Based on entropy or feature statistics (Fang et al., 2022).

• Learned policies: Lightweight networks predict optimal split points (Auto-Split, Ma et al., 2022).

• Variance-based partitioning: High variance features signal complex regions needing deeper layers (Hang et al., 2019).

Thought Prompt:

"Is there a point where dynamically searching for the best split adds more delay than the gain from optimized computation? When is static splitting better than dynamic?"

EdgeRL again comes into play here. Its reinforcement learning agent adapts dynamically to device state (battery, bandwidth, motion), selecting both the best split layer and model variant per input and situation. This yields robust performance across varying edge loads — especially useful in ad-hoc, mission-critical deployments like UAVs or field robotics.

Applications:

• Mobile AR gaming: Dynamically offload heavy scenes to the cloud when device gets overloaded.

• Industrial IoT: Real-time monitoring adapts depending on object complexity.

Pros:

• Dynamic splitting is adaptive but computationally expensive to manage; RL methods (like EdgeRL) offset this through policy learning over time.

---

5.5 Dynamic Splitting + Early Exiting: The Next Level

Conceptually, combining dynamic splitting and early exiting would create extremely powerful, truly input-adaptive systems:

• Edge device: Processes just enough layers, dynamically choosing split or exit.

• Server: Continues computation and exits early if possible.

This has not been fully solved yet — it would require precise, fast runtime decisions balancing latency, bandwidth, and confidence.

Thought Prompt:

"Should a device first check for early exit before deciding to split, or split first and let the server decide? Which order would maximize efficiency?"

Toward Integration:

• BranchyNet offers the early exit mechanisms.

• EdgeRL offers dynamic split decision-making.

• Future systems may integrate BranchyNet-style exits into EdgeRL's learned policies, creating an RL-optimized, exit-aware split system.

---

5.6 Mixture of Experts (MoE): Specialized Models at the Edge

A new class of architectures gaining traction for edge optimization is the Mixture of Experts (MoE).

Unlike traditional models that activate all parameters for every input, MoEs activate only a sparse subset of "expert" subnetworks, each specialized for different types of data or tasks. This yields two major advantages:

• Efficiency: Since only a small portion of the model runs per input, inference becomes lighter and faster — especially helpful for edge deployment.

• Modularity: Experts can be trained or updated independently, enabling more scalable and maintainable systems.

Recent work such as Zhao et al., 2024 explores using MoE in multimodal settings, routing inputs (e.g., text, image, video) to different combinations of experts based on their content and complexity.

Thought Prompt:

"What if your smartwatch had multiple tiny neural networks, each tuned for a different health signal — like heartbeat vs. motion?
Could an MoE selectively activate only the ones it needs, saving energy while improving precision?"

---

Why MoE is Promising for Edge:

• Enables selective computation: Only the needed experts are invoked.

• Supports input-conditional behavior, which aligns well with heterogeneous data on edge devices.

• Offers a path toward scalable model growth without increasing per-input compute.

Applications:

---

🛰️ Our Proposal: NIC-Level Expert Routing Inspired by MoE

Building on the MoE paradigm, we propose a new direction: what if the expert selection process — the gating mechanism — was offloaded directly to the router’s Network Interface Card (NIC)?

Thought Prompt:

"Can we eliminate the need for an intermediate software layer by running lightweight classification directly at the NIC of the router — and selectively forward packets to the correct expert server, just like a MoE would?"

In our proposal, the NIC itself hosts a simple routing model — such as k-means clustering or rule-based classification — and uses this to dispatch traffic directly to the appropriate expert model server. The actual expert LLMs remain on standard backend servers.

This brings the concept of MoE-style sparsity into the physical network fabric, enabling a hardware-level gating mechanism that mimics neural routing — without adding software bottlenecks.

Why This Could Be Transformative:

• Reduces latency and overhead: Packet routing happens right at ingress, avoiding extra network layers.

• Enables sparse activation of backend LLMs: Only the necessary expert is invoked per input.

• Scales elegantly for edge or IoT setups: Distributed classification offloads central logic and speeds up decision time.

• Saves compute and bandwidth: Eliminates unnecessary inter-node communication.

---

Theoretical NIC Configuration Needed:

Suitable platforms for this include:

• NVIDIA BlueField DPU

• Intel Mount Evans IPU

• Netronome Agilio SmartNIC

• Or a software-defined approach using eBPF/XDP in the Linux kernel for prototyping

---

Hardware Summary:

• CPU: 24-core ARM (aarch64) @ 2.0 GHz

• RAM: 23 GiB total (only ~1.8 GiB free in the snapshot)

• Swap: 15 GiB

• GPU: No NVIDIA GPU detected (nvidia-smi not found and likely none present)

• Architecture: ARM64 (Little Endian)

LLM Support on This OCTEON Server

For deployments where expert models are still hosted on the backend (as in our case), it’s crucial to ensure the hardware supports quantized LLMs efficiently. Based on testing and configuration:

This setup is capable of hosting small, quantized LLMs as backend “experts,” while the NIC-level classifier routes each incoming request directly to the correct expert based on content or domain — thus forming a complete MoE-inspired pipeline from network to model.

---

6. Observations & Patterns: Edge Inferencing in Action

From real-world deployments to simulation studies, several patterns have emerged that highlight both the power and limitations of edge inferencing in networked environments.

Recurring Design Patterns

---

Common Bottlenecks

1. Dynamic Environments
   RF-based systems and real-time inference pipelines often degrade in unstable or noisy conditions.

2. Overconfidence in Early Exits
   Branch classifiers sometimes misclassify rare or ambiguous inputs, leading to safety concerns.

3. Cloud Dependency Tradeoffs
   Compression and split-based models still rely on cloud uptime and uplink quality.

4. Model Version Fragmentation
   Managing multiple lightweight vs. heavyweight versions introduces orchestration complexity.

---

What Works Well

• Hybrid Edge-Cloud Systems outperform purely local or cloud-based inference in most real-time applications.

• Sparse Activation Techniques (MoE, early exits) deliver compute savings without proportional loss in accuracy.

• Input-Adaptivity — such as dynamic model splitting — is essential for generalizing across hardware, environments, and workloads.

---

7. Conclusion – What We Learned

Edge inferencing is no longer an afterthought or secondary optimization layer — it’s becoming the foundation of intelligent, distributed systems across telecom, IoT, industrial monitoring, and autonomous environments.

Key Takeaways:

• Location Matters: Inference close to data origin = lower latency, higher privacy, and reduced bandwidth.

• Model Adaptivity Is Crucial: Systems that adapt per input (e.g., via RL or entropy) are better suited for heterogeneous, constrained environments.

• Hardware-Software Co-Design Is the Future: Solutions like NIC-level inference and compiler-optimized TinyML show that tight integration of hardware and AI is essential.

• Sparsity + Specialization = Scalability: From early exits to Mixture of Experts, selective computation enables both efficiency and modular growth.

---

8. Future Work & Opportunities

While the current exploration highlights the growing importance and feasibility of edge inferencing across diverse networking applications, several open challenges and research directions remain.

1. Unified Scheduling Across Dynamic Workloads
   Most existing solutions optimize for either early exiting, model splitting, or expert routing in isolation. Future systems must integrate these mechanisms holistically — enabling dynamic scheduling that can choose between exiting early, splitting computation, or routing to an expert model based on the context of the input, network state, and device condition. A unified framework balancing energy, accuracy, latency, and bandwidth in real time is still lacking.

2. Efficient Expert Gating at the Network Layer
   While our proposed NIC-level expert routing suggests a novel direction, practical implementation still demands rigorous investigation. Questions remain around optimal gating functions, update mechanisms, hardware compatibility, and security — particularly for real-time LLM workloads and diverse traffic patterns. Future work could explore training low-footprint, high-precision classifiers for programmable dataplanes that align with LLM intent classification or modality detection.

3. Benchmarking Multi-Stage Edge Pipelines
   There is limited standardization in evaluating multi-component edge systems (e.g., SplitEE + MoE + Quantized LLMs). Creating comprehensive benchmarks that simulate realistic edge workloads — considering fluctuating compute, link quality, and adversarial conditions — would enable better comparison across solutions. This includes not only performance metrics (e.g., latency, accuracy, energy) but also developer complexity, retraining costs, and interoperability across edge-cloud fabrics.

4. Privacy-Preserving Expert Routing
   With growing concern over data privacy and regulatory constraints (e.g., GDPR, HIPAA), routing decisions made at the edge must be explainable and privacy-aware. Future research should investigate how to embed privacy guarantees into expert gating — possibly via federated learning, secure enclaves, or homomorphic encryption — without sacrificing responsiveness or scalability.

5. Adaptive MoE for Streaming and Low-Latency Tasks
   Most current MoE research focuses on large-scale static inputs (e.g., long documents or images). Extending this to edge-native workloads such as streaming sensor data, RF signals, or time-series telemetry will require new mechanisms for continual learning, low-latency expert switching, and memory-efficient routing. There is an opportunity to bridge MoE architectures with temporal models like LSTMs, Transformers, and attention-based encoders tailored for real-time environments.

6. Unified Scheduling Across Dynamic Workloads
   Most existing solutions optimize for either early exiting, model splitting, or expert routing in isolation. Future systems must integrate these mechanisms holistically — enabling dynamic scheduling that can choose between exiting early, splitting computation, or routing to an expert model based on the context of the input, network state, and device condition. A unified framework balancing energy, accuracy, latency, and bandwidth in real time is still lacking.
   Additionally, early exit has traditionally been applied in classification tasks; extending this mechanism to non-classification domains such as object detection, language modeling, or regression opens a new research frontier. This would require redefining exit criteria, uncertainty quantification, and task-specific thresholds beyond simple confidence scores.

In summary, while edge inference has matured from concept to deployment, the next phase of research lies in converging modular innovations — model optimization, routing strategies, and hardware integration into cohesive, intelligent, and adaptable edge AI infrastructures.

9. References / Links

1. Edge Perception: Intelligent Wireless Sensing at Network Edge

2. AI on the Edge: Rethinking AI-based IoT Applications Using Specialized Edge Architectures

3. Shadow Puppets: Cloud-level Accurate AI Inference at the Speed and Economy of Edge

4. Sharing and Caring of Data at the Edge

5. Mobile Edge Intelligence for Large Language Models: A Contemporary Survey

6. Semantic Data Sourcing for 6G Edge Intelligence

7. Neural Compression and Filtering for Edge-assisted Real-time Object Detection in Challenged Networks

8. Network Anomaly Detection in Distributed Edge Computing Infrastructure

9. BranchyNet: Fast Inference via Early Exiting from Deep Neural Network

10. SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud

11. Performance Characterization of Expert Router for Scalable LLM Inference

12. Edge-PRUNE: Flexible Distributed Deep Learning Inference

13. BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems

14. Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge

15. Temporal Decisions: Leveraging Temporal Correlation for Efficient Decisions in Early Exit Neural Networks

16. EC-SNN: Splitting Deep Spiking Neural Networks for Edge Devices

17. Distributed Deep Neural Networks over the Cloud, the Edge and End Devices

18. EdgeShield: A Universal and Efficient Edge Computing Framework for Robust AI

19. DNNSplit: Latency and Cost-Efficient Split Point Identification for Multi-Tier DNN Partitioning

20. EdgeRL: Reinforcement Learning-driven Deep Learning Model Inference Optimization at Edge

21. A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques

22. A Survey on Inference Optimization Techniques for Mixture of Experts Models

23. The Case for Hierarchical Deep Learning Inference at the Network EdgeEdit this text

Contributions:

Jarupula Sai Kumar

I explored the overview of edge inferencing and its growing significance in modern AI systems. My focus was on network anomaly detection in distributed edge computing infrastructures, particularly drawing insights from the paper "Network Anomaly Detection in Distributed Edge Computing Infrastructure." This paper led me to understand the need for efficient, real-time anomaly detection mechanisms within edge environments, where computing resources are often constrained. I also studied hierarchical deep learning inference models for the edge, which help enable scalable, low-latency AI processing at the network edge while addressing challenges like limited bandwidth and high energy consumption.

Somya Jain

I focused on optimization techniques for edge inferencing, which involves running AI models directly on edge devices to reduce latency, conserve bandwidth, and protect privacy. I explored early exit and model splitting, examining how early exit reduces computational costs and latency, especially for non-classification tasks. I also looked into dynamic splitting as a more efficient approach compared to fixed splitting, which can be resource-inefficient. Additionally, I studied expert routing, where specific computations are handled by specialized models, and its applications in IoT, autonomous vehicles, and healthcare, where quick, efficient decision-making is critical. I also identified key challenges in edge inferencing, including privacy concerns, dynamic scheduling, and hardware integration.

Vinaykumar Kadari

I focused on techniques to improve efficiency, scalability, and adaptability in edge inference. I explored BranchyNet, which reduces latency and energy consumption through early exit branches, ideal for resource-constrained edge devices. I also studied the Expert Router, which optimizes LLM inference by routing prompts to specialized models, enhancing throughput and minimizing latency for hybrid edge-cloud systems. Additionally, I examined EdgeRL, which uses reinforcement learning to dynamically optimize deep learning model inference based on real-time resource and network conditions, improving latency and energy efficiency.