Skip to main content

Introduction to AI Execution Hardware in DENK SDK

When working with neural networks for image analysis, selecting the right execution hardware is crucial for achieving optimal performance. Execution providers determine how and where computational tasks are processed. This guide introduces the different execution providers supported by the DENK SDK to help you make an informed decision for your specific use-case.

Why Execution Providers Matter​

Different execution providers utilize various hardware resources to perform neural network inference. The choice of execution provider can significantly impact the speed, efficiency, and responsiveness of your applications. By selecting the appropriate execution provider, you can leverage the full potential of your hardware, ensuring faster and more accurate neural network predictions.

Supported Execution Providers​

1. x86_64 CPU​

Description​

The x86_64 CPU execution provider uses the central processing unit (CPU) of your computer to run neural network inferences. This option is universally supported and easy to set up, making it suitable for development, testing, and scenarios where GPU resources are not available.

Some CPUs, such has mobile laptop CPUs, have an internal GPU that can be used to accelerate your inference. See DirectML

SDK Package​

  • Python: denk_api_python_wrapper_cpu.whl
  • C#: denk_api_csharp_wrapper_cpu.nupkg
  • Docker: supported
  • C++ DLL: supported

Pros and Cons​

  • Pros: Universally supported, simple setup, and no additional hardware required.
  • Cons: Slower performance compared to GPU-based execution.

2. NVIDIA GPU (CUDA)​

Description​

The NVIDIA GPU (CUDA) execution provider utilizes NVIDIA graphics cards to perform inference. GPUs are highly efficient at handling parallel computations, making them ideal for tasks involving neural networks. Using NVIDIA GPUs can provide significant speed improvements compared to CPUs.

In some specific cases, such as very large images, this provider is faster than the universal DirectML execution provider because we can leverage cuda functions that we can't use with DirectML.

SDK Package​

  • Python: denk_api_python_wrapper_win_gpu.whl (Windows), denk_api_python_wrapper_lin_gpu.whl (Linux)
  • C#: denk_api_csharp_wrapper_gpu.nupkg
  • Docker: supported
  • C++ DLL: supported

Pros and Cons​

  • Pros: Exceptional performance for large-scale and real-time applications.
  • Cons: Requires compatible NVIDIA GPU hardware and CUDA installation.

3. DirectML (DML)​

Description​

DirectML is a hardware-accelerated machine learning component, part of the DirectX family, which enables the use of various GPU types for inference. This includes not only NVIDIA GPUs but also Intel integrated GPUs and AMD GPUs. DirectML abstracts the hardware-specific optimizations, allowing for flexibility across different GPU manufacturers.

SDK Package​

  • Python: denk_api_python_wrapper_win_dml.whl (Windows only, includes CPU support)
  • C#: denk_api_csharp_wrapper_dml.nupkg
  • Docker: not supported
  • C++ DLL: supported

Pros and Cons​

  • Pros: Supports a wide range of GPUs (Intel integrated GPUs, AMD GPUs, and NVIDIA GPUs), offering flexibility and potentially better cost-efficiency.
  • Cons: Requires Windows operating system and compatible DirectML-supported hardware.

4. On-Edge AI Accelerators​

Description​

On-Edge, low power AI accelerators are a good choice for high performance inference, e.g. directly on a camera. Due to the dynamic of the accelerators at the moment contact us here for more information and if we can solve your use-case together.

Pros and Cons​

  • Pros: Fast, low power and cost-efficient inference.
  • Cons: No out-of-the box support at the moment. Higher implementation cost for the whole user-story.

Choosing the Right Execution Provider​

Development and Testing​

For development purposes, the x86_64 CPU execution provider is sufficient. It allows you to develop and test your applications without needing specific GPU hardware.

Production and High-Performance Needs​

For production environments or applications requiring high-performance, real-time processing:

  • NVIDIA GPU (CUDA): Offers the best performance for large-scale and intensive applications.
  • DirectML (DML): A versatile choice if you need support for non-NVIDIA GPUs, providing flexibility without compromising too much on performance.

Understanding these execution providers and their capabilities ensures you select the right hardware to meet your application's needs, optimizing both performance and cost.

By strategically selecting the execution provider, you can ensure that your neural network inference operations run efficiently and effectively, leveraging the full potential of your available hardware. If you need further guidance, our support team is available to help you make the best choice for your specific setup.