How to bring deep learning to embedded systems

AI is a common topic in the news these days. All big tech companies dive into the AI business. However, as a firmware engineer, I have never dealt with AI in microcontrollers. I have heard of machine learning and deep learning due to Alphago a few years ago.

I attended Arrow Tech Expo yesterday. It was a good chance to catch up on some trends in the firmware world. One of the seminars I attended was sponsored by NXP: Expanding What's Possible on the Edge. I have been just curious about how to apply AI to microcontrollers but never willing to find out the answer on my own. This session gave me a good idea of the microcontroller's roadmap and applying AI to microcontrollers.

AI, Machine learning and Deep learning are frequently used without much distinction. Technically, there are differences:

Approaching to develop AI is very different than traditional programming. Traditional programming finds out the answer based on the rule and data. However, Machine learning defines the rule based on Data and the answer. Deep learning, the subset of machine learning, processes two steps which are training and inference.

Training phase: In this phase, the AI model is trained or "learned" using a labeled dataset. The goal is to enable the model to recognize patterns and make predictions or classifications based on input data.

Inference phase: In this phase, the trained model is used to make predictions or classifications on new, unseen data. The trained model is deployed and utilized to perform real-time tasks.

Since the inference phase is required mostly calculation, the microcontroller can handle it. To optimize this calculation, NXP adds NPU(Neural Processing Unit). Newer processors contain this unit to solve easy AI problems. From NXP's roadmap, i.MX series contain NPU. However, what happened if your processor does not have NPU?In the benchmark, i.MX RT which does not have NPU solves an AI problem with FPU(Floating-Point Unit) since it does not have NPU but optimizes for heavy calculation. You can find out more information about NPU here.