Leveraging Azure IoT for AWS DeepRacer

In this article, I talk about an internal project I developed with the AWS DeepRacer and Microsoft Azure IoT services. The goal was to figure out a way to detect different types of Fazer chocolates in the surroundings by navigating the AWS DeepRacer around the environment and stream them in order to visualize the statistics in real-time.

What is the AWS DeepRacer?

The AWS DeepRacer is an autonomous 1/18th scale race car designed to test reinforcement learning models by racing on a physical track. It uses the car’s camera to view the track conditions and decide on the speed and steering levels.

Planning process

The first step was to figure out how to facilitate the streaming of data. I chose the Azure IoT platform for this purpose and started off with creating an Azure IoT Hub. This is an Azure managed service in the cloud which acts as a central message hub for cloud-to-device or device-to-cloud telemetry. I created a Linux VM to simulate a device to make sure all connections work as intended. Then I registered the AWS DeepRacer as an IoT Edge device in the IoT Hub. Finally I SSH’d into the AWS DeepRacer and installed the Azure IoT Edge runtime, which is what turns a device into an IoT Edge device. I won’t go too much in detail into the Azure IoT setting up process because there is excellent documentation on Microsoft Azure along with sample codes to get you started.

Implementing computer vision aspect

Now I needed to train a machine learning model to detect the different types of chocolate bars. I started by taking a video of the Fazer chocolates from all possible angles and in different lighting sessions. Data augmentation is a key factor in a good model performance. I sliced the video into frames using ffmpeg. I did not want to train my own model from scratch nor did I wanted to wait to build a transfer learning model, so I decided to opt for the Custom Vision service from Azure Cognitive Services. The result was a pretty decently working machine learning model which I could easily export as a Docker container containing a Tensorflow model and service code. This allowed me to deploy the container directly on the AWS DeepRacer for inferencing and eliminating latency in predictions.

Next I deployed two modules, one responsible for capturing frames from the camera, and the other for performing the image classification on the chocolates.

Challenges

I did not need to write the scoring script for the image classification because it was part of the Docker export from the Custom Vision service. The modules were then deployed to the AWS DeepRacer. However, my problems had just begun. I realized the Tensorflow library used for classification cannot run on the AWS DeepRacer. After hours of searching online, I realized that is due to the fact that Tensorflow binaries have been using AVX instructions after Tensorflow 1.6 and that was not available on the AWS DeepRacer. I was lucky enough to find a custom built Tensorflow wheel file without AVX. All what was left to do was to modify my Dockerfile for the classifier to wget the custom built whl file instead of pip, and it worked like a charm. Now that Tensorflow was running, I realized that images were not being captured. The AWS DeepRacer camera cannot be directly accessible via the Docker container (ROS). At this point, the fastest way to get this working was to use an external web-camera connected to the AWS DeepRacer via USB, which is exactly what I did.

Conclusion

I already had a reinforcement learning model on the AWS DeepRacer which did pretty well on a physical track, but I also had the option to manually control the car. For the purposes of this project, I decided to manually control the car, since the model on the car was tuned towards racing on a physical track. All tests passed and the predictions were constantly being delivered to the IoT Hub. I forgot to add negatives to the classifier, which I then ended after my initial test run. Finally we had the real-time data being flowed into PowerBI, which concluded the project.

All in all, this was an amazing learning experience. If you have any questions or would like to know more, contact me.

Check more from the video