Snickerdoodle FPGA Project: Image Processing and Position Visualization using Glasgow Linux

The goal of this project is to use the Snickerdoodle board. This board is unique as it has an FPGA on board that can be utilized. This is a Xilinx product and we have used Vivado to generate our IP core and flash the device. Below there is a demo video of our product.

Although the product itself is working, there were several things that didn’t work as expected. Mostly it was on the software and operating system-related issues. The boards are flashed with a custom Linux image based off of Ubuntu 18.04 Bionic. However, it seems to be a fork of it and has a customized device tree to be able to use the onboard FPGA. They named the new version”Glasgow”. They have the apt mirrors set to use Bionic, but some tools that are installed require the name to be on the system available. To be more specific, when the command cat /etc/os_release the name should be one of the official releases.

Docker installation uses that command and when trying to run their installation, the remote server cannot find the Glasgow name. They have another server to pull from that is given in the installation guide. Due to limited time, we skipped using Docker and made individual Linux services on the device. It is possible to get a build and install all components, or try to change the mirror name and all the other hurdles that come up. There is a way to do it for sure, but with limited time, we opted. out of that option.

Since Docker didn’t work either we chose to remove Kafka from our plan and replace it with ZeroMQ. It is much easier to set up and also very easy to add to your code. Kafka would be too much on the embedded device. ZeroMQ is lightweight and we chose to do this, it performed well and is a very good option for embedded devices when services need a message queue to have them buffered.

Aside from these changes one of the most time-consuming problems that we occurred was the campus network blocking updates and downloads performed from the device. Half of the devices were configured and used the new network and the other half were using the old network. The devices with the camera module were using the network so every time we had to make a change we had to go off the network as it wasn’t something we can change on the device to bypass the firewall.

The initial design also included a Web Socket service that pushes all the clients to the new position data. Once again due to limited time, we had to revert to a backup plan where the client would poll the HTTP server when it is ready to request a new data point. To be able to do that unfortunately we ditched the plan where multiple people can join the demo. If there were a couple more days, there would be a working version of that implemented as well. There were conflicts when the loop waiting for a new message got blocked by the web socket server and vice versa. There is for sure a way to solve it but didn’t make it in time.

Even though this is not a problem, we also did make changes to how the interface was implemented. The initial plan was to use React and ThreeJS to visualize the ball position but we chose to use vanilla javascript and a simple HTML page served through Nginx. This turned out to be much easier to implement than using React as we didn’t have many components. There was just one scene and two objects that moved to the point from the received position data.

We encountered these problems on the software side of the project, but overall it worked and we were able to implement the system. The hardware created an overlay with the ball being marked white and everything else black. We read that image and processed them through the diagram below.