Wouldn’t it be good if you had an RC car that completely drove itself? Well in this article I will explain how yo can build one and train it to be a self-driving dream.
So what exactly is a self driving RC car? It is a completely autonomous self-driving car. You will not need a control stick, it will drive itself. It will require quite a bit of time investment to build and train it though, so be prepared for that.
Now you understand the concept, let’s delve in ad look at which RC car you will need (Click for the 7 of the best RC cars), what on board computer will be required, the software and required training needed.
Are Self Driving RC Cars Similar to a Full Size Vehicle?
When you think of a self driving car you probably think of the Google Waymo or Tesla’s autopilot models. Self driving cars seems (and often are) the ultimate luxury and innovation. But all self driving cars, no matter the size, use the same basic type of engineering –a code accessing sensory data.
Commercial vehicles will use many cameras and multiple sensors, but RC cars will just need one. Self driving RC cars, much like their full size counterparts, use technology to learn and emulate the owner’s driving.
Using a few simple parts, you can easily turn any model of RC car into a car that can follow a track, detect traffic lights, and stop signs, and understand and respond to them too.
There are a few different ways to do this, and this article will mostly focus on how to set up your RC car using Raspberry Pi.
The Self Driving RC Car Concept
I know this sounds a little daunting. I mean it is one thing to replace a part or two, another thing to build a car from the ground up, but a completely different thing to turn your RC car into a self driving car.
But do not worry, I have done some research and wrote up this guide to help you on your way. I will be explaining how to build a self driving RC car, and some difference in the programs used to run them and methods used to build them.
All of them will focus on Raspberry Pi as the computer of choice, but there will be differences in the deep learning libraries (or neural networks) used and the programs the applications run through.
Many hobbyists turn to Raspberry Pi as their computer of choice, the car is independent and connects to just one computer.
Self Driving Cars Basic Components
As stated above, self driving RC cars are smaller, simpler versions of commercial self driving cars. These versions use a few simple items and a relativity easy to follow set of steps.
In this article, we will discuss both. Building your own self driving RC car requires very few items:
- an toy/RC car
- a camera
- a computer system.
Most hobbyists choose to use Raspberry Pi for their computer, but vary the type of car used depending on their intended goal (pleasure or racing).
The car you choose to use doesn’t matter (I’ve seen people use toy car before), but it does need to be big enough to hold the Raspberry Pi computer. Do not worry though, you will not need a massive RC car.
Raspberry Pi & Its Role
Raspberry Pi is a small, low cost computer. It is roughly the size of a credit card, this computer plugs into a monitor or TV and uses a standard keyboard and mouse.
Though it does not look like it, Pi does everything a desktop computer can do. Simple to use, Pi helps anyone get into computing and helps teach people how to program in Scratch and Python –two simple and easy to use programming languages.
Because of its size, cost, and user friendliness, most hobbyists choose to use Raspberry Pi in their self driving RC cars. It is small enough to fit on any sized car and you can easily view it on whatever type of monitor you have access to.
Raspberry Pi’s purpose in self driving cars is to receiving data from the camera and use the training you will give it to maneuver around. (More on what training is and why it is so important below!)
Using a multi threaded TCP (Transmission Control Protocol) server program, Pi receives streamed images and ultrasonic data.
Those images and data are converted and decoded into numpy arrays (a grid of values). The train image is matched with the train label –you will have to input this yourself while you are training your car.
Once training is complete, your car will create predictions as it is driving based on the matched train images and labels. This is the stage to go through and correct errors. Make sure your labels match your images and vice versa; crop any distractions out of the images; and help the RC car make the correct predictions.
There may be mistakes with the matched images or labels, so make sure you go through each command so that the right answer follows.
Getting Started
Like I said, getting started can be a little daunting, but a self driving RC car is actually pretty simple. Once your car is built, you will just need to drive it around a track to capture images, steering angles and traffic commands. (Remember, if you are just training your car to race in a closed circuit track, you will not need to gather as much thorough information.)
Training your car sets up the autopilot. While in training, the car takes images, sends them to your server of choice( a lot of people like to use an Amazon server), and gets commands in return.
The server collects those images and driving information (like speed and angles), and if your choose to use the Donkey server you can view live video of what the car sees and a virtual joystick.
But there is no need to stick to the Donkey server if there is one that you are familiar with or one than you rather use. Other servers have their perks. Once the server has those images and positions, it sends the data to a neural network model (a deep level library) in the software (the autopilot).
Once the car is trained, the models on the server can be loaded onto the car, and it should perform as it would if you were controlling it. Pretty simple right?
A lot of hobbyists use a neural network to transmit its data. The main advantage of using a neutral network is that once it is trained, it will come up with predictions very fast.
Below is a step by step guide for building a self driving car using Raspberry Pi and the Donkey server. You can use any type of server you are comfortable with, I will go through some differences between using different servers and neural networks.
Set Up Directions for Raspberry Pi + Donkey Server
- Print a top plate and a roll cage: Using a 3D printer, you will print a top plate and a roll cage. You can download the files here: https://www.thingiverse.com/thing:2260575. If you do not have access to a 3D printer, those parts can be ordered from Shapeways. It is very important that you clean your parts. You might need to re drill the holes, or at least clear away little plastic pieces.
- Assemble the top plate and roll cage. Using a 2.5x12mm screwdriver, screw into the slot on the side of the roll cage. Once you get the nut in, attach the bottom plate.
- Connect the servo shield to the Raspberry Pi. Connect to 3.3V, the two l2C pins (SDA and SCL) and ground.
- Attach Raspberry Pi to the bottom plate. Screw Raspberry Pi and the servo shield onto the top plate. Make sure the ports, this includes the Ethernet & USB port, face forward.
- Attach the camera. Make sure it is facing the right way. You should have plenty of space for the ribbon.
- Lastly, attach all the finishing details.
- Attach the roll bar assembly. Use the R-clips that came with the original body of the car.
- Run the servo cables to the car: Reserve channel zero on the servo for the throttle cable and steering is channel 1.
Now you car’s body is all set up. You will still need to do a bit of software set up before you are ready to train your car.
Software Set Up
- First you will need to set up Raspberry Pi. Download a zipped disk image of Raspberry Pi and unzip it. Slap in your Secure Digital (SD) card & spin up a disk. Put the SD card back into Raspberry Pi with the downloaded disk images on it. Plug a monitor, keyboard and mouse into your Pi, turn on the battery and connect to Wifi.
- Firing Up the Pilot Server: A dedicated server will be used for the piloting calculations by Donkey. To begin driving, you will need to fire up the DCS (also known as the Donkey Server).
Once you have got your software setup, you will be able to start driving and training your car.
Drive Your Car
The most important step to creating a self driving RC car is training. No matter what neural network you are using, you will need to train you car.
The purpose of doing so is to familiarize your car with steering angles, street commands, and other obstacles. Using images and corresponding data, your car will learn and make predictions once it is fully trained.
- Connect Pi to SSH (Secure Shell). Once that is done, all you need to do is drive around a track. Pi should start requesting directions from the server. (See below on how to build a track for you to train your RC car on!)
- Start Training It: Once your car is good to go, start training it. Remember it will need some time to learn how to perform before it can truly be self driving.Once it has been going around collecting a ton of images (at least 1,000 if not more!), you should be able to turn it onto autopilot, using this command code: python scripts/train.py –sessions <your session name> –name <name your autopilot>
And viola! You have a self driving car using Raspberry Pi and Donkey server.
Racetrack Building
Some of us will have access to an RC racetrack, but some will need to create one for the cars to train on. For a quick and simple track, you will need wide black masking tape. Make sure it is wide enough to be seen with the camera.
You will need chalk to outline the track and measuring tape to measure distance. Find a large open space (a driveway, basement, or garage work great), map out your track, and you are good to train your car.
When you are looking for an open space, test your car wheels on it. You do not want too much friction or resistance from your wheels. Try to vary your track as much as possible to give your car a thorough training. If you are planning a racing your car, you do not need to vary your track with curves, angles, stop signs or narrow spaces, but you will want to map out the track as it would look like when you are racing your car.
Using Apache MXNet
At Medium, Vincent Lam ported the code from DonkeyCar to Apache MXNet. His goal was to study them and make improvements. DonkeyCar uses Keras with TensorFlow backend. Lam used Apache MXNet with Gluon API.
MXNet needs to be installed onto Pi and onto your development machine as well. Easy enough to do, just download and save MXNet to Pi; boot it up on Pi and SSH into the Pi.
Gluon API’s Gluon Trainer feeds the model with training data and trains it according to the loss function property of the model. Gluon separates its data into training and test datasets, generates mini-batches and applies augmentations to data.
Keep in mind that while the model may be very accurate, it can still make mistakes. Proper and thorough training, and even possible assistance, may be needed to help your car when it makes mistakes or does not get clean input.
Once you have taken as many photos as you feel necessary, Lam suggests going through and cropping them a bit. Some of them may contain unnecessary and distracting information, which can confuse your car as it’s driving and may end up in it making mistakes. Plus cropping reduces the amount of data to process, making it easier and faster to load.
The car needs to respond as fast as possible so it needs to process data as fast as possible, especially if you are racing them. Lam found that to maximize optimization, you should perform Raspberry Pi’s data in numpy format and accelerating neural networks on Raspberry Pi using TVM Stack or Amazon SageMaker Neo (Amazon is what Conway and Roscoe used).
Small Self Driving RC
Conway and Roscoe built their Donkey car with the purpose of racing. They used a large RC car and a simple racetrack to train their car on.
Building an RC car is not just for engineers and computer savvy people; you can easily use a few small items (most of which Conway & Roscoe used as well) to make your own smaller version and learn a bit more about computing in the process.
Rodrigo Cava at Medium used a Raspberry Pi board, an external battery, a small RC car, and a L298N control board for his RC car. The first task was to make the car computerized. In order to do so, Cava used the L298N control board for the monitors and Raspberry Pi to control the car itself.
Regardless of the method you use, you still need to train your car by driving it around, taking pictures, and matching those images with your commands. After a thorough training Cava, trained his neural network using Keras + TensorFlow. Simple, but a lengthy process nonetheless, Cava was able to build a self driving RC car that could also recognize stop signs.
Keras and TensorFlow
In my first example, and full step by step process, used Donkey as their deep learning neural networks. But many other at home engineers use Keras + TensorFlow. So what are they and what are the differences?
Both TensorFlow and Keras are deep learning libraries. These libraries enable machines to learn commands, codes, and other directions so that they are able to import them themselves. TensorFlow is one of the most well known libraries.
It has a large community, and the number of commits and forks on TensorFlow Github repository alone show why it has such a widespread popularity. But it is not the easiest or most user friendly library to use.
As you will read below of why Keras is more user friendly, TensorFlow offers something that Keras does not –the ability to tweak in a much more fine tune way. Keras allows you to quickly build and test a neural network, but you can get more in depth with TensorFlow. This is great if you are doing research or building a deep level model.
Queues and other multithreaded applications work better with TensorFlow due to its ability to execute multiple threads simultaneously.
Equipped with a specialized debugger, TensorFlow provides visibility into the internal structure and states of running graphs. These insights can be used to facilitate debugging a variety of bugs at all states of testing, making it easier for you to fix your data as you go along.
Another deep learning library, Keras is built on top of TensorFlow. It is a high level API (Application Programming Interface). But it is much more user friendly and easier to use than TensorFlow.
If you want to build and test a neural network quickly, with minimal lines of code, Keras is your best option. You can easy build complex and simple neural networks with Keras. The Model and Sequential APIs are powerful enough to do whatever you want. Everything in Keras can be represented as modules.
The reason why a lot of RC engineers use both is to maximize both of their highlights. TensorFlow with its full control, debugger, and ability to fine tune everything, and Keras with its user friendliness and quick working ability.
Final Thoughts
If you are computer and tech savvy, building a self driving RC car might be a fun project for you to take on. It is not just for hobbyists. Even those less than tech savvy may find joy in building a self driving car as you are able to learn about neural networks and coding in a very clear way.
It is no small task and should not be rushed. It takes a lot of time to gather all the necessary information and a lot of time to make sure each image matches with the correct data.
Building a self driving RC car will take time, especially if you are trying to make the perfect model. Time must be spent on testing and training so that the car can accurately make predictions and can overcome mistakes or unfamiliar roadblocks.
Raspberry Pi is one of the most simple ways to build a self driving car, but improvements can be made using different servers and neural networks. Pick the neural network you are most comfortable with (or the one you want to learn the most about!).
You will only need a few items and some patience before you get to work on your self driving RC car. Good luck!
Related Questions:
What neural network can you use? Most hobbyists and engineers will use TensorFlow + Keras. TensorFlow is more complex and allows you to fine tune your data as it comes in; Keras is much more simple to use and allows you to make and test a neural network quickly.
But you should use the neural network you feel the most comfortable using or if you are up to even more of a challenge, you could try using a new network to test your skills
What materials do you need? You will need very few items to build your car. The basic items needed are an RC car, a Raspberry Pi board, and a camera. You can get as complex as you want with your car, by adding more cameras or other sensors.
Do I need to use a certain sized car? No, Any size RC or even toy car will work. You will need to buy the right sized camera and make sure your Raspberry Pi can fit on the car, but it doesn’t matter if you use a large RC car or a smaller brand. You can further customize your car by building your own car from scratch.
How do I test my car? You will need a track. You can either find a local track to train on, or you can build your own using tape and chalk in your driveway, garage, or basement.
How long does it take to make a self driving RC car from start to finish? On average this project will take you about 2 months to complete.
Physically putting the car together is very simple and should only take a few hours, but time is needed to train your car and match the trained images.
You should try to get as many images as you can so that your car will not need assistance. Training is the bulk of your work and can take weeks to complete.
This will depend on how automated you want your car to be. If you are just racing it with other cars in a closed circuit track, you won’t need to do as much training work. Your training work can be focused on avoiding obstacles and moving in a straight line.
But if you want your car to be fully automated and recognize traffic signs and make predictions, you will need to train it for any and all obstacles it may come across. You can always retrain your car, but it is much more simple to give it a full training from the beginning.
Once my car is built and trained is it completely self driving? Yes it is, but it may still need a little help from time to time. Like human drivers, self driving cars can make errors. Training never really stops. But in general, for the purposes RC cars fill, your car should be completely self driving once it is trained.
