Roobt the robot, detecting license plates.
ENPH 353 Software Project Course - Winter 2021
Goal
In this project course, we were provided with a Gazebo simulation with roads, pedestrians and parked blue “cars” with license plates. In a team of two, we had to program a driving robot in Python to detect as many plates as possible and read them correctly using a neural network.
Result
You can see our robot in action and some license plate predictions here:
Driving
We decided to go with PID to keep things simple for the driving. My teammate took care of all the driving and pedestrian detection code. He used HSV thresholding with a polygon mask for road detection. It worked really well! 🚗
Plate detection
We used HSV thresholding with blurring, erosion and dilation to detect specific contours above and below the license plates. Using the corners of these contours, we cropped the plates and applied a perspective transform to straighten them.
Then, we applied a different HSV thresholding on these cropped plates to detect the characters. Using the center of mass of each character, we cropped them easily and sent them to our neural network for prediction.
Neural network to read plates
Type: Convolutional
Module: tensorflow.keras
Training data: characters from a provided license plate generator + cropped characters captured by our robot.
We transformed the generated characters using ImageDataGenerator from tf.keras.preprocessing.image.
First tries:
The first models we trained were only predicting a few plates correctly. Some letters were read as numbers and vice-versa.
How we improved our results:
- Downsize and re-upsize generated characters to better reproduce the low quality of simulation characters.
- Increase the number of characters captured in the simulation (vs generated) for training.
- Train the captured characters and generated characters separately to control the number of epochs and avoid overfitting.
- Have two models: one for letters and one for numbers.