Fun intro to electronics for me. The robot was made by me and my dodgy engineering skills and was built to a budget.
The robot doesn't quite achieve its goal of killing weeds as the weed identification model isn't quite there and I haven't got round to adding perimeter wire.
An MVP that kills weeds but doesn't do it systematically i.e. goes in random directions, wouldn't be too far away but I don't currently have the time to pursue this.
'Finished' product
'wait 3' is very important
Beginnings of the robot
Calibrating the magnetometer. I drove the robot around in circles and recorded the x, y, z values for the magnetic field. To be useful for robot navigation, the measurements should be centred around zero and the magnet field should be even in all directions (so I'm measuring the Earth's field and not the magnets on board the robot). To calibrate it (i.e. find constants that you mix with the raw readings to get 'calibrated' readings) I used the Kåsa circle fitting algorithm on the points (I didn't use z readings).
The robot should use the ultrasound turret to detect the bin, move out the way and then return back on course on the same straight line in the same direction. The highlighters mark the positions that the robot was facing when it finished this task the previous four times from starting in the same position. Consistency of movement was a large struggle during this project.
Trying to make the robot turn 360 degrees reliably was a bit of a challenge. I think I achieved it reliably on this surface but it wasn't reliable on the patio.
Wiring schematic for the Pi Pico used to control the autonomous movement.
Camera on the robot pointed at my phone screen.
Scale diagram of the 'floor plan' of the top layer of the robot. The large circle is the weed killer reservoir.
The enemy. This is what most photos in my dataset look like.
Testing some version of the weed spotting CNN. The correct/wrong is whether the model correctly predicted that there was a weed in the photo above. For photos with weeds, there is a green cross for the labelled position of the weed and a red cross for where the model predicted the weed was. The model still predicts the location even if it didn't think it was probable that there was a weed in the photo. In these testing samples, the model does appear to get some stuff correct.
An image in the patio category of MIT's Places365 dataset. This dataset contained the largest collection of patio photos I could find and it had 5,000 images of them. I tried to use this and a dataset of weeds as transfer learning for my own dataset. As you might imagine, classification of this dataset vs weeds is a lot easier than my own dataset and doesn't train the model in a particularly helpful way.
The main source of inspiration for this project was autonomous lawnmowers. The have a similar high level goal but are just missing the identification and targeted shooting.
Autonomous lawnmowers are a relatively modern thing with only the newer ones being able to follow routes (TODO check) but these use GPS RTK which is a bit out of my budget.
As my domain of operation was more limited (a relatively flat patio), I thought that I could get away with dead reckoning. Turns out I couldn't.
Dead reckoning is notoriously rubbish but I thought that with enough sensors it would be close enough.
The robot used: wheel encoders, a magnemometer, a gyroscope and ultrasound. It fused these measurements together to calculate heading and distance travelled via a Kalman filter
TODO check and rewrite
Treats readings as samples from a normal distribution, multiplies together the distributions + reliability to get best estimate of true distribution
. Based on these measurements, to go to a specific location, it uses PID
Proportional, integral, derivative. Smoothly (e.g. no overshooting) achieves a desired outcome value by changing input values
After much fiddling (changing motors, changing wheels, tuning constants), I couldn't get reliable turns and so the error accumulates and the robot can no longer go in straight lines.
Pictures of highlighters.
Machine learning on custom dataset
I needed an algorithm that could identify the positions of weeds. A simple checking of green pixel doesn't work as all the grass ends up dead and so I needed a machine learning model.
As it's relatively complex and image-based, a CNN was probably the right tool and handily I have my own https://alistair.hanlonhome.co.uk/cnn (which this does use). I couldn't find a weed-on-patio dataset and so I made by own by driving the robot around manually and taking photos.
The dataset is relatively sensible, it doesn't expect anything out of reach for a human and it includes a mix of weed and non-weed containing photos along with important non-weed cases e.g. lawn.
I had around 2,000 photos of my patio and used extensive image augmentation. However, I think the patterns were too complex for the limited number of unique weeds (my patio isn't that bad) in the dataset.
I managed to make the model overfit but couldn't ever get it to generalise to the test dataset. The model could easily do a simpler "plant vs outdoor patio area" dataset but couldn't do my garden.
I also tried transfer learning from this dataset and this didn't help. The main takeaway is that a machine learning model aims to minimise its loss. If learning what you what it to learn is not the easiest way to minimise loss, it will not do this.
Android app on BLE
To control the robot, I made an Android app which connects to the Pi Pico via BLE. On Android this isn't too hard but trying to it on the Pico in C with limited documentation was a bit of a pain.
The app has throttle controls for the motors, commands for the robot e.g. autonomous mode, and a livestream of the onboard camera.
The Pi Zero streams the video through RTSP and then also hosts a HTTP server which allows the app to send requests to save photos on the Pi (useful for dataset collection).
(This is before I realised you had to manually focus the camera)
It's quite fun actually making stuff
Having something tangible is quite nice. Especially fun if you're under a budget as you end up with some funny bodges:
Trying to glue tiny magnets to the wheels was a bit finicky (metal tweezers weren't particularly helpful). These were for recording the speed of the wheel (using a hall effect sensor) for dead reckoning.
I thought that the stepper motor was significantly interfering with the magnetometer readings. Luckily, it wasn't.
Might not have measured properly. Electrical tape sorted this out.
Lego contraption to calibrate the ultrasound turret direction. The ultrasound turret turns left and right until it sees an object within 15cm and then sets that as the forward facing position.
Wire mess.
Wheel magnets round 2. I needed higher accuracy and more responsive speed velocity measurements and so needed more magnets. The new strategy was to place them evenly spaced on double sided tape and then transfer this to wheel and glue it all down and peel off the tape.
You also learn stuff that you wouldn't encounter from just writing software.
Some of the interesting things I learnt from doing more hardware stuff was:
I2C, SPI, pull up resistors, PWM, soldering, magnetometer calibration,
It is a bit more expensive than software and cost around £106.54 (I thought it was cheaper than that) but if you didn't have to prototype, it could be done for cheaper.
But it does have two computers, motors, and a camera on it along with other gizmos.
I don't currently have the time to pursue this but if I did, the steps needed to make it work would be
Add perimeter wire (as robot lawn mowers use) which has a square wave which the robot detects (by inducing a current) to prevent the robot running away
Either improve the weed dataset or accept the fact that the dataset will have to be simpler
Accept that without GPS RTK, the robot will not be able to move systematically and change the autonomous mode to account for this e.g. obstacle avoidance.
