In part 1 I looked at the reasons why we use a state estimation filter for Savage Solder’s localization system and what the options were. This time, I’ll look at the specific measurements we take and the resulting states that we estimate along with some implementation details.

Measurements

Our localization system uses the following measurements:

• GPS: UTM (Universal Transverse Mercator) referenced position from a u-blox 6 GPS sampled at 4Hz
• Gyroscope: We use the yaw rate from a Pololu MiniIMU-9 sampled at 50Hz
• Odometer: We feed the sensors from our sensored brushless motor to an AVR microcontroller and count motor steps to measure odometry at 50Hz. This is not perfect, (for example, it suffers from backlash), but works reasonably well.

Each of these measurements is integrated into the state estimation filter at their natural rate. The filter only emits a solution on each IMU update, just to reduce the variability in the rest of the system.

Estimated States

The Unscented Kalman Filter (UKF) estimates the following system states:

• X/Y Position: The position is estimated in a UTM coordinate reference. We nominally estimate where a position under the center of the rear axle is, but the car is small enough that lever arms don’t really come into consideration.
• Velocity Velocity is estimated assuming no lateral slip.
• Heading: We assume that the car is always perfectly flat, and that the only orientation degree of freedom which matters is heading.
• Curvature: We assume that the car moves in a car like way, and thus, given no change in steering input, it will drive in a circle.
• Odometer Scale Factor: The only error term we estimate for the odometer is a scale factor. This is largely determined by errors in the diameter of the tires, but also is affected by the texture of the terrain that is being traversed.
• Gyroscope Bias: Gyroscopes have many error factors which can be modeled. For our purposes, since we have accurate GPS most of the time, we get away with only estimating the bias of the gyroscope.

Constant Selection and Heuristics

When designing a state estimation filter, there are a lot of constants that need to be selected. For Kalman filters, you need an initial covariance and process noise for all estimated states, and measurement noise for all dimensions of the measured values. By choosing a specific selection of constants, you can trade off between the bandwidth (performance or accuracy) of the filter, and its stability. Generally, if you make the filter more accurate, you increase the risk that the filter will become unstable if the system model or measurement models do not accurately track the real world.

For Savage Solder, we used a largely ad-hoc approach, with no rigorous backing for our constant selection. We picked process noises that roughly corresponded to the amount of system error we expected to see, and did the same with measurement noises. One exception was GPS measurement error. Since we are using a naive total state filter, and GPS measurements from cheap receivers are highly time correlated (because of their own internal estimation filters), there are a number of other factors to consider. If the GPS measurement is set too high in this formulation, heading cannot be estimated accurately. If it is set too low, the global position will jump around a lot with every GPS update. In no case is the filter’s assumption, that of a measurement corrupted purely by white noise, satisfied, so measurements noises that are too low also run the risk of filter divergence. We just picked a happy medium that tends to work well in our simulations and practice.

There are also a fair number of heuristics included. For instance, if we know the car is stopped from the odometer, we artificially fix the heading and curvature estimates and covariances. At least when it is driving under its own power, Savage Solder rarely rotates in place. Without holding these fixed, noise in the GPS and gyroscope can cause the estimated heading to drift with no bounds and the heading uncertainty to grow arbitrarily large.

Global and Local Coordinate System

As I mentioned early in part 1, different subsystems in Savage Solder have differing requirements for the state estimates they consume. Some, like global driving, want the estimate to be close to the globally correct position so that the car can drive close to a pre-surveyed path. Others, like cone tracking, would much rather have a locally consistent frame of reference that is divorced from global coordinates entirely.

We solve this by deriving a separate local coordinate estimate from the global estimate emitted by the UKF. This local estimate always starts at position 0, 0 with a heading of 0. At each time step, the velocity, heading, and curvature of the global solution are integrated forward. This results in an estimate which is by definition smooth, yet still is relatively accurate over short distances, as the velocity and heading inputs have full access to the sensor suite of the car.

Coming Up…

In the next, and final installment, I’ll look at some techniques we are, or want to be exploring to improve the localization solution on Savage Solder.

Today was the first day Savage Solder was firing on all cylinders in our Sparkfun AVC 2013 practice. We ran a pretty large number of circuits, and all systems seemed to be working pretty well, missing the hoop and ramp only a couple of times. Here is a video of a flawless run through the course at 11mph.

When it isn’t crashing into trash cans, Savage Solder uses a combination of several sensors to form accurate estimates about where it is in the world. This series describes the sensors and techniques that it uses to identify where it is in the world, providing data that is useful for the driving and cone tracking subsystems.

Principles

First, why does the car need to know where it is? Several other subsystems in Savage Solder rely on having accurate estimates of where the car is, measured in different ways. First, when driving to try and find a cone, it follows a path surveyed by GPS. In that case, the driving algorithm needs to know where the car is relative to the pre-surveyed path in order to select the appropriate steering and throttle commands. Here, absolute accuracy is most useful. We rely on surveyed paths to avoid obstacles, and the driving algorithm is relatively insensitive to position estimates that move around as GPS measurements change.

When moving in proximity to cones, the car needs a locally stable coordinate system in order to identify where the cone is relative to the car itself. For this, an estimate that is smooth is most important, as the car’s position relative to a cone can only change in a continuous manner.

For both of these cases, the localization subsystem on Savage Solder takes measurements from various sensors, and produces an estimate of the current position of the car. This estimate is updated on a regular basis, and fed to each other subsystem which needs to know where the car is.

Estimation

At the top level, Savage Solder uses a total-state unscented Kalman filter to incorporate measurements and produce localization estimates. Kalman filters are a class of algorithms which, given a series of noisy measurements of a system, estimate what the internal state variables the system are. In the localization problem, the noisy measurements are things like a GPS reading, a gyroscope measurement, or the value from an odometer, while the estimated state is perhaps the latitude and longitude of the vehicle along with its heading, heading rate and speed.

The Unscented Kalman filter is just one type of estimation algorithm. Traditionally, a designer would select it over other non-linear estimators when higher accuracy is required. As compared to estimators like the Extended Kalman Filter, it achieves this higher accuracy through an increase in computational cost.

For Savage Solder, our PC based computer platform has power to spare, so we chose the unscented Kalman filter (UKF) for a different reason. We used the UKF because it only requires numerical system models. Most non-linear Kalman filters, like the EKF, require analytical derivations of the partial derivatives of the system and measurement functions. These can be involved to create. When you want to change your design, and modify the estimated states or measurement variables, the process is slow and error prone. The UKF only requires a numerical definition of the system and measurement functions themselves. Because of this, we can experiment with different filter structures much more rapidly.

Next Installment

In the next installment, I’ll look in more detail at the measurements we use, and the states that we estimate.

We got in our first day of autonomous practice this weekend for the Sparkfun AVC 2013. Some simple issues gave us some problems in the morning, but by the end of the day we had enough either resolved or worked around enough of them that we were achieving pretty good runs on a practice course. I’ve got video of our best run below, but first a couple of caveats:

• We have a debugging creep at the start of the run to verify heading.
• Our practice hoop wasn’t on the course. We don’t yet have a visual detector for the hoop, or our “hoop deflector” mounted on the car.
• There is a debugging pause after it slaloms through the barrels, before it moves on to the ramp.

Enjoy!

Today we took Savage Solder out to take some calibration image data of the various Sparkfun AVC obstacles and to experiment with manual navigation through an AVC style course. Unfortunately, we had some electronics failures with our replacement ESC that will take some repair. Before that though, we managed to collect a fair amount of useful data, and did a few jumps off the ramp. While not impossible, it looks to be pretty challenging. For small cars, the end of the ramp is actually pretty high. For large cars, the ramp is surprisingly narrow and was not trivial to hit when driving manually. This video is of the slowest run we dared do, higher speeds made better landings.

 

Robogames 2013 has come and gone, with all manner of amazing robot mahem unleashed. Unfortunately, Savage Solder didn’t have the best performance in Robomagellan, hitting a trash can on its first run, and a newly added picnic table on its second. Our motor controller popped after that run, so we didn’t get a third attempt. Hopefully I will post a more detailed post-mortem later, but for now, I’ve uploaded the video of our best competition performance this year.

The final pieces of the cone detection and tracking system for Savage Solder are the tools we used to derive all the constants necessary for each of the algorithms. In part 1 (cone detector) and part 2 (cone tracker) I described how we first pick out possible range and bearings to cones in an image, and then take those range and bearings and turn them into a local Cartesian coordinate through the tracking process. As mentioned there, each of those stages has many tunable knobs.For the first stage, the following are the key parameters:

• Hue range - The window of hues to consider a pixel part of a valid cone.
• Saturation range - The range of saturation values to include.
• Minimum and maximum aspect ratio - Valid cone like objects are expected to be moderately narrow and tall.
• Minimum and maximum fill rate - If the image is crisp, most of the pixels in the bounding box, (but not all) should meet the filtering criteria.

Additionally, the cone tracker has its own set of parameters:

• Detection rate - given a real world cone at a certain distance, how likely are we to detect it?
• False positive rate - given a measurement at a specific range, how likely is it to be false?
• Range and bearing limits - At what range and bearing should we reject measurements.

What we did in 2012, during the preparation for our first RoboMagellan, was take a lot of pictures of cones during our practice runs. Savage Solder normally is configured to save an image twice a second all the time it is running. These image datasets formed the basis of our ability to tune parameters and develop algorithms that would be robust in a wide range of conditions.

The basic idea we worked off was to create a metric for how good the system is, and then evaluate the metric over a set of data using our current algorithms and constants. You can then easily tweak the constants and algorithms as much as you want without running the car one bit and have good confidence that the results will be applicable to actual live runs.

Most of the metrics we wanted involved knowing where the cones actually were in the images. If we could just robustly identify cones in images programmatically, we wouldn’t really be worrying about this to begin with. So instead, we created a set of tools that let us rapidly mark, or annotate, images to indicate where the absolute ground truth of cones could be found. This is just a simple custom OpenCV program with some keyboard shortcuts that let us classify the cones in about 3000 images in a couple of hours. We selected images from varying times of day, angles, and lighting conditions, so that we would have a robust training set.

Then, with a little wrapper script, we ran our cone detection algorithm over each of the frames. As mentioned in the on the cone detector, it outputs one or more range/bearing pairs to each of the prospective cones in the image. The quality metric then scores the cone detector based on accuracy of bearing and range to real cones, missed detections, and false detections. It also keeps histograms of each detection category by range. In the end, for a given set of cone detector parameters, we end up with a table that looks like the one below.

Range	Detection Rate	False Positive Rate
3m	91%	2%
5m	95%	7%
7.5m	93%	21%
10m	89%	34%
15m	72%	26%
20m	37%	23%

After each run over all the images, we would try changing the parameters to the cone detector, then seeing what the resultant table would like. Ideally, you have a much higher detection rate than false positive rate over the ranges you care about. The table above is actually the final one we were able to achieve for 2012, which allowed us to reliably sense cones at 15 meters of range using just our 640x480 stock webcam.

We got our final day of testing in at Danehy park in preparation for Robogames 2013 this weekend. Everything worked as well as we expected, the only things that would have caused problems on race day were trying to fit through areas too narrow for the basic GPS we were getting at the time. All in all, our speed, cornering, and tracking performance are much improved over 2012.

Along with many others, we’re hoping to enter our car into the Sparkfun AVC 2013 this year! In support of our effort, I put together some datasets and 3D assets that might be useful to other participating teams, especially if they do any work in simulation.

Georeferenced Data

First, I used the most recent measurements from April 9th on the AVC website to create a kml file showing where each of the obstacles is, along with the course boundaries. I’ve included a snapshot from Google Earth below to show what it contains.

Next, I extracted some freely available USGS ortho imagery and created a cropped version covering just the course. I essentially used the techniques described in my post on LIDAR imagery, except that the ortho imagery came from the USGS National Map Viewer. I created both a geotiff that has georeferencing information, and an opaque .png which is projected identically, but has no embedded information. On that last image, I did a little gimp hackery to edit out cars which would be in the path of the actual course.

• sfe-avc2013-course-ortho.tif - geotiff
• sfe-avc2013-course-ortho-cleaned.png - PNG with cars removed

3D Models

I also made a set of somewhat (OK very) crude blender models of each of the different obstacles present on the course to approximately correct dimensions. This includes the barrels, the hoop, and the ramp.

avc2013-barrel.blend	avc2013-hoop.blend	avc2013-ramp.blend

Drivethrough Video

Finally, just for fun, I recorded a screencast of one of the first drivethroughs of Savage Solder on the simulated Sparkfun AVC Course. This was with just the bare minimum of changes from what we’ll use at RoboMagellan in Robogames in two weeks, so for now it was navigating purely by GPS and IMU. It drives through the hoop about one time in three, which is enough to catch a good video! As in my previous simulation videos, the yellow line is where the absolute surveyed GPS path is. The green line is where the car thinks the GPS path is at any moment, which will differ due to GPS error and heading error.

 

 

Last time I covered the techniques we use on Savage Solder to pick out orange traffic cones from webcam images in the “Cone Detector”. This time, I’ll look at the next stage in that pipeline, turning the range and bearing reported by the cone tracker into Cartesian coordinate estimates of where any given cone is.

Cone Tracker Overview

As mentioned last time, our “Cone Tracker” takes as input the range and bearing, (along with their uncertainties), from the cone detector. It also receives the current navigation state of the car. This includes things like the current estimate of geodetic position (latitude and longitude), current map position, (UTM easting and northing), and an unreferenced local coordinate system (x, y). For each of these, it reports the vehicle’s speed and heading.

I won’t go into the details of each of these coordinate systems here, but since the cone tracker actually only uses the local one, I can discuss it a bit. The local coordinate system on Savage Solder starts out at x,y=(0m,0m) with a heading of 0 degrees at the beginning of every run. As time progresses, the heading and position are updated primarily based on our onboard dead reckoning sensors. The local coordinate system is smooth, and never jumps for GPS updates. As a consequence, it isn’t really useful to compare positions in it from more than a minute or two apart, nor is it useful to do GPS based navigation. Fortunately, for cone tracking, from the time to when we see the cone to when we touch it is usually only a couple of seconds total over which time the local solution is very accurate.

Now, let’s cover some broad details of the implementation. The guts of our cone tracker consists of a bank of identical Kalman Filters. Each Kalman filter estimates the position of one particular cone and the error in that estimate. This lets the car keep around several possible cones that could be in sight at one time while still distinguishing them. By storing the cones in a local coordinate system, it allows for easy driving towards, or alongside, the cone and accurate speed control leading up to it. The position uncertainty could be used to control behavior, but we don’t bother in our current implementation.

New and Old Cones

Additionally, the tracker has to handle seeing new cones for the first time, and discarding cones that maybe were false detections in the cone detector. It does this by assigning each cone a “likelihood”, which is just a probability that the cone is a true detection. When data arrives that match the cone well, its likelihood is increased. When the available data doesn’t match the cone very well, or no cone is observed at all when one is expected, the likelihood is decreased.

If a range and bearing arrive which corresponds to no known cones, a new one is created with a relatively low likelihood. Then once it has reached a certain threshold, it is reported to the outside world as a valid target. Conversely, when an existing cone’s likelihood reaches a level which is too low, it is removed entirely on the thesis that it was actually a false detection to begin with.

More specifically, the likelihood update is handled using Bayes theorem. We have an empirically derived table showing, for our detector, the odds that a cone will be detected or a false one will be seen at various ranges. These are used to fill in the various probabilities in the equations.

Incorporating New Measurements

A “measurement” in this problem is simply the range and bearing that the cone detector reports. To incorporate a new measurement, the tracker looks through all the cones currently in its filter bank. For each, it computes a measure of the likelihood that the given cone could produce the current measurement. This is done using what’s called the Mahalanobis distance. The Mahalanobis distance is just a measure of how far away you are from an uncertain target in a multi-dimensional space.

If the best matching cone has a Mahalanobis distance small enough to count as valid, then the standard Kalman filter update equation is applied to that cone. If no cones match, then we create a new one as described above.

Scale Factor

One final detail, is that in addition to estimating the position of each cone, the tracker also estimates its “scale” as seen by the cone detector. The image based detector we use has the property that the range values are likely to have a fixed scale error for a number of reasons. One, the cones could actually be bigger or smaller than the regulation sized ones. Second, lighting conditions can sometimes cause a fraction of the cone to be not detected, which will result in the cone being seen as smaller than it actually is.

Since the range values are not correct, the position will be similarly off. This error isn’t as bad as it seems, since we (and most every RoboMagellan entrant), approach the cone in a straight line. Thus the position errors will always be directly towards or away from the robot, and as long as you keep moving, you’ll hit it eventually.

Savage Solder has two reasons to care. First, is that we decelerate from a high speed to an appropriate “bumping” speed with just a half meter to spare. Thus, if the cone is estimated as too close, we run the risk of colliding with it at an unsafe speed and damaging our front end. Secondly, we have a mode where the car can just “clip” the cone by driving alongside it and tapping it with a protruding stick. This allows us to avoid stopping entirely when the geometry allows it. However, if the position estimate is incorrect here, we might miss the cone entirely.

Final Process

To summarize, the cone tracker handles each new range and bearing with the following steps:

1. A cone detection results in a new range, bearing, and uncertainty measurement.
2. Existing cones in the Kalman filter bank are checked to see what the likelihood is each could have produced this measurement.
3. If one was likely enough, then:
   1. The position and scale factor are updated using the Kalman filter equations.
   2. The likelihood estimate is updated according to our current false positive and false negative estimates.
4. If none was likely enough, then a new cone is created with a position fixed at the exact position implied by the local navigation solution and the range and bearing.
5. Any cones which we had expected to see but didn’t have their likelihood decreased.
6. Any cones which have too low of a likelihood are removed.
7. Finally, all the cones which are close to the vehicle, and have a high enough likelihood are reported to the high level logic.

Caveats and Next Steps

One of the challenges with this approach is that there are a lot of constants to tune. I’ll cover the details in a later post, but for most of them we tried to find some way to empirically measure reasonable values for our application. Another problem was debugging. printf doesn’t cut it when you have this much geometry. For that, we created a number of custom debugging and visualization tools which help show what is going on in the system, some of which I’ll cover later too.