In “Building mjmech dependencies with bazel“, I described my rationale as it were for attempting to build all of the mjmech dependencies within bazel for cross compilation onto the raspberry pi. mjmech has two big dependencies which were going to cause most of the transitive fallout:
gstreamer – We use gstreamer to interface with the webcam, format RTSP streams for FPV on the control station, and to render the control station and heads up display. Granted, not all of gstreamer is used, but we do depend on features that require ffmpeg and X11.
opencv – The use of opencv had been minimal to non-existant previously, as we hadn’t actually done any computer vision on the robot itself. However, one of the big motivations for switching to the raspberry pi in the first place was to at least to be able to do active target tracking onboard.
And then there are a few other direct dependencies that are “easy”, if nothing else because they have such few transitive dependencies.
boost – The use of boost is almost exclusively the header only parts, boost::date_time, boost::filesystem, boost::program_options, boost::test, and boost::python. Of these, only boost::python has any transitive dependencies.
fmt – This text formatting library has no further dependencies whatsoever.
log4cpp – This is just used for writing textual debug output and has no transitive dependencies.
snappy – We use snappy to compress logged data, but it depends on nothing.
I started with the simple, no dependency packages from the second set. The strategy here is to, for each package, create a tools/workspace/FOO/repository.bzl with a FOO_repository method to download the upstream tarball, and a corresponding tools/workspace/FOO/package.BUILD which contains the bazel BUILD file describing how to build that package.
The first version of the planetary gearbox as 3d printed from Shapeways required a fair amount of post-machining to get all the pieces to fit together. I wanted to get to a point where I could just order some parts and have a reasonable expectation of them mostly working out of the box. To make that happen, I’d need to get a better understanding of where the tolerances were coming from.
Understanding the problem
Shapeways provides a fair amount of documentation on the processes and accuracy you can expect generally. Most of this is detailed in “Design rules and detail resolution for SLS 3D printing“, however the results there have some limitations. Primarily, they are only applicable to the specific geometries tested. Shrinkage is qualified as +- 0.15% of the largest dimension, and is likely influenced by the exact printed geometry. Secondarily, in the documented tests, the designers had full control over the part alignment in the print. The standard shapeways platform does not allow you to orient parts, you are at the whim of their technicians where the Z axis will end up.
For the gearbox, I had numerous fit points that needed to have controlled tolerances. The input and output bearing both needed a press fit for both sides. The internal gear for the planetary gearing needed a press fit, and the front and back shells also have a lip which would be more rigid if the fit was snug.
My solution? Print slight variants of the relevant pieces of each fit point with each radial dimension printed in increments of 0.1mm.
For each part, I broke out the calipers to measure the as printed size, and also attempted manual press fits of each part. I didn’t manage to put any identifying features on each of the prints, which probably annoyed the Shapeways technicians and made my life a bit harder. I just assumed that the sizes came back in increasing order despite the part number markings, which I’m pretty sure were incorrect. This resulted in the following table:
The second version of the gearbox had many other changes in addition to these, but this let me get a lot closer to the correct fit on the full assembly.
Previously, I set up bazel to be able to cross compile for the raspberry pi using an extracted sysroot. That sysroot was very minimal, basically just glibc and the kernel headers. The software used for SMMB has many dependencies beyond that though, including some heavyweight ones such as gstreamer and I needed some solution for building against them.
There were two basic options:
Install all the dependencies I cared about on an actual raspberry pi, and extract them into the sysroot.
Build all the dependencies I cared about using bazel’s external projects mechanism.
The former would certainly be quicker in the short term, at the expense of needing to check in or otherwise version a very large sysroot. It would also be annoying to update, as I would need to keep around a physical raspberry pi and continually reset it to zero in order to generate suitably pristine sysroots.
The second option had the benefit of requiring a small version control footprint — just the URLs and hashes of each project, along with suitable bazel build configuration. It is also perfectly compatible with a fully hermetic build result. However, it had the significant downside that I would need to write the bazel build configuration for all the transitive dependencies of the project.
I decided to take a stab at the second route, partly because of the benefits, but also to see just how hard it would be.
Since bazel does not yet have recursive WORKSPACE parsing I went with a structure that is used in the Drake open source project. The top level project has a tools/workspace directory that contains one sub-directory for each dependency or transitive dependency. Within that directory is a default.bzl that contains one exported function, add_default_repositories. It is intended to be called from the top level WORKSPACE file, and it creates bazel rules for all necessary external dependencies.
The drake project doesn’t support cross compilation, so most of their BUILD rules are of the form, “grab the pre-compiled flags from pkg-config”. However, the same structure will work just fine even with non-trivial compilation steps.
So that this could be easily used across multiple dependent projects, I put all the resultant rules into a new github project: https://github.com/mjbots/bazel_deps Like the drake repository, it has a default.bzl with a single export, add_default_repositories that is used by dependent projects.
Next up, working through actually packaging the dependencies!
In the previous post, I described the motivation for switching the mjmech build system to bazel. For that to be useful with Super Mega Microbot, I first had to get a toolchain configured for bazel such that it could produce binaries that would run on the raspberry pi.
First, I needed to pick the compiler I would be using and how to get the target system libraries available for cross compilation. In the past I’ve always done the gcc/binutils/gnu everything cross toolchain dance, however here I thought I would try something a bit more reproducible and see if I could make clang work. Amazingly, a single clang binary supports all possible target types! clang-6, which can be had through a PPA on Ubuntu 16.04 and natively on Ubuntu 18.04 supports it out of the box.
For the target system libraries, I wrote a small script: make_sysroot.py which when aimed at a live raspberry pi, will extract the Linux kernel headers and glibc headers and binaries. These are stored in a single tar.xz file, with the latest version checked into the tree: 2018-06-10-sysroot.tar.xz.
bazel has a legacy mechanism for configuring toolchains, called the CROSSTOOL file. It is not exactly pretty, is moderately underdocumented, but at least therearea few write ups online for how to create one. The CROSSTOOL I created here is minimally functional, with options for both host/clang and rpi/clang with a few caveats:
It isn’t hermetic, it relies on the system installation of clang
It includes hard coded paths to micro releases of clang
The C++ main function isn’t handled correctly yet, and you have to ‘extern “C”‘ it to link applications
With a functioning CROSSTOOL, the next step is to declare the “cc_toolchain_suite” and “cc_toolchain” definitions. Those are defined in the tools/cc_toolchain/BUILD file and don’t have anything particularly complex in them.
Finally, I created a repository.bzl which is intended to be imported in client projects. This provides a relatively simple API for a client project to import the toolchain, and gets the sysroot extracted properly.
Using it in a bazel project
The top level README gives the steps to use the toolchain, which while not too bad, still requires touching 4 non-empty files (and 2 possibly empty files). Once that is done, you can use:
bazel test --config=pi //...
To build your entire project with binaries targeted to run on the raspberry pi!
The first piece I tackled when switching to the Raspberry Pi 3B+ for Super Mega Microbot was building our existing control software. The software we used for the 2016 Robogames is largely C++ and was built with SCons, https://github.com/mjbots/mjmech/. For all our previous platforms for both Savage Solder and SMMB, we had just built the software on the device itself, which while a little slow, was certainly convenient and required very little sophistication from the build system. Raspian is debian based, so this shouldn’t be hard, right?
“apt” was my friend and at least got the compiler and all the build requirements installed. Then I went to build and discovered that some of our C++ translation units required more than 1G of RAM individually to compile. Ouch. That rendered building on the target with the current software infeasible. Either I could rewrite the software to be less compilation time RAM intensive, or switch to a cross compilation model.
Building on the platform was always somewhat slow, so I decided to try and take the leap and get a proper cross compilation environment set up. I’ve done that in SCons before, but it was never pretty, as SCons doesn’t have the best mechanisms for expressing things that need to be built on the host vs the target, and describing any given cross compilation tool chain is always challenging. Plus SCons itself is slow for any moderately sized project. Instead, I opted to give bazel (https://bazel.build) a try. I’ve used it professionally for a while now, and while it is a mixed bag, I am an eternal optimist when it comes to its feature set. Right now, versus SCons, it can support:
Specifying multiple C++ toolchains
Switching between a single host toolchain and a single target toolchain
Gracefully pulling in external projects controlled by hash
It runs locally very fast, can parallelize across large numbers of cores, and finally has a functioning content addressable cache
The two big features which have seemingly been very close for a long time now, but aren’t quite there yet are:
Arbitrary C++ toolchain configuration within a single build (i.e. build binaries for multiple target platforms as part of a single build)
Next up, I’ll describe how I configured bazel with the cross toolchain necessary to build binaries on the raspberry pi 3 (b+).
One of the major challenges SMMB had in Robogames 2016 was in overall walking speed. It is using HerkuleX DRS-201 servos, which are roughly comparable to the Dynamixel servos that other entrants were using, but the physical geometry of the robot is such that is hard to get it to move quickly with that class of servos. The center of gravity is too high, especially with the gimbal mounted turret. The R-Team bots all use very low slung machines that scoot along. I could go that route, but why do things the easy way?
Instead, I’ve been working to take some ideas from fellow Boston-ite Ben Katz and build some actuators that would permit truly dynamic motion. He got a leg jumping using Hobbyking brushless motors with some simple FOC control The biggest differentiators vs the Dynamixel / HerkuleX class of actuators would be low mechanical inertia, high transient power, low backlash, and high speed. I’m just getting started here, but have managed to build up a 5x planetary gearbox driven by a Turnigy Elite 3508 (so a fair amount smaller than what Ben did, but more appropriately sized for Mech Warfare), and a VESC 6 as an interim motor controller. It is designed for electric skateboards, but has minimal position control support. Although, as my bruised hand can attest, it isn’t super stable and flips out occasionally.
The first prototype is assembled and has been spun up, although a fair amount of dremel time and shims were required to get everything to fit together.
Super Mega Microbot, that beloved and neglected creation, is due for a facelift. The biggest challenge we had at the last competition was the instability of the USB bus on the odroid-U2 when we had both a USB camera and USB 5GHz wifi adapter attached. Cue 2.5 years of waiting, and one aborted attempt, and it looks like the problem is solved!
The aborted attempt
The challenge in this problem is that almost no single board computers in the odroid-ish form factor have both:
a non-USB camera option that works
integrated 5GHz wifi, or any kind of high speed interface that would allow for a non-USB based 5GHz wifi
There are many contenders which have one or the other, or a nominal camera interface, but the board support package that is released to amateurs doesn’t support it. Not only that, almost no boards have any high speed interfaces except USB, which means there aren’t even options for doing anything better.
For a moment though in 2017 I thought I had the problem solved with the introduction of the Intel Joule. On paper it ticked all the boxes, dual camera ports, with software that worked, integrated 5GHz wifi, a supported GPU, and on paper enough of a community that support would be not an issue. The only downside was that as a system on module, it required a fair amount of a carrier board to be able to actually use it in an end application. That said, I did try, and built up a carrier board to be able to mount it in the turret of SMMB.
However, I wasn’t actually able to get the Joule to boot on this carrier board, despite it matching the reference board schematic in every way I could check. To double down on the failure, Intel discontinued the Joule shortly after I had the prototype carrier board in hand which, unsurprisingly, reduced my incentive to try and get it working.
Lo and behold, with sufficient time, comes the announcement of the Raspberry Pi 3 Model B+. On paper it solves nearly every problem as well as the Joule, including needing much less support from a carrier board to be functional.
Onboard 5Ghz wifi
Camera port, with off the shelf camera modules and functional software
Onboard ethernet (although through USB, sigh)
Onboard serial which can run at high data rates (>= 1Mbps)
Stock debian based linux
A production guarantee until 2023!
Not quite as fast as the Joule or Odroid
The GPU doesn’t support any form of GPGPU very easily
Only 1G of RAM
I ordered some and got to work, with results that are definitely more promising, although not without their share of stumbles and pitfalls, and it will definitely take more than one post to describe. So… more for next time.
Earlier in April we took Super Mega Microbot out to California to compete in Mech Warfare during Robogames 2016. Thanks to the R-TEAM organizers who made the event happen this year. We were really excited, and the event as a whole went off really well! There were a lot of functional mechs attending, and many fights that were exciting to watch.
We managed to play 5 official matches, in the double elimination tournament, finishing in 3rd place overall. When it worked, SMMB worked really well. Our first loss was a very close battle, the score keeping system had us winning by 2 and the judges had us losing by 2. (The scoring system wasn’t super reliable, so there were human judges calling hits). Our second loss was caused when the odroid’s USB bus on SMMB stopped working mid-match, causing us to lose camera and wifi.
Since our last matches, we tried to improve a number of things, while some worked, not all of them are entirely successful yet:
Faster walking: The new mammal chassis is about twice as fast as the old lizard one, but we didn’t get much time to make it work really well, so we were still one of the slower mechs at Robogames. Also, the shoulder bracket, even on its second revision, still had several partial failures during matches and will need to be rebuilt in metal to be strong enough.
Stabilized camera: The new gimbal stabilized turret actually worked really well. We were able to reliably hit moving targets from the full length of the arena while in motion. It still has room for improvement, but overall was very reliable.
5GHz Video transport: We updated our video to use a custom protocol over multicast 5GHz wifi, so that we could completely control the amount of link layer retransmissions. When it worked, this worked very well. We were able to get 720p video with 200ms latency, even in the presence of significant interference. However, adding the external 5GHz wifi card to our odroid seems to have made the USB bus overall somewhat unstable, and one of our matches ended prematurely when the entire USB port died, taking our camera and wifi with it.
Thanks to Kevin from R-TEAM, we managed to capture overhead video of all our matches, and have the video as seen on our operator console for each official match as well.
Well, that took longer than I expected! I last showed some progress on a gimbal stabilized turret for Mech Warfare competitions more than six months ago. Due to some unexpected technical difficulties, it took much longer to complete than I had hoped, but the (close to) end result is here!
Here’s a quick feature list:
2 axis control: Yaw and pitch are independently actuated.
Brushless: Each axis is driven by a brushless gimbal motor for high bandwidth no-backlash stabilization.
Absolute encoders: Each axis has an absolute magnetic encoder so that accurate force control can be applied to each gimbal, even at zero speed.
Fire control: High current outputs for driving an AEG motor, an agitator motor, and a low current output for a targetting laser are present.
7v-12V input: Supports 2S-3S lipo power sources.
12V boost: When running from 2S lipo, it can boost the gimbal drive up to 12V for additional stabilization authority.
HerkuleX protocol: The primary control interface uses a native Dongbu HerkuleX protocol; support for other UART based protocols which will work at 3.3V CMOS levels should be easy.
USB debugging support: A USB port is present to return high rate debugging information and allow configuration and diagnostics to be performed.
BMI160: This IMU is used as the primary source of inertial compensation data. The board hardware supports a second IMU, to be placed on the main robot, but the firmware does not yet support that configuration.
This gimbal design contains three custom boards, a breakout board for the BMI160 IMU, a breakout board for the AS5048B magnetic encoder sensor, and the primary board which contains the rest of the logic.
BMI 160 Breakout
The first board is simple; it is a basically just a breakout board for the BMI160 inertial sensor. It provides the BMI160 itself, some decoupling capacitors, and a 0.1 inch 4 pin connector for the I2C bus.
I had these prototypes made at MacroFab which I highly recommend as a great provider of low-cost turnkey PCB assembly.
This, like the BMI160 breakout board, just has decoupling capacitors, the chip itself, and connectors. It additionally has mounting holes designed to fit onto the 3508 gimbal motor. This was printed at OSH Park and hand-assembled.
Gimbal control board
The primary gimbal control board contains most of the system functionality. It is designed to mechanically mount directly above the yaw gimbal motor, as the yaw absolute magnetic encoder is in the center on the underside of the board.
This prototype was also built at MacroFab, who did an excellent job with this much more complex assembly.
The connectors and features are as follows:
Power and Data: A 4 pin JST-XH connector in the upper right brings in power and data from the main robot.
Debug USB: A debugging protocol is available on this micro-USB port.
Camera USB: Two 4 pin JST-PH connectors provide a convenience path for the camera USB. The turret’s camera connects to the top connector, and the main robot connects to the side facing connector.
I2C peripherals: 3, 4 pin JST-ZH connectors have identical pinout and connect to external I2C peripherals. These are used for the primary IMU, the pitch absolute magnetic encoder, and the optional secondary IMU.
Arming switch: This switch is connected directly to the enable pin on the MC33926, and is also connected to an input on the STM32F411.
Programming connector: The 6 pin JST-PH connector has the same pinout as Benjamin Vedder’s VESC board, and can program and debug the STM32F411.
Weapon connector: A 2×4 0.1 inch pin header has power lines for the AEG drive, the agitator drive and the laser. It has an extra row of pins so that a blank can be used for indexing.
Gimbal connectors: 2, 3 pin 0.1 inch connectors power the yaw and pitch gimbal brushless motors.
The firmware was an experiment in writing modern C++11 code for the bare-metal STM32 platform. Each module interacts with others through std::function like callbacks, and the entire system is compiled both for the target, and the host so that unit tests are run. Dynamic memory allocation is this close to being disabled, but it was necessary for newlib’s floating point number formatting routines, which just allocate a chunk of memory the first time you use them. Otherwise, there is no dynamic memory used at all.
It relies on a CubeMX project template for this board. Most of the libraries CubeMX provides have too little flexilibity to be used for this application, so much of the bit twiddling is re-implemented in the gimbal firmware. CubeMX is great for configuring the clock tree and pin alternate functions however, especially in a complex project like this.
Both configuration and telemetry rely on a templated C++ visitor pattern to perform compile time reflection over mostly arbitrary C++ structures. Any module can register a structure to be used for persistent configuration. Those structures can be changed through the debugging protocol, and can be written to and read from flash at runtime. Each module can also register as many telemetry structures as necessary. These can be emitted over the debugging protocol either at fixed intervals, or whenever they update.
There are three possible modes, the first of which is what I call “open-loop”, and is based on the same principles as the BruGi brushless gimbal, where no absolute motor feedback is available. In that mode, a PID controller operates with the axis error as the input, and the output is the actual phase position of the BLDC controller. In this mode, the integral term does most of the work in stabilization, so the overall performance isn’t great.
The second mode still uses a PID controller, but now the output is an offset to the BLDC phase necessary to hold the current position as measured by the absolute encoders. This effectively makes the output a direct mapping to force applied to the motor, although of course a non-linear mapping. This mode results in much better overall performance and is easier to tune.
Finally, there is a third debugging mode that lets you just hard command specific BLDC phases. This is useful for calibrating the mapping between BLDC phase and absolute encoder phase.
The debugging protocol is partially human readable, but telemetry data is encoded in the same binary format as used elsewhere in the mjmech codebase. tview is the debugging application we use to read that data, as well as configure and control the overall system.
The bottom pane just has a serial console, where you can send arbitrary things over the virtual serial port. tview directly supports relatively few commands from the debugging protocol, and for instance has no UI to operate the stabilizer or fire control, so for now these are done by hand in that window.
The left pane has two tabs, one with a configuration tree and the other with a telemetry tree. The configuration tree shows all structures which were registered as configurable, and allows you to change them in the live system. The telemetry tree shows all structures registered as telemetry structures, and reports their values live as the system is operating.
The right pane has a live plot window where any of the values in the telemetry tree can be plotted versus time. It is just an embedded matplotlib plot, so all the normal plot interaction tools are available, plus some from mjmech’s tplot, like the ability to pan and zoom the left and right axes independently.
And last but not least, here is a short video demonstrating the turret stabilizing a camera and firing some blanks at a target as our mech walks around.
I have some incremental progress to report on various parts of Super Mega Microbot. First, I have a draft fully assembled leg for a mammal walking configuration. It is mostly just the stock Dongbu brackets, with a custom Shapeways print at the final joint holding a standoff and rubber stopper.
Second, I’ve been working on a gimbal stabilized turret. I have video from a prior incarnation below:
And a first draft of a 3D printed turret bracket that permits a full range of motion of the turret: