In the previous post, I described the motivation for switching the mjmech build system to bazel. For that to be useful with Super Mega Microbot, I first had to get a toolchain configured for bazel such that it could produce binaries that would run on the raspberry pi.
First, I needed to pick the compiler I would be using and how to get the target system libraries available for cross compilation. In the past I’ve always done the gcc/binutils/gnu everything cross toolchain dance, however here I thought I would try something a bit more reproducible and see if I could make clang work. Amazingly, a single clang binary supports all possible target types! clang-6, which can be had through a PPA on Ubuntu 16.04 and natively on Ubuntu 18.04 supports it out of the box.
For the target system libraries, I wrote a small script: make_sysroot.py which when aimed at a live raspberry pi, will extract the Linux kernel headers and glibc headers and binaries. These are stored in a single tar.xz file, with the latest version checked into the tree: 2018-06-10-sysroot.tar.xz.
bazel has a legacy mechanism for configuring toolchains, called the CROSSTOOL file. It is not exactly pretty, is moderately underdocumented, but at least therearea few write ups online for how to create one. The CROSSTOOL I created here is minimally functional, with options for both host/clang and rpi/clang with a few caveats:
It isn’t hermetic, it relies on the system installation of clang
It includes hard coded paths to micro releases of clang
The C++ main function isn’t handled correctly yet, and you have to ‘extern “C”‘ it to link applications
With a functioning CROSSTOOL, the next step is to declare the “cc_toolchain_suite” and “cc_toolchain” definitions. Those are defined in the tools/cc_toolchain/BUILD file and don’t have anything particularly complex in them.
Finally, I created a repository.bzl which is intended to be imported in client projects. This provides a relatively simple API for a client project to import the toolchain, and gets the sysroot extracted properly.
Using it in a bazel project
The top level README gives the steps to use the toolchain, which while not too bad, still requires touching 4 non-empty files (and 2 possibly empty files). Once that is done, you can use:
bazel test --config=pi //...
To build your entire project with binaries targeted to run on the raspberry pi!
One of the major challenges SMMB had in Robogames 2016 was in overall walking speed. It is using HerkuleX DRS-201 servos, which are roughly comparable to the Dynamixel servos that other entrants were using, but the physical geometry of the robot is such that is hard to get it to move quickly with that class of servos. The center of gravity is too high, especially with the gimbal mounted turret. The R-Team bots all use very low slung machines that scoot along. I could go that route, but why do things the easy way?
Instead, I’ve been working to take some ideas from fellow Boston-ite Ben Katz and build some actuators that would permit truly dynamic motion. He got a leg jumping using Hobbyking brushless motors with some simple FOC control The biggest differentiators vs the Dynamixel / HerkuleX class of actuators would be low mechanical inertia, high transient power, low backlash, and high speed. I’m just getting started here, but have managed to build up a 5x planetary gearbox driven by a Turnigy Elite 3508 (so a fair amount smaller than what Ben did, but more appropriately sized for Mech Warfare), and a VESC 6 as an interim motor controller. It is designed for electric skateboards, but has minimal position control support. Although, as my bruised hand can attest, it isn’t super stable and flips out occasionally.
The first prototype is assembled and has been spun up, although a fair amount of dremel time and shims were required to get everything to fit together.
Super Mega Microbot, that beloved and neglected creation, is due for a facelift. The biggest challenge we had at the last competition was the instability of the USB bus on the odroid-U2 when we had both a USB camera and USB 5GHz wifi adapter attached. Cue 2.5 years of waiting, and one aborted attempt, and it looks like the problem is solved!
The aborted attempt
The challenge in this problem is that almost no single board computers in the odroid-ish form factor have both:
a non-USB camera option that works
integrated 5GHz wifi, or any kind of high speed interface that would allow for a non-USB based 5GHz wifi
There are many contenders which have one or the other, or a nominal camera interface, but the board support package that is released to amateurs doesn’t support it. Not only that, almost no boards have any high speed interfaces except USB, which means there aren’t even options for doing anything better.
For a moment though in 2017 I thought I had the problem solved with the introduction of the Intel Joule. On paper it ticked all the boxes, dual camera ports, with software that worked, integrated 5GHz wifi, a supported GPU, and on paper enough of a community that support would be not an issue. The only downside was that as a system on module, it required a fair amount of a carrier board to be able to actually use it in an end application. That said, I did try, and built up a carrier board to be able to mount it in the turret of SMMB.
However, I wasn’t actually able to get the Joule to boot on this carrier board, despite it matching the reference board schematic in every way I could check. To double down on the failure, Intel discontinued the Joule shortly after I had the prototype carrier board in hand which, unsurprisingly, reduced my incentive to try and get it working.
Lo and behold, with sufficient time, comes the announcement of the Raspberry Pi 3 Model B+. On paper it solves nearly every problem as well as the Joule, including needing much less support from a carrier board to be functional.
Onboard 5Ghz wifi
Camera port, with off the shelf camera modules and functional software
Onboard ethernet (although through USB, sigh)
Onboard serial which can run at high data rates (>= 1Mbps)
Stock debian based linux
A production guarantee until 2023!
Not quite as fast as the Joule or Odroid
The GPU doesn’t support any form of GPGPU very easily
Only 1G of RAM
I ordered some and got to work, with results that are definitely more promising, although not without their share of stumbles and pitfalls, and it will definitely take more than one post to describe. So… more for next time.
Earlier in April we took Super Mega Microbot out to California to compete in Mech Warfare during Robogames 2016. Thanks to the R-TEAM organizers who made the event happen this year. We were really excited, and the event as a whole went off really well! There were a lot of functional mechs attending, and many fights that were exciting to watch.
We managed to play 5 official matches, in the double elimination tournament, finishing in 3rd place overall. When it worked, SMMB worked really well. Our first loss was a very close battle, the score keeping system had us winning by 2 and the judges had us losing by 2. (The scoring system wasn’t super reliable, so there were human judges calling hits). Our second loss was caused when the odroid’s USB bus on SMMB stopped working mid-match, causing us to lose camera and wifi.
Since our last matches, we tried to improve a number of things, while some worked, not all of them are entirely successful yet:
Faster walking: The new mammal chassis is about twice as fast as the old lizard one, but we didn’t get much time to make it work really well, so we were still one of the slower mechs at Robogames. Also, the shoulder bracket, even on its second revision, still had several partial failures during matches and will need to be rebuilt in metal to be strong enough.
Stabilized camera: The new gimbal stabilized turret actually worked really well. We were able to reliably hit moving targets from the full length of the arena while in motion. It still has room for improvement, but overall was very reliable.
5GHz Video transport: We updated our video to use a custom protocol over multicast 5GHz wifi, so that we could completely control the amount of link layer retransmissions. When it worked, this worked very well. We were able to get 720p video with 200ms latency, even in the presence of significant interference. However, adding the external 5GHz wifi card to our odroid seems to have made the USB bus overall somewhat unstable, and one of our matches ended prematurely when the entire USB port died, taking our camera and wifi with it.
Thanks to Kevin from R-TEAM, we managed to capture overhead video of all our matches, and have the video as seen on our operator console for each official match as well.
Well, that took longer than I expected! I last showed some progress on a gimbal stabilized turret for Mech Warfare competitions more than six months ago. Due to some unexpected technical difficulties, it took much longer to complete than I had hoped, but the (close to) end result is here!
Here’s a quick feature list:
2 axis control: Yaw and pitch are independently actuated.
Brushless: Each axis is driven by a brushless gimbal motor for high bandwidth no-backlash stabilization.
Absolute encoders: Each axis has an absolute magnetic encoder so that accurate force control can be applied to each gimbal, even at zero speed.
Fire control: High current outputs for driving an AEG motor, an agitator motor, and a low current output for a targetting laser are present.
7v-12V input: Supports 2S-3S lipo power sources.
12V boost: When running from 2S lipo, it can boost the gimbal drive up to 12V for additional stabilization authority.
HerkuleX protocol: The primary control interface uses a native Dongbu HerkuleX protocol; support for other UART based protocols which will work at 3.3V CMOS levels should be easy.
USB debugging support: A USB port is present to return high rate debugging information and allow configuration and diagnostics to be performed.
BMI160: This IMU is used as the primary source of inertial compensation data. The board hardware supports a second IMU, to be placed on the main robot, but the firmware does not yet support that configuration.
This gimbal design contains three custom boards, a breakout board for the BMI160 IMU, a breakout board for the AS5048B magnetic encoder sensor, and the primary board which contains the rest of the logic.
BMI 160 Breakout
The first board is simple; it is a basically just a breakout board for the BMI160 inertial sensor. It provides the BMI160 itself, some decoupling capacitors, and a 0.1 inch 4 pin connector for the I2C bus.
I had these prototypes made at MacroFab which I highly recommend as a great provider of low-cost turnkey PCB assembly.
This, like the BMI160 breakout board, just has decoupling capacitors, the chip itself, and connectors. It additionally has mounting holes designed to fit onto the 3508 gimbal motor. This was printed at OSH Park and hand-assembled.
Gimbal control board
The primary gimbal control board contains most of the system functionality. It is designed to mechanically mount directly above the yaw gimbal motor, as the yaw absolute magnetic encoder is in the center on the underside of the board.
This prototype was also built at MacroFab, who did an excellent job with this much more complex assembly.
The connectors and features are as follows:
Power and Data: A 4 pin JST-XH connector in the upper right brings in power and data from the main robot.
Debug USB: A debugging protocol is available on this micro-USB port.
Camera USB: Two 4 pin JST-PH connectors provide a convenience path for the camera USB. The turret’s camera connects to the top connector, and the main robot connects to the side facing connector.
I2C peripherals: 3, 4 pin JST-ZH connectors have identical pinout and connect to external I2C peripherals. These are used for the primary IMU, the pitch absolute magnetic encoder, and the optional secondary IMU.
Arming switch: This switch is connected directly to the enable pin on the MC33926, and is also connected to an input on the STM32F411.
Programming connector: The 6 pin JST-PH connector has the same pinout as Benjamin Vedder’s VESC board, and can program and debug the STM32F411.
Weapon connector: A 2×4 0.1 inch pin header has power lines for the AEG drive, the agitator drive and the laser. It has an extra row of pins so that a blank can be used for indexing.
Gimbal connectors: 2, 3 pin 0.1 inch connectors power the yaw and pitch gimbal brushless motors.
The firmware was an experiment in writing modern C++11 code for the bare-metal STM32 platform. Each module interacts with others through std::function like callbacks, and the entire system is compiled both for the target, and the host so that unit tests are run. Dynamic memory allocation is this close to being disabled, but it was necessary for newlib’s floating point number formatting routines, which just allocate a chunk of memory the first time you use them. Otherwise, there is no dynamic memory used at all.
It relies on a CubeMX project template for this board. Most of the libraries CubeMX provides have too little flexilibity to be used for this application, so much of the bit twiddling is re-implemented in the gimbal firmware. CubeMX is great for configuring the clock tree and pin alternate functions however, especially in a complex project like this.
Both configuration and telemetry rely on a templated C++ visitor pattern to perform compile time reflection over mostly arbitrary C++ structures. Any module can register a structure to be used for persistent configuration. Those structures can be changed through the debugging protocol, and can be written to and read from flash at runtime. Each module can also register as many telemetry structures as necessary. These can be emitted over the debugging protocol either at fixed intervals, or whenever they update.
There are three possible modes, the first of which is what I call “open-loop”, and is based on the same principles as the BruGi brushless gimbal, where no absolute motor feedback is available. In that mode, a PID controller operates with the axis error as the input, and the output is the actual phase position of the BLDC controller. In this mode, the integral term does most of the work in stabilization, so the overall performance isn’t great.
The second mode still uses a PID controller, but now the output is an offset to the BLDC phase necessary to hold the current position as measured by the absolute encoders. This effectively makes the output a direct mapping to force applied to the motor, although of course a non-linear mapping. This mode results in much better overall performance and is easier to tune.
Finally, there is a third debugging mode that lets you just hard command specific BLDC phases. This is useful for calibrating the mapping between BLDC phase and absolute encoder phase.
The debugging protocol is partially human readable, but telemetry data is encoded in the same binary format as used elsewhere in the mjmech codebase. tview is the debugging application we use to read that data, as well as configure and control the overall system.
The bottom pane just has a serial console, where you can send arbitrary things over the virtual serial port. tview directly supports relatively few commands from the debugging protocol, and for instance has no UI to operate the stabilizer or fire control, so for now these are done by hand in that window.
The left pane has two tabs, one with a configuration tree and the other with a telemetry tree. The configuration tree shows all structures which were registered as configurable, and allows you to change them in the live system. The telemetry tree shows all structures registered as telemetry structures, and reports their values live as the system is operating.
The right pane has a live plot window where any of the values in the telemetry tree can be plotted versus time. It is just an embedded matplotlib plot, so all the normal plot interaction tools are available, plus some from mjmech’s tplot, like the ability to pan and zoom the left and right axes independently.
And last but not least, here is a short video demonstrating the turret stabilizing a camera and firing some blanks at a target as our mech walks around.
I have some incremental progress to report on various parts of Super Mega Microbot. First, I have a draft fully assembled leg for a mammal walking configuration. It is mostly just the stock Dongbu brackets, with a custom Shapeways print at the final joint holding a standoff and rubber stopper.
Second, I’ve been working on a gimbal stabilized turret. I have video from a prior incarnation below:
And a first draft of a 3D printed turret bracket that permits a full range of motion of the turret:
We just concluded the first ever Boston Mech Warfare Expo this weekend, held at Jaybridge Robotics in Cambridge. We had a lot of exciting mech fighting action, learned a lot, and even had a fair amount of exciting repairs! There are many hours of video I need to sort through, but in the meantime, here are some photos:
To act as a forcing function in our development of Super Mega Microbot, we put together a Mech Warfare style qualification video, showing it driving around for more than 5 minutes shooting at some targets. Here it is!
I’ve also put together a short demonstration video, showing how to download, install, and use legtool with a gazebo simulated quadruped.
In lieu of actual documentation (or possibly to start generating the text of some), I’ve written some text describing what legtool does and how to to use it below.
legtool overview and installation
legtool is a python library and PySide based graphical application useful for pose generation, inverse kinematics, and gait generation for legged robots. It consists of a set of python libraries, and a graphical debugging application which uses those libraries to allow a developer to explore legged locomotion. It has primarily been tested on Ubuntu linux, but may be portable to other operating systems.
legtool is not yet packaged for installation, so all you need to do is clone the github repository and install the dependencies as documented in the README.
git clone https://github.com/jpieper/legtool.git
Legtool by default creates a configuration file in ~/.config, however, you can specify an alternate configuration on the command line using the -c option. If you have multiple robots, or a simulation and an actual robot, you will want to keep multiple configurations around.
The first tab which is active, the “servo” tab, contains controls to select the servos to control and configure them, along with controls to maintain a list of named poses, where each pose consists of a set of servo positions.
Here you can move servos individually to verify their operation, and compose poses. The only use for poses currently is in the later inverse kinematics tabs, so there is no need to define more than idle, minimum, and maximum.
Inverse kinematics tab
The second tab, the IK tab, lets you configure which servos are in which leg, what sign they have, and also the geometry of the legs. The lower right corner visualizes the achievable region of operation in 2D. You can select alternate projections, and change the centerpoint of the test using the IK Offset group. Clicking on the rendering will command that leg to that position.
The third, and currently final tab, contains controls to allow configuring the placement of all shoulder joints, marking the center of gravity, and configuring the gait engine. It provides several debugging facilities to allow visualizing the results of the gait engine.
The geometry configuration is in the upper left. The shoulder position can be set individually for each leg. The center of gravity and idle position are common to all legs. The gait configuration area in the lower left selects which gait to use, and then exposes a number of configuration options for that gait.
The remainder of the tab is devoted to interactive tools for debugging the gaits:
Gait graph: The gait graph in the upper middle shows for the duration of a single cycle, when each of the legs is in the air.
Geometry view: Visible in the upper right, this shows a 2D rendering of where the shoulders, legs, body, center of gravity, and support polygon are located.
Graphical command view: Visible in the lower right, the graphical command view shows which commands are feasible given the current geometry and gait configuration. Any two command axes can be selected, after which the feasible region is shown in green, regions where the speed is limited are in yellow, and totally infeasible regions are in red.
Playback: The playback scrubber allows you to slide back and forward through a gait cycle, visualizing the results in the geometry view or on the simulated or real robot.
Textual command area: All of the 9 degrees of command freedom are settable here, in addition to the two available in the graphical command view.