It seems that I’m learning much about PCB design the very hard way. Back last year I wrote up my discovery of MLCC bias derating. Now I’ll share some of my experiences with MLCC cracking on the first production moteus controllers.
When I was first putting the production moteus controllers through their test and programming sequence, I observed a failure mode that I had yet to have observe in my career (which admittedly doesn’t include much board manufacturing). When applying voltage, I got a spark and puff of magic smoke from near one of the DC link capacitors on the left hand side. In the first batch of 40 I programmed, a full 20% failed in this way, some at 24V, and a few more at a 38V test. I initially thought the problem might have been an etching issue resulting in voltage breakdown between a via and an internal ground plane, but after examining the results under the microscope and conferring with MacroFab determined the most likely cause was cracking of the MLCCs during PCB depanelization.
Here’s a video describing the problem and potential solutions in way more detail than I’ll go into:
Needless to say, I hadn’t managed to see this failure in the 100 or so previous moteus controllers I’ve built, or I would have figured this out and resolved it!
For this first round of production controllers, I went and replaced every single capacitor near the edge of the board with a TDK variant that has internal soft termination, then tested them all at max voltage and a little beyond. Future revisions will use that variant of capacitor everywhere, as well as relocating the capacitors to reduce the mechanical stress they experience during manufacturing, handling, and installation.
Like with the fdcanusb, I built a programming and test fixture for the moteus controllers. The basic setup is similar to the fdcanusb. I have a raspberry pi with a touchscreen connected via USB to a number of peripherals. In this case, there is a STM32 programmer, a fdcanusb, and a label printer. Here though, unlike with the fdcanusb fixture, I wanted to be able to test the drive stage of the controllers and the encoders too.
My solution was to create a mechanical fixture that each board slots onto, with pogo pins that connect to test points for the phase outputs.
While it doesn’t make as good a connection as the solder through holes normally used to connect a motor, it is good enough to verify that the controller works. As a side-bonus, it also makes it trivial to test that the absolute magnetic encoder works properly.
This video shows how the programming and testing process works, and walks through testing a few boards.
Thus I spun a new revision r3, basically just to fix all the blue wires so that I could have some spares without having to worry about the robustness of my hot glue. While I was at it, I updated the logo:
As seems to be the way of things, a few days after I sent this board off to be manufactured, I realized that the CAN port needed to actually be isolated, since when the switches are off, the ground is disconnected from the rest of the system. Sigh. Guess that will wait for r4.
Finally returning back to other pieces of my quad roadmap, I finished getting an updated power distribution board ready for the quad A0. This board is one I had made many months ago and mostly brought up, but then didn’t quite finish. The r1 was when I first discovered my unfortunate stm32g4 pinout problems that doomed 3 of my in flight boards. The pictures here are of r2, which suffered from yet more pinout problems, resulting in more than my usual number of blue wires. Fortunately, identifying those problems here let me fix them ahead of time for the fdcanusb and moteus r4 boards.
This version has a probably overkill XT90 input connector, 6x XT30 output connectors, a connector for a lighted toggle switch and an FD-CAN port. The CAN port will eventually allow me to implement a soft power off, although I haven’t done that yet.
When hooked up to a moteus dev kit, it does do the proper pre-charging thing:
Update 2020-01-15: All the development kit slots are full. Thanks for your interest!
I’ve now received all the supplies I need to make up development kits for the moteus controller and to make a test quadruped!
I’m planning on making a few development kits from this production run so others can experiment with the moteus brushless controllers. Some people have already expressed interest in getting one — you have hopefully been contacted earlier. If you are interested in getting an opportunity to buy an early access kit and haven’t heard from me yet, fill out this form!
One of the necessary pieces for bringing up the moteus brushless controller and for ongoing development with it is being able to communicate with the device on the desk. There aren’t many options for desktop FDCAN communication currently, and certainly none that are in the affordable range occupied by the CANUSB family of devices which I’ve used before and was very happy with. Thus I created “fdcanusb”, a USB to FDCAN converter that allows one to communicate with FDCAN devices via a USB interface using a documented protocol, no drivers necessary.
The notable features:
USB 2.0 full speed host interface
ISO 11898-1: 2015 FDCAN interface on industry standard DB9 connector
Standards compliant bitrates up to 1/5Mbps supported
Software controllable termination
Frame sizes up to 64 bytes
Non-standards compliant bitrates allowed
Documented CDC ACM based host protocol (virtual COM port)
Apache 2.0 licensed firmware based on the STM32G474 controller
All for an expected sales prices of around $100.
This does come with some caveats: For one there is no galvanic or optoisolation, you get a common mode range of around +- 12V. Another is that using just a USB 2.0 full speed interface means it may not be able to keep a FDCAN bus fully saturated at high bitrates. Finally, the firmware will start out with just the bare bones capabilities, but can be extended to support features such as error injection, triggers, buffering, and more compact protocols in the future.
I’ve got the first functioning prototypes of these boards in hand now:
Another step in my plan for the next revision of the moteus servo mk2, is an updated controller board. As mentioned in my roadmap, I wanted to revise this board to make improvements in a number of domains:
Communications: Now instead of RS485, the primary communications interface is FD-CAN. This supports data rates of up to 8 Mbit and packet lengths up to 64 bytes. The header is nominally at the original CAN bit rate, but I have no need to be standards compliant and am running very short busses so I may run everything at the higher rate.
Connectors: Now there exist power connectors, in the form of XT30 right angle connectors and they are also daisy chainable like the data connectors. Additionally, all the connectors exit from the bottom of the board to make routing easier in configurations like the full rotation leg.
Controller: This uses the relatively new STM32G4 controller series. It is lower power than the STM32F4, supports FD-CAN, and also supports closely coupled memory, which may allow me to improve the speed of the primary control loop execution by 3 times.
Voltage range: This board now has 40V main FETS, with all other components at 50V rating or higher. Thus it should be safe with inputs up to 8S (34V or so).
It still maintains a number of the capabilities of the moteus r3.1 controller:
Integrated FOC encoder: An AS5048 encoder is mounted in the center of the back, which allows direct mounting above the rotor for FOC control.
Form factor: The entire board is 45x54mm, with M2.5 mounting holes. It is smaller than a 60mm BLDC motor and much smaller than an 80mm one.
Integrated current sensing: It uses a DRV8323 to drive the FETS, which includes current sensing for closed loop current control.
My first attempt at this, “r4”, came back from fabrication in an nonredeemable state. I used the digikey supplied footprint for the STM32G4 UQFN part, which looked mostly correct on the surface. However, while the footprint was good, the pinout was for the TQFP variant! This resulted in me shorting out several power pins to ground right next to the exposed pad in a way I couldn’t easily rework.
r4.1 seems to be in better shape so far. It powers up, and I now have blinking lights!
Next up is actually porting the control software to the new controller and communications interface.
Next up in Super Mega Microbot 2’s existence is being able to run untethered. Before that can happen, I need to be able to plug in a battery, and hopefully not have everything explode. As seen with the IMU junction board, even minor inductive links can result in chips getting toasted. I had thought that just adding sufficient capacitance to each of the point-of-load converters would resolve the issue, but in fact that almost made it worse.
Thus, I built a simple pre-charge board that I could put in line with the main power. It has two big FETs, one power resistor, an ATTiny44, and the random regulators and glue necessary to make it work. The microcontroller has one job. On power on, it waits a bit, energizes the “pre-charge” FET which has the power resistor in line. Then, a short while later, it energizes the main FET through which all power will flow.
I did some minimal qualification testing first with a single motor which went fine. Then I tested it against the whole quadruped, where I scoped the output ground line. Here, you can see that the output ground line initially rises linearly with the ramp up rate of the lab supply I was using to test. Then, about 80ms later after the ATTiny has powered on, it energizes the pre-charge FET and the output ground asymptotically approaches the resistance of the power resistor. Then again, at 100ms after that, the main FET is engaged and the output ground voltage drops all the way to 0 (or close enough modulo the FET on-resistance).
After that, all that was left was to try it with a real battery:
Higher input voltage: The old system ran at 2S, so 7.2V nominal. Now we’re running at 5S, so 18.5V nominal.
RS485 data: The HerkuleX based robot used TTL level data communications. moteus uses RS485.
Daisy chained power: With the new raspberry pi based computer in the turret, I now need to have an additional power and data port up on the mobile part of the turret.
No camera passthrough: Similarly, since the camera is directly attached to the raspberry pi 3, I don’t need to mess with having a connector to pass it through anymore.
As usual, I sent it off to MacroFab and waited. A seemingly very short time later and poof, here it was!
Bringing this up was more annoying than it could have been, mostly from a software perspective. The moteus and imu junction firmware were both based on the original gimbal software, but refactored to be usable across different projects. At the same time, that was where I had developed the RS485 based multiplex transport library. So, now was the time to bite the bullet and convert the gimbal software to use those common libraries.
Since the gimbal board has another unique processor compared to everything else, I broke it out into a separate git repository:
The old project was initially CubeMX based. When porting to rules_mbed and moteus/mjlib, I was in a hurry, so just copy and pasted much of the CubeMX initialization into the new tree and didn’t use any mbed APIs at all. It took me a while to remember how all the CubeMX initialization was glued together and which pieces of it were relevant before all the peripherals started working properly.
I then proceeded to mechanically integrate it together into the unused turret.
I once again had to remember how to calibrate and operate the thing. Doing this once every 9 months is kind of painful! However, I did manage to get it all working again, and ready to be integrated onto the mech.