Category Archives: robots

power_dist r4.5b

Here is yet another new product announcement! In the same line as the new pi3hat, here is a new minor revision of the power_dist, the r4.5b:

The changes are largely the same as for the new pi3hat:

  • The input voltage range is extended from 10-44V, to 10-54V.
  • The CAN-FD port has +-58V bus fault protection, up from +-12V.
  • Additionally, the measurement noise of the output current has been improved from 300mA to approximately 30mA.

Check it out at today!

moteus-n1 beta release

Implied by my previous writeup on pin selection for external connectors, we’ve got a new variant of the moteus controller to announce today in beta form, the moteus-n1!

This variant is intended to be more feature-full, higher performance (and higher cost). Here are some bullet points of the biggest differentiators with r4.11:

moteus n1moteus r4.11
Price$159 USD @ qty 1$104 USD @ qty 1
SizeAs small as 46mmx46mmx8mm with optional back connectors omitted.

58% of the volume, 87% of the top down footprint size of r4.11
External PeripheralsEach of the auxiliary connectors supports SPI, UART, ADC, SW & HW Quadrature, Hall sensors, and I2C.

5V and 3.3V is provided on each connector to power peripherals (100mA for each voltage available combined between both connectors).

I2C pullups are configurable on each connector.

All 4 pins on AUX2 are 5VT, the two non-SPI pins on AUX1 are 5VT.
ENC (AUX1) supports SPI, Hall and ADC.

ABS (AUX2) supports UART and I2C and are both 5VT.
RS422Built in RS422 transceiver for communicating with RS422 / BiSS-C encoders (BiSS-C / SSI not yet supported in software).None
Voltage8-54V, 48V nominal8-44V
Peak Output100A output phase current, 1200W100A output phase current, 500W
Continuous Output10A output phase current ambient, 20A w/ thermal management10A output phase current ambient, 20A w/ thermal management
CAN fault tolerance58V bus fault tolerance12V bus fault tolerance
Power ConnectorsSolder pads for DC bus input, one always present XT30, one optional XT302x XT30

The short form is that the n1 does not really expand the set of motors that can be usefully driven, but does enable operation in 48V systems, is more compact and electrically robust and provides significantly improved external peripheral support.

Don’t worry, the existing r4.11 controllers aren’t going anywhere and are still being produced. The moteus-n1 series just complements them by enabling new applications.


Both the moteus n1 and r4.11 share the same open source firmware, although the n1 requires a newer release to operate. Command, control, and monitoring work identically on both, including tview, the python library, and all CAN formats. The only exception is that the n1 has more external pin configurations possible as shown in the reference documentation’s pin capability table.


The mechanical form factor of the n1 is different from r4.11, and thus the existing devkit bracket and heat spreader can not be used. New variants exist for both of those that are compatible with the n1.

The XT30 and JST PH3 board connectors are now available for sale separately in order to convert n1’s to have daisy chain connectors.

Finally, all the external peripheral connectors use JST-GH connectors, so we now have headers and pre-crimped wires available for the JST GH6 (used for the RS422), JST GH7 (used for AUX2) and JST GH8 (used for AUX1).

Beta availability

Starting today we’re making some moteus-n1 units available in a paid beta program, with errata documented here. Up to 2 controllers in either devkit or bare board form (both bare board and devkits variants count towards the total) can be ordered per customer via the links above. There is no technological limit if you try to exceed that, you will just have your order cancelled with a note to try again with fewer.

Good luck!

moteus external connector pin selection

moteus r4.11 has two external connectors, the ABS connector (AUX2) and the ENC/AUX1 connector. The ABS connector was designed initially just to have 2 I2C pins. The ENC connector just has the random pins that were used for the onboard encoder SPI plus one more. Thus the range of external accessories that can be connected is somewhat haphazard and not necessarily all that useful.

When working on a more ground up revision of the controller, I wanted to improve that situation to expose more connectivity options on still a relatively limited connector set. The idea was to use 2 connectors, one which has 5 I/O pins and the other with 4 I/O pins. The onboard encoder SPI would still be accessible on the larger connector to use for at least one external SPI encoder, but how much other functionality could be crammed into the remaining pins? To start, lets see what possible options there are in the current firmware and supported by the STM32G4 microcontroller that moteus uses:

  • SPI: The larger connector by definition would have a set of SPI lines, MISO, MOSI, and SCK (now sometimes termed CIPO/POCI and COPI/PICO).
  • I2C: I2C requires two lines, one for data and one for clock.
  • ADC: Sine/cosine encoders and general purpose ADC inputs require analog inputs.
  • Quadrature: Quadrature encoders require two signal lines.
  • UART: Asynchronous serial lines can be used for a variety of purposes.
  • 5V Tolerant: While the STM32G4 used in moteus is 3.3V native, it can be convenient to support 5V inputs.

To be useful in the moteus firmware, most of these capabilities need to be accessed through STM32 specific hardware. The one exception is quadrature inputs, for which the firmware can manage slow to moderate rates using interrupts alone, but high rates requires hardware decoding. Complicating this, the STM32G4 only provides access to specific hardware peripherals on specific pins through the alternate function map:

My challenge was to figure out which microcontroller pins to assign to the 9 (5 on AUX1, 4 on AUX2) ports which maximized the number of hardware peripherals that could be used on each connector. There are a few additional twists that make this process more challenging than one would expect.

Multiple STM32 pins per connector pin

It is possible to connect multiple STM32 pins to the same external connector pin. With this, the software for any given user requested configuration can leave the unused pin in a high impedance mode where they will largely not effect the output. There are some constraints with this though, caused by the STM32 architecture.

If a pin without analog functionality is connected to an analog signal, then it has a permanently connected schmitt trigger attached. This will cause undesired behavior and power consumption at certain analog voltage levels. Pins with analog functionality have an additional switch to disconnect this. Thus if a user visible pin is intended to have analog inputs, then all the STM32 pins must have analog functionality.

Similarly, if a connector pin is intended to be 5V tolerant, then every STM32 pin connected to it must also be 5V tolerant.

The analog input pins are sprinkled across the 5 different ADC converters present on the STM32G4. Ideally, the pins would not all use the same ADC, so that the sampling window could fit into the existing ADC sampling time of the main interrupt service routine.

Doing the search

I first attempted to conduct this search by hand, but found that I had a hard time wrapping my head around the possibilities, kept getting lost back-tracking and ultimately could not keep all the constraints in mind at once. So… I wrote a tool! I ended up making a brute-force python script that consumes a simple one-pin-per-line encoding of the capabilities, takes some optional constraints like pins or peripherals to not use, and finds all possible configurations which optimize a metric.

Portion of pinout definition

I used this in two separate phases. First I ran it in a mode on the 4 user-pin connector to find a configuration where all user pins were 5V tolerant. Then for the 5 user-pin connector, I excluded the pins and peripherals used on the 4 pin connector, and added the constraint that the non-SPI pins had to be 5V tolerant. The onboard magnetic encoder also connected to these SPI pins is not 5V tolerant, so there was no reason to aim for that here. On this second phase, there were bonus points in the metric for how many other peripherals could be crammed into these 5V tolerant pins, since they could be used even while using the onboard magnetic encoder.

The tool has a few separate classes for each of the constraints. Each evaluates a pin configuration or subset of pins, and returns whether that constraint has been met, is inconclusive, or is impossible to meet. Enumerating the possible sets of pins was slightly complicated because of the optional “pin doubling” that can occur. I ended up using an encoding of the problem that made this not too troubling.


In the end, I met nearly all of my goals. The 4 pin connector looks like:

Connector PinSTM32G4 PinFunctions
1PF15V / SPI / ADC
2PA10 / PF05V / SPI / UART_RX / I2C_SDA / ADC
3PA11 / PC45V / SPI / UART_TX / I2C_SCL / ADC / QUAD_3A

The only real downsides here are that if hardware quadrature is used, then neither USART nor I2C can be used simultaneously.

For the 5 pin connector, the following assignment was chosen:

Connector PinSTM32G4 PinFunctions
1PA5 / PB14SPI / QUAD_1A / ADC
5PB3 / PB95V / QUAD_1B / I2C_SDA / UART_TX

Here, the only bonus metric which was not satisfied was having ADC capabilities on the non-SPI pins. Thus to use ADC functionality on the 5 pin port, the onboard magnetic encoder must be disabled.


It probably doesn’t make sense to spend this much time on pin configuration for a purpose built board. In this case, since the number of external peripherals connected to moteus can be relatively large and each end-user may have a different idea of what constitutes a useful configuration, I think it was worth the effort to maximize flexibility of the exposed pins.

UART tunneling with moteus

With the release of more flexible I/O support, the moteus controller auxiliary port can be used to monitor encoders using an onboard UART. Now, with firmware release 2023-02-01, those UART pins can be used as an arbitrary logic level serial port controlled by the application! Let’s see how to use it below.

First, you will need to look at the pin configuration table to find pins that support UART functionality, and configure them as UART in the “aux?.pins” configuration tree. Next, “aux?.uart.mode” should be set to “kTunnel”, along with the desired baud rate. That’s it on the configuration front.

To send and receive data from the serial port requires using the diagnostic mode CAN protocol. To date, diagnostic mode channel “1” has been used to send and receive diagnostic mode commands from moteus itself. Now two new channels are available:

  • 2: aux1
  • 3: aux2

Unfortunately, tview does not yet have support for these. However, the python library and moteus_tool do. If you are not running any moteus using application at the same time, you can start moteus_tool with “moteus_tool –console –diagnostic-channel 3” to send and receive data from the aux2 serial port rather than the normal moteus diagnostic protocol. Similarly, python applications can use the “moteus.Stream” class and specify an alternate channel, like:

c = moteus.Controller()
aux2_uart = moteus.Stream(c, channel=3)
await aux2_uart.write(b"Data to write on the aux2 UART pins")
print(await aux2_uart.readline())

That’s it for now, good luck!

moteus clock synchronization

The moteus controller, when it implements its control algorithms, uses the internal RC oscillator of the onboard STM32G4 microcontroller to calculate things like velocity and to advance position over time. Typically, this is accurate to within 0.5% which is more than sufficient for most applications. However, there are cases where it does matter.

One common case is when multiple moteus controllers are operated together, and either the relative velocities of the controllers must match closely, or the time required to complete long trajectories must match closely. For example, if a trajectory would take 100s to complete, then a 0.5% difference in clock rate between two controllers would result in one completing 0.5s before the other.

As of firmware release 2023-02-01, there is now a mechanism whereby an application can synchronize the clock of a moteus to the host time base. If this is done for multiple devices, then they will all share the same time base and, on average, produce identical velocities and trajectory time-to-completes.


To make this work, there are two pieces. First we need to be able to change the rate at which the microcontroller’s clock operates. The microcontroller does provide a trim mechanism for exactly this purpose. At the factory it is calibrated and then the firmware is able to further tweak the result in approximately 40kHz increments, as compared to the 16MHz base RC oscillator frequency. For moteus, that works out to about 0.25% increments of speed. This trim was already exposed as an undocumented configuration option clock.hsitrim, but not in a form suitable for modification online.

So, the first part of this work creates a new register, “0x071 Clock Trim” to which an integer can be written. This integer is an offset to apply to the factory programmed trim, so 0 results in running at the factory default rate, 1 results in running ~0.25% faster and -1 results in running ~0.25% slower.

The second part of this work provides a way for the host application to measure how fast the moteus time base is progressing relative to the host time. This operates through the “0x070 Millisecond counter” register, which merely reports the number of elapsed milliseconds as counted by moteus.

Using these two features, an application works as follows: At a regular rate, it polls the millisecond counter to see how many milliseconds moteus thinks has passed. If that is more than have passed on the host, the moteus clock can be slowed down. If it is less, then the moteus clock can be sped up. Additionally, a measure of the total time elapsed since application start can be used to zero out the overall drift. Doing this in a robust way requires a little bit of thought, but there is an example python implementation in the moteus repository showing how it can be done:

Caveats and Conclusion

One caveat is that since the resolution of time rate adjustment is relatively large, 0.25%, velocities on multiple controllers may instantaneously differ by approximately that much. Only averaged over time would they match exactly. Secondly, the synchronization procedure is more complex than base control and may introduce other failure modes.

Synchronizing clocks is not likely to be used all that often, but in some cases it can be simpler than other approaches. If it might be useful to you, take a look a the example script above, or drop into the mjbots Discord #moteus channel and ask!

moteus firmware 2023-02–01

Partly to celebrate moteus controllers being back in stock and partly because a lot of important work has backed up, we’ve just released a new firmware version for moteus (2023-02-01) that has a little bit of something for everyone. Future posts will examine some of these features in more detail, but for now you just get the bullet list:

  • Support sending and receiving arbitrary data from a UART configured on either of the auxiliary ports
  • Permit I2C encoders to operate at up to 1kHz
  • Report control position, velocity, and torque as well as the errors in tracking them over the register protocol as 0x038-0x03d
  • Provide support for synchronizing the clock of a moteus controller with a host application
  • Report hall effect errors
  • Expose the drv8353 error flags as register 0x140 and 0x141
  • Fix register 0x006 (ABS port position) to be reported in revolutions

Resistive heater dummy load

While testing moteus controllers, it is often necessary to experiment with high power conditions. For short durations, any decent sized brushless motor can work, as the windings have a non-zero thermal mass and take a little bit to warm up. However, when testing at high power for extended duration, it can be hard to find a way to get rid of all output energy. Even blowing a fan directly onto a motor only gets you so far when you are trying to get rid of 1kW.

Thus enter my resistive dummy load:

This is just a block of DC water heaters screwed into a plastic container. They are wired in series with some high current inductors to roughly approximate the inductance and resistance of a motor in the range of what is normally driven by moteus. When conducting a test, the container can be filled with water to greatly increase the available thermal mass (and if need be boil away the water).

Parts and Assembly

I have used this fixture with two different elements, a 24V 900W one, and a 12V 600W one depending upon what resistance I want to test with:

The container is just a basic polyproplene plastic one, so that it should be safe up to at least the boiling point of water:

The inductors are 33uH, 30A:

To assemble, I used a 1.25″ hole saw to cut each of the holes, then used a 1″ NPT nut to fasten each element in place. Each phase connected to 3 of the inductors in parallel in series with 4 of the heating elements. All three phases were tied together in the center to form a wye topology.

Debugging bare-metal STM32 from the seventh level of hell

Here’s a not-so-brief story about troubleshooting a problem that was at times vexing, impossible, incredibly challenging, frustrating, and all around just a terrible time with the bare-metal STM32G4 firmware for the moteus brushless motor controller.


First, some things for context:

moteus has a variety of testing done on every firmware release. There are unit tests that run with pieces of the firmware compiled to run in a host environment. There is a hardware-in-the-loop dynamometer test fixture that is used to run a separate battery of tests. There is also an end-of-line test fixture that is used to run tests on every board and some other firmware level performance tests.

Because of all that testing, we’re pretty confident to release new firmware images once all the tests have passed, and try to ship out boards with firmware that is within a week or two of the newest on all boards and devices that go out the door. That said, there is some effort made to ensure that large orders all have the same firmware on them. Thus, my saga started when I went to re-program a few dozen boards using the end-of-line test fixture so that they could all match the most recent version.

The first symptom

When I went to re-program them, a large portion of the boards failed tests surrounding the quality of the current sense measurements, indicating there was too much noise in the current sense measurements, specifically when driving 0 current. That could mean that there were soldering problems on the board, or that the test fixture had corroded contacts, or potentially firmware issues. In response, the test fixture got its contacts cleaned very thoroughly, I verified this was happening across many boards all of which had passed earlier, and there were only 3 changesets that affected the firmware in any way, all of which seemed pretty innocuous.

Once I had given up on the problem being a fluke, I opened up tview on the end-of-line fixture and sure enough, wow, there was a problem:

Note how the values of servo_stats.adc_cur3_raw seem to bounce between what looks like their true value and 2048. I have seen problems like this before, related to ADC configuration and clock rate (as – haveothers), but absolutely nothing about the ADC configuration has changed in more than a year, so surely that can’t be it, can it?

The first diagnostic step

So, first things first. Now that I can observe a problem, is it reproducible. I used git bisect across the relevant firmware versions, and sure enough, one of the changes was positively correlated with the problem: 64f2a82575795d782ff3806ea2036f4cd2f02ef0 However, that change does absolutely nothing with the ADCs or the current sense pipeline, or the STM32 register configuration at all. So, I tried to create a more minimal version of that change which would still trigger the problem. What I got was this:

diff --git a/fw/bldc_servo_structs.h b/fw/bldc_servo_structs.h
index abbe26e..f06c16c 100644
--- a/fw/bldc_servo_structs.h
+++ b/fw/bldc_servo_structs.h
@@ -509,7 +509,7 @@ struct BldcServoConfig {
   // debug UART at full control rate.
   uint32_t emit_debug = 0;
-  uint32_t field1;
+  uint32_t field1 = 0;
   BldcServoConfig() { = 0.005f;

So, adding the initialization of a member in a random structure (the one that holds PID gains among others), triggered the issue. If the initialization was only of a uint8_t or uint16_t, no problem, but a uint32_t, float, or uint64_t did it.

Well, “that’s odd”.

Clearly that change shouldn’t have any impact, so if the problem is at the C++ level, it must be undefined behavior somewhere else, and if it isn’t at the C++ level, it could be anywhere. So, my next step was to look at the difference in the disassembly to see what that code change wrought that the STM32 would see.

This is from “meld”, with a set of custom filters to remove most spurious changes related to addresses changing. But yikes, that one extra initialization results in a *lot* of churn in the assembly. If we look at the structure constructor, the change we expect is there in that we can see that the field is getting newly initialized.

However, with “-O3” optimizations on, gcc-11 makes all kinds of different decisions at various points. Instructions are re-ordered, different registers are used, entire blocks of code are re-ordered in their memory layout and execution, and extra padding is added or removed. There are many changes, any of which could be interacting with whatever undefined behavior is in the system.

Taking a step back

Since looking at the disassembly wasn’t going to be easy, I decided to take a step back and see if I could observe what was different in the system when it was running between the good and not-good states. Most likely some peripheral was configured incorrectly, with the ADCs being a prime candidate, but the clock tree could also be a culprit.

When debugging STM32s, I sometimes use the PyCortexMDebug project, which lets gdb use the vendor provided SVDs to interpret the contents of all registers. Here, I wanted to print out every register on every peripheral just to see what was different. PycortexMDebug doesn’t natively give you a way to do that. However, it can list all the peripherals it knows about, which I wrote to a file and pre-processed to remove the human level annotation. Then using gdb’s “python-interactive” mode, I could do a:

> regs = [x.strip() for x in open('/tmp/all_regs.txt').readlines()']
> for reg in regs:
>   gdb.execute('svd/x ' + reg)

Which did the trick — at least after copy and pasting the output from the terminal. I didn’t bother figuring out how to get it written to a file. So, now, I have two giant files with every peripheral register, one from a firmware that was working, and one from a firmware that was exhibiting the extra noise. I went through them line by line and found…. nothing.

Some registers were different of course, but the only ones were timer values, and data registers on the ADC and SPI peripherals, and the system control block depending upon if the code happened to be in an interrupt when I stopped to sample it. No configuration values or anything that would point to a problem. Sigh.

More backing up

OK. So maybe there is a peripheral register that isn’t in the SVD that would correlate with the problem? My next step was to use gdb to dump the entire peripheral address space to an srec file in both cases.

dump srec memory /tmp/out.srec 0x40000000 0x51000000

Note, this does take a *long* time, at least 15 minutes with the hardware I was using.

What did I earn for my hard earned wait? Bupkis, nothing, nada, squat. After looking through every single byte that was different, the only ones that had changed were the same ones that the svd method above turned up, plus a bit of random noise in the “reserved” section between peripherals that looked like genuine bus noise. Notably, not any configuration registers on any peripheral at all.

Even more backing up

OK. So if the problem isn’t in a peripheral register, maybe there is some difference in program state that is causing the problem? Maybe a stack overflow or something? So, I switched to SRAM dumps. First, I modified my startup assembly to start out with guard bytes across all of SRAM so that I could verify the stack hadn’t overflowed (not even close). I also used that to verify that the code which was copied into CCM SRAM on startup hadn’t overflowed or been stomped on (it hadn’t). Next I did a diff between the working and non-working states.

Here, there were a lot more differences as the firmware has a lot of state that varies from run to run. With the structure of the moteus firmware, most storage ends up being allocated on the C/C++ stack from a fixed size pool. This means that most of the variables don’t have a useful entry in the symbol table, even though their address is consistent from run to run. To identify what each change was, I started the firmware afresh with a breakpoint on _start, then added a hardware watchpoint on the address of interest.

b _start
watch *0x20004560 # (for example)
continue (as many times as necessary)

And then looked to see what modified that particular memory location to determine what it was doing. I methodically went through every difference, about 50 of them. I found things like the buffer used to hold CAN-FD frames, timers, nonce counters, the values read by the position sensor and current sensor, and many other things that all seemed perfectly reasonable.

Yet another approach doomed to give no useful information.

Back to an earlier approach

Whatever the problem was, it appeared to be in state on the STM32 that was not accessible to mere mortals. Probably a peripheral got into a bad state that wasn’t exposed via its registers or something. If I couldn’t find the state that was different, could I at least make a “minimal code difference” which was actually minimal?

My C++ minimal difference was pretty small, just the addition of an “=0” to a field initializer. However, that resulted in significant changes in the output program. To make things a little bit more controllable, I tried adding some __asm__("nop") entries to the constructor in question and sure enough, some counts of NOPs would trigger the problem and others wouldn’t. However, they still resulted in large differences in the output.

So then I undertook the painstaking step of gradually turning off optimizations in each function that I saw had changed. In some cases it was as easy as sticking a __attribute__((optimize("O1"))) on the definition. However, in many cases gcc/C++ requires the inline definitions be pulled out-of-line to make that annotation. Both because of that, and just because of bad luck, often these changes would result in my “nop” trick no longer triggering a failure. I worked methodically though, trying new functions until I was eventually able to make a minimal assembly diff that failed.

diff --git a/fw/bldc_servo_structs.h b/fw/bldc_servo_structs.h
index 95db9fe..8916d4e 100644
--- a/fw/bldc_servo_structs.h
+++ b/fw/bldc_servo_structs.h
@@ -533,6 +533,11 @@ struct BldcServoConfig {
     pid_position.ilimit = 0.0f;
     pid_position.kd = 0.05f;
     pid_position.sign = -1.0f;
+    asm volatile (
+        "nop;"
+        "nop;"
+    );
   template <typename Archive>

And the assembly diff is solely:

Solely the addition of the 2 nops!


As before, I’m using the same regexes with meld to exclude spurious changes related to addresses and literals. The exact set of expressions is below:

asm_address      ^.{20}
stm32_pc         08[0-9a-f]{6}
stm32_pc2        (80[012345][0-9a-f]{4})
stm32_addr       \+0x[0-9a-f]+>
stm32_literal    #[0-9]{2,5}

Trying to understand this a bit more

So far we have learned that simply adding two NOPs to one function that is totally unrelated to the problem in question causes the ADC to become noisy in an odd way. I tried some experimenting to learn more about the failure.

What does adding more NOPs do? The answer… 1 or 2 NOPs fails, 3 or 4 NOPs works, 5 or 6 fails, etc.

Hmmm…. my current top two theories are that either a) it is the instruction layout or b) the execution timing that results in the difference. To rule out one or the other, I made up a series of 8 NOPs, and then substituted a jump in for the first NOP that skipped to one of the later NOPs. That way I could adjust the execution cycle time of the relevant function one by one without changing any layout. That had no effect. Which meant it must have to be the physical layout of the code, not the timing.

The grind

At this point, I undertook what was perhaps the most arduous debugging task yet. To figure out which code was unhappy about having its instruction address changed, I bisected adding NOPs. This wasn’t super straightforward, because as mentioned, gcc’s optimizations generally mean that adding a NOP to a random function results in all kinds of changes all over the place. My procedure was roughly like this:

  1. Identify where in the address space I wanted to add a NOP.
  2. Find a nearby function that was written by me, and not a template expansion or library function.
  3. Switch it to be O1/O0
  4. See if I can still trigger the problem at any of my former test points by adding NOPs (turning off optimizations on the one function sometimes re-ordered everything)
  5. If I can’t, then pick a different function and go back to 1
  6. If I can, then bisect over all my current test points (which may be in a different order than the last bisection) to find the latest address space point where I can add a NOP to trigger the problem

While brutal, I figured this was sure to result in finding the culprit.

And sure enough, after about 15 steps, each taking around 5-10 minutes, it did. I thought the following two lines were the culprit:

        (map_adc_prescale(kAdcPrescale) << ADC_CCR_PRESC_Pos) |
        (1 << ADC_CCR_DUAL_Pos); // dual mode, regular + injected
    ADC345_COMMON->CCR =
        (map_adc_prescale(kAdcPrescale) << ADC_CCR_PRESC_Pos) |
        (1 << ADC_CCR_DUAL_Pos); // dual mode, regular + injected

The two lines that configure the ADC prescaler! But, wait, didn’t we verify that the ADC prescaler as read from the peripheral registers was the same in both instances? Why yes, we certainly did.


(gdb) svd/x ADC12_COMMON
Registers in ADC12_Common:
	CSR:  0x000A000A  ADC Common status register
	CCR:  0x000C0001  ADC common control register
	CDR:  0x00000000  ADC common regular data register for dual and triple modes
(gdb) svd/x ADC345_COMMON
Registers in ADC345_Common:
	CSR:  0x000A000A  ADC Common status register
	CCR:  0x000C0001  ADC common control register
	CDR:  0x05250000  ADC common regular data register for dual and triple modes

Not working:

(gdb) svd/x ADC12_COMMON
Registers in ADC12_Common:
	CSR:  0x000A000A  ADC Common status register
	CCR:  0x000C0001  ADC common control register
	CDR:  0x00000000  ADC common regular data register for dual and triple modes
(gdb) svd/x ADC345_COMMON
Registers in ADC345_Common:
	CSR:  0x000A000A  ADC Common status register
	CCR:  0x000C0001  ADC common control register
	CDR:  0x05270002  ADC common regular data register for dual and triple modes

For good measure, I tested using stepi to walk through the initialization in the bad state to see if it was somehow related to wall clock timing, but that didn’t make a difference.

Narrowing things down

To avoid the “flavor-of-the-day” the gcc optimizer gives you and make my life easier for experimenting, I rewrote those two lines in inline assembler, just hard-coding the required CCR value:

    asm volatile(
        "str %2, [%0];"
        "str %2, [%1];"
        : "r" (&ADC12_COMMON->CCR),
          "r" (&ADC345_COMMON->CCR),
          "r" (0x000C0001)

I added in NOPs before, in between, and after the two stores. To my surprise, in all 3 places failures could be induced, but only on every 4th NOP. Which meant my identification of these two lines was incorrect.

Thus, false alarm. I kept moving down the function, replacing sections with inline assembler and then bisecting with NOPs until I reached the following section:

    ADC1->CR |= ADC_CR_ADEN;
    ADC2->CR |= ADC_CR_ADEN;
    ADC3->CR |= ADC_CR_ADEN;
    ADC4->CR |= ADC_CR_ADEN;
    ADC5->CR |= ADC_CR_ADEN;

Here, all 5 ADCs are turned on in rapid succession after previously having all their pre-requisite startup operations and delays performed. NOPs placed before this could cause the ADCs to get into the bad state, but NOPs immediately after did not. Placing NOPs between them always seemed to make the following sections work without problem. Once I had at least 3 NOPs between each, then no amount of change above could cause a failure.

Finally, a decent hypothesis and solution

It seems that the ADCs on the STM32G4 do not like to be turned on in rapid succession, and if they do, bad things can happen like having the prescaler flipped to a different value without it showing in the corresponding register. In this case, the flash accelerator was probably delaying the initialization when the ADEN sets happened such that they crossed a fetch boundary. Then when two of them ended up in the same pre-fetch block, they would get turned on too quickly together. Maybe it causes a local brownout or something? Somewhat recently I upgraded to gcc-11, which probably did a better job of packing these enables into a smaller amount of code space.

I guess that’s an errata for you.

With that understanding, a solution is trivial. Just initialize the ADCs one by one instead of all at once. The initialization sequence for the ADC is documented as requiring a wait until the ADRDY flag is set, so the fix is just to wait for that for each ADC in turn before enabling the next one. For good measure, since initialization isn’t time critical, I switched the whole process to be serial for each ADC, as I expect that is the more tested path with the hardware.

What is the lesson? Hardware is hard? Persistence pays off? I guess you can decide!

As a bonus, now that I know one of the prime symptoms to look for to troubleshoot bad prescalers (unusual bit flips around 2048), I discovered that I could get a bit more performance around the 0 current point by increasing the moteus prescalers a bit (75df013).

moteus firmware release 2022-07-11

It has been on github for a few days now, but I’m excited to announce the newest moteus firmware release, 2022-07-11. This release includes some big features, and some quality of life improvements all around.

  • Flexible I/O subsystem: This release includes the new flexible I/O subsystem. This adds support for many new encoder types and lets you connect them up in a wide variety of ways.
  • Cogging torque compensation: Preliminary support for cogging torque compensation is present in this release. It works pretty well on a number of motor types already, future articles will describe it in more detail.
  • Encoder eccentricity compensation: This feature lets you linearize the output position and velocity in the face of non-linear encoder readings. A write-up for it is also forthcoming in the not-too-distant-future.
  • Transparent no-BRS CAN-FD communication: If your CAN-FD network is only capable of operating at 1Mbps, and you send queries frames with BRS turned off, moteus will now respond in kind. This eliminates most needs to change the CAN bus frequency due to marginal electrical properties.

This is an exciting time for moteus, and the new features will keep coming!