Category Archives: robots

Debugging bare-metal STM32 from the seventh level of hell

Here’s a not-so-brief story about troubleshooting a problem that was at times vexing, impossible, incredibly challenging, frustrating, and all around just a terrible time with the bare-metal STM32G4 firmware for the moteus brushless motor controller.


First, some things for context:

moteus has a variety of testing done on every firmware release. There are unit tests that run with pieces of the firmware compiled to run in a host environment. There is a hardware-in-the-loop dynamometer test fixture that is used to run a separate battery of tests. There is also an end-of-line test fixture that is used to run tests on every board and some other firmware level performance tests.

Because of all that testing, we’re pretty confident to release new firmware images once all the tests have passed, and try to ship out boards with firmware that is within a week or two of the newest on all boards and devices that go out the door. That said, there is some effort made to ensure that large orders all have the same firmware on them. Thus, my saga started when I went to re-program a few dozen boards using the end-of-line test fixture so that they could all match the most recent version.

The first symptom

When I went to re-program them, a large portion of the boards failed tests surrounding the quality of the current sense measurements, indicating there was too much noise in the current sense measurements, specifically when driving 0 current. That could mean that there were soldering problems on the board, or that the test fixture had corroded contacts, or potentially firmware issues. In response, the test fixture got its contacts cleaned very thoroughly, I verified this was happening across many boards all of which had passed earlier, and there were only 3 changesets that affected the firmware in any way, all of which seemed pretty innocuous.

Once I had given up on the problem being a fluke, I opened up tview on the end-of-line fixture and sure enough, wow, there was a problem:

Note how the values of servo_stats.adc_cur3_raw seem to bounce between what looks like their true value and 2048. I have seen problems like this before, related to ADC configuration and clock rate (as – haveothers), but absolutely nothing about the ADC configuration has changed in more than a year, so surely that can’t be it, can it?

The first diagnostic step

So, first things first. Now that I can observe a problem, is it reproducible. I used git bisect across the relevant firmware versions, and sure enough, one of the changes was positively correlated with the problem: 64f2a82575795d782ff3806ea2036f4cd2f02ef0 However, that change does absolutely nothing with the ADCs or the current sense pipeline, or the STM32 register configuration at all. So, I tried to create a more minimal version of that change which would still trigger the problem. What I got was this:

diff --git a/fw/bldc_servo_structs.h b/fw/bldc_servo_structs.h
index abbe26e..f06c16c 100644
--- a/fw/bldc_servo_structs.h
+++ b/fw/bldc_servo_structs.h
@@ -509,7 +509,7 @@ struct BldcServoConfig {
   // debug UART at full control rate.
   uint32_t emit_debug = 0;
-  uint32_t field1;
+  uint32_t field1 = 0;
   BldcServoConfig() { = 0.005f;

So, adding the initialization of a member in a random structure (the one that holds PID gains among others), triggered the issue. If the initialization was only of a uint8_t or uint16_t, no problem, but a uint32_t, float, or uint64_t did it.

Well, “that’s odd”.

Clearly that change shouldn’t have any impact, so if the problem is at the C++ level, it must be undefined behavior somewhere else, and if it isn’t at the C++ level, it could be anywhere. So, my next step was to look at the difference in the disassembly to see what that code change wrought that the STM32 would see.

This is from “meld”, with a set of custom filters to remove most spurious changes related to addresses changing. But yikes, that one extra initialization results in a *lot* of churn in the assembly. If we look at the structure constructor, the change we expect is there in that we can see that the field is getting newly initialized.

However, with “-O3” optimizations on, gcc-11 makes all kinds of different decisions at various points. Instructions are re-ordered, different registers are used, entire blocks of code are re-ordered in their memory layout and execution, and extra padding is added or removed. There are many changes, any of which could be interacting with whatever undefined behavior is in the system.

Taking a step back

Since looking at the disassembly wasn’t going to be easy, I decided to take a step back and see if I could observe what was different in the system when it was running between the good and not-good states. Most likely some peripheral was configured incorrectly, with the ADCs being a prime candidate, but the clock tree could also be a culprit.

When debugging STM32s, I sometimes use the PyCortexMDebug project, which lets gdb use the vendor provided SVDs to interpret the contents of all registers. Here, I wanted to print out every register on every peripheral just to see what was different. PycortexMDebug doesn’t natively give you a way to do that. However, it can list all the peripherals it knows about, which I wrote to a file and pre-processed to remove the human level annotation. Then using gdb’s “python-interactive” mode, I could do a:

> regs = [x.strip() for x in open('/tmp/all_regs.txt').readlines()']
> for reg in regs:
>   gdb.execute('svd/x ' + reg)

Which did the trick — at least after copy and pasting the output from the terminal. I didn’t bother figuring out how to get it written to a file. So, now, I have two giant files with every peripheral register, one from a firmware that was working, and one from a firmware that was exhibiting the extra noise. I went through them line by line and found…. nothing.

Some registers were different of course, but the only ones were timer values, and data registers on the ADC and SPI peripherals, and the system control block depending upon if the code happened to be in an interrupt when I stopped to sample it. No configuration values or anything that would point to a problem. Sigh.

More backing up

OK. So maybe there is a peripheral register that isn’t in the SVD that would correlate with the problem? My next step was to use gdb to dump the entire peripheral address space to an srec file in both cases.

dump srec memory /tmp/out.srec 0x40000000 0x51000000

Note, this does take a *long* time, at least 15 minutes with the hardware I was using.

What did I earn for my hard earned wait? Bupkis, nothing, nada, squat. After looking through every single byte that was different, the only ones that had changed were the same ones that the svd method above turned up, plus a bit of random noise in the “reserved” section between peripherals that looked like genuine bus noise. Notably, not any configuration registers on any peripheral at all.

Even more backing up

OK. So if the problem isn’t in a peripheral register, maybe there is some difference in program state that is causing the problem? Maybe a stack overflow or something? So, I switched to SRAM dumps. First, I modified my startup assembly to start out with guard bytes across all of SRAM so that I could verify the stack hadn’t overflowed (not even close). I also used that to verify that the code which was copied into CCM SRAM on startup hadn’t overflowed or been stomped on (it hadn’t). Next I did a diff between the working and non-working states.

Here, there were a lot more differences as the firmware has a lot of state that varies from run to run. With the structure of the moteus firmware, most storage ends up being allocated on the C/C++ stack from a fixed size pool. This means that most of the variables don’t have a useful entry in the symbol table, even though their address is consistent from run to run. To identify what each change was, I started the firmware afresh with a breakpoint on _start, then added a hardware watchpoint on the address of interest.

b _start
watch *0x20004560 # (for example)
continue (as many times as necessary)

And then looked to see what modified that particular memory location to determine what it was doing. I methodically went through every difference, about 50 of them. I found things like the buffer used to hold CAN-FD frames, timers, nonce counters, the values read by the position sensor and current sensor, and many other things that all seemed perfectly reasonable.

Yet another approach doomed to give no useful information.

Back to an earlier approach

Whatever the problem was, it appeared to be in state on the STM32 that was not accessible to mere mortals. Probably a peripheral got into a bad state that wasn’t exposed via its registers or something. If I couldn’t find the state that was different, could I at least make a “minimal code difference” which was actually minimal?

My C++ minimal difference was pretty small, just the addition of an “=0” to a field initializer. However, that resulted in significant changes in the output program. To make things a little bit more controllable, I tried adding some __asm__("nop") entries to the constructor in question and sure enough, some counts of NOPs would trigger the problem and others wouldn’t. However, they still resulted in large differences in the output.

So then I undertook the painstaking step of gradually turning off optimizations in each function that I saw had changed. In some cases it was as easy as sticking a __attribute__((optimize("O1"))) on the definition. However, in many cases gcc/C++ requires the inline definitions be pulled out-of-line to make that annotation. Both because of that, and just because of bad luck, often these changes would result in my “nop” trick no longer triggering a failure. I worked methodically though, trying new functions until I was eventually able to make a minimal assembly diff that failed.

diff --git a/fw/bldc_servo_structs.h b/fw/bldc_servo_structs.h
index 95db9fe..8916d4e 100644
--- a/fw/bldc_servo_structs.h
+++ b/fw/bldc_servo_structs.h
@@ -533,6 +533,11 @@ struct BldcServoConfig {
     pid_position.ilimit = 0.0f;
     pid_position.kd = 0.05f;
     pid_position.sign = -1.0f;
+    asm volatile (
+        "nop;"
+        "nop;"
+    );
   template <typename Archive>

And the assembly diff is solely:

Solely the addition of the 2 nops!


As before, I’m using the same regexes with meld to exclude spurious changes related to addresses and literals. The exact set of expressions is below:

asm_address      ^.{20}
stm32_pc         08[0-9a-f]{6}
stm32_pc2        (80[012345][0-9a-f]{4})
stm32_addr       \+0x[0-9a-f]+>
stm32_literal    #[0-9]{2,5}

Trying to understand this a bit more

So far we have learned that simply adding two NOPs to one function that is totally unrelated to the problem in question causes the ADC to become noisy in an odd way. I tried some experimenting to learn more about the failure.

What does adding more NOPs do? The answer… 1 or 2 NOPs fails, 3 or 4 NOPs works, 5 or 6 fails, etc.

Hmmm…. my current top two theories are that either a) it is the instruction layout or b) the execution timing that results in the difference. To rule out one or the other, I made up a series of 8 NOPs, and then substituted a jump in for the first NOP that skipped to one of the later NOPs. That way I could adjust the execution cycle time of the relevant function one by one without changing any layout. That had no effect. Which meant it must have to be the physical layout of the code, not the timing.

The grind

At this point, I undertook what was perhaps the most arduous debugging task yet. To figure out which code was unhappy about having its instruction address changed, I bisected adding NOPs. This wasn’t super straightforward, because as mentioned, gcc’s optimizations generally mean that adding a NOP to a random function results in all kinds of changes all over the place. My procedure was roughly like this:

  1. Identify where in the address space I wanted to add a NOP.
  2. Find a nearby function that was written by me, and not a template expansion or library function.
  3. Switch it to be O1/O0
  4. See if I can still trigger the problem at any of my former test points by adding NOPs (turning off optimizations on the one function sometimes re-ordered everything)
  5. If I can’t, then pick a different function and go back to 1
  6. If I can, then bisect over all my current test points (which may be in a different order than the last bisection) to find the latest address space point where I can add a NOP to trigger the problem

While brutal, I figured this was sure to result in finding the culprit.

And sure enough, after about 15 steps, each taking around 5-10 minutes, it did. I thought the following two lines were the culprit:

        (map_adc_prescale(kAdcPrescale) << ADC_CCR_PRESC_Pos) |
        (1 << ADC_CCR_DUAL_Pos); // dual mode, regular + injected
    ADC345_COMMON->CCR =
        (map_adc_prescale(kAdcPrescale) << ADC_CCR_PRESC_Pos) |
        (1 << ADC_CCR_DUAL_Pos); // dual mode, regular + injected

The two lines that configure the ADC prescaler! But, wait, didn’t we verify that the ADC prescaler as read from the peripheral registers was the same in both instances? Why yes, we certainly did.


(gdb) svd/x ADC12_COMMON
Registers in ADC12_Common:
	CSR:  0x000A000A  ADC Common status register
	CCR:  0x000C0001  ADC common control register
	CDR:  0x00000000  ADC common regular data register for dual and triple modes
(gdb) svd/x ADC345_COMMON
Registers in ADC345_Common:
	CSR:  0x000A000A  ADC Common status register
	CCR:  0x000C0001  ADC common control register
	CDR:  0x05250000  ADC common regular data register for dual and triple modes

Not working:

(gdb) svd/x ADC12_COMMON
Registers in ADC12_Common:
	CSR:  0x000A000A  ADC Common status register
	CCR:  0x000C0001  ADC common control register
	CDR:  0x00000000  ADC common regular data register for dual and triple modes
(gdb) svd/x ADC345_COMMON
Registers in ADC345_Common:
	CSR:  0x000A000A  ADC Common status register
	CCR:  0x000C0001  ADC common control register
	CDR:  0x05270002  ADC common regular data register for dual and triple modes

For good measure, I tested using stepi to walk through the initialization in the bad state to see if it was somehow related to wall clock timing, but that didn’t make a difference.

Narrowing things down

To avoid the “flavor-of-the-day” the gcc optimizer gives you and make my life easier for experimenting, I rewrote those two lines in inline assembler, just hard-coding the required CCR value:

    asm volatile(
        "str %2, [%0];"
        "str %2, [%1];"
        : "r" (&ADC12_COMMON->CCR),
          "r" (&ADC345_COMMON->CCR),
          "r" (0x000C0001)

I added in NOPs before, in between, and after the two stores. To my surprise, in all 3 places failures could be induced, but only on every 4th NOP. Which meant my identification of these two lines was incorrect.

Thus, false alarm. I kept moving down the function, replacing sections with inline assembler and then bisecting with NOPs until I reached the following section:

    ADC1->CR |= ADC_CR_ADEN;
    ADC2->CR |= ADC_CR_ADEN;
    ADC3->CR |= ADC_CR_ADEN;
    ADC4->CR |= ADC_CR_ADEN;
    ADC5->CR |= ADC_CR_ADEN;

Here, all 5 ADCs are turned on in rapid succession after previously having all their pre-requisite startup operations and delays performed. NOPs placed before this could cause the ADCs to get into the bad state, but NOPs immediately after did not. Placing NOPs between them always seemed to make the following sections work without problem. Once I had at least 3 NOPs between each, then no amount of change above could cause a failure.

Finally, a decent hypothesis and solution

It seems that the ADCs on the STM32G4 do not like to be turned on in rapid succession, and if they do, bad things can happen like having the prescaler flipped to a different value without it showing in the corresponding register. In this case, the flash accelerator was probably delaying the initialization when the ADEN sets happened such that they crossed a fetch boundary. Then when two of them ended up in the same pre-fetch block, they would get turned on too quickly together. Maybe it causes a local brownout or something? Somewhat recently I upgraded to gcc-11, which probably did a better job of packing these enables into a smaller amount of code space.

I guess that’s an errata for you.

With that understanding, a solution is trivial. Just initialize the ADCs one by one instead of all at once. The initialization sequence for the ADC is documented as requiring a wait until the ADRDY flag is set, so the fix is just to wait for that for each ADC in turn before enabling the next one. For good measure, since initialization isn’t time critical, I switched the whole process to be serial for each ADC, as I expect that is the more tested path with the hardware.

What is the lesson? Hardware is hard? Persistence pays off? I guess you can decide!

As a bonus, now that I know one of the prime symptoms to look for to troubleshoot bad prescalers (unusual bit flips around 2048), I discovered that I could get a bit more performance around the 0 current point by increasing the moteus prescalers a bit (75df013).

moteus firmware release 2022-07-11

It has been on github for a few days now, but I’m excited to announce the newest moteus firmware release, 2022-07-11. This release includes some big features, and some quality of life improvements all around.

  • Flexible I/O subsystem: This release includes the new flexible I/O subsystem. This adds support for many new encoder types and lets you connect them up in a wide variety of ways.
  • Cogging torque compensation: Preliminary support for cogging torque compensation is present in this release. It works pretty well on a number of motor types already, future articles will describe it in more detail.
  • Encoder eccentricity compensation: This feature lets you linearize the output position and velocity in the face of non-linear encoder readings. A write-up for it is also forthcoming in the not-too-distant-future.
  • Transparent no-BRS CAN-FD communication: If your CAN-FD network is only capable of operating at 1Mbps, and you send queries frames with BRS turned off, moteus will now respond in kind. This eliminates most needs to change the CAN bus frequency due to marginal electrical properties.

This is an exciting time for moteus, and the new features will keep coming!

Flexible I/O: Worked examples

First, if you haven’t already, check out these introductory posts:

Second, this article is available in video form:

Third, the official reference documentation can be read here. This describes in detail all of the possible configuration values.

With those niceties out of the way, lets get into the examples!

1: Onboard AS5047P

This configuration is the default for moteus, and uses the onboard absolute AS5047P magnetic encoder for all positioning.

  • Torque: Yes
  • Velocity: Good
  • Position: Good

Configuration: This is the default for moteus, so no configuration is required.

Discussion: This configuration provides a decent all-around compromise between complexity and performance. Torque control is available and velocity control is good outside of ultra-slow regimes. The position is absolutely known to within one rotation of the rotor, across power cycles.

2: Hall effects / hoverboard

Here, the built-in hall effect sensors from a brushless motor are connected to three pins on the ENC port (aux1). Hall effect sensors are sometimes the only sensored commutation method practical to integrate into a motor system, and are the most common means of providing commutation signals for things like hoverboard motors.

  • Torque: Yes
  • Velocity: OK
  • Position: Incremental Only

Hardware: First, for moteus r4.8/11, either a Pico-SPOX 6 connector (TE 5-1775444-6) must be soldered onto the blank ENC pads of a moteus board or a cable must be soldered directly on to the pads. Then a matching cable harness must be constructed to mate the motor’s hall effect sensors to the Pico-SPOX connector (Housing: Molex 0874390601, Terminal: Molex 0874210100). If the sensors require 5V, then a boost regulator and level translators will be required. Otherwise, the 3.3V output of the ENC port can provide up to 100mA of power to the sensors, and their outputs should be connected to the C, K, and O pins (4, 5, and 6 on the connector).

Configuration: This example requires both configuration and alternate options specified duration calibration.

conf set aux1.spi.mode 1      # disabled
conf set aux1.pins.1.mode 6   # hall
conf set aux1.pins.1.pull 1   # pull-up
conf set aux1.pins.2.mode 6   # hall
conf set aux1.pins.2.pull 1   # pull-up
conf set aux1.pins.3.mode 6   # hall
conf set aux1.pins.3.pull 1   # pull-up

conf set aux1.hall.enabled 1  # enabled

conf set motor_position.sources.0.type 4   # hall

To calibrate, two additional options must be passed:

--cal-hall              # Instruct moteus_tool to use hall effects
--call-motor-poles 30   # How many motor pole pairs are present

Discussion: Performance-wise, this is a relatively poor option. Torque control is available, which is mostly what is used in hoverboard applications. Since the effective encoder resolution is very coarse, velocity control only works well at moderate to high speeds (say 0.3 Hz or higher). Position control is *not* absolutely referenced (i.e. it starts from 0 every time you power on) and it is only good to around 4 degrees for 30 pole pair hoverboard motors. With the current moteus configuration options, there is often jitter when stopped when using velocity or position control.

3: External AS5047P

This configuration can be used when the performance of the onboard AS5047P would be adequate, but for whatever reason, the moteus controller cannot be positioned immediately behind the sense magnet.

  • Torque: Yes
  • Velocity: Good
  • Position: Good

Hardware: Like with the hall effect example, for moteus r4.8/11, this requires that a Pico-SPOX connector be populated on the ENC blank pads (TE 5-1775444-6) or that a cable be directly soldered to the pads. A cable must then be constructed to attach the external AS5047P to the mating Pico-SPOX connector (Housing: Molex 0874390601, Terminal: Molex 0874210100). All of the pins are required:

  • G: Ground
  • 3: 3.3V
  • C: SPI chip select
  • K: SPI clock

As the SPI lines are high frequency, they must be relatively short (<20cm), and mounted in such a way to avoid electrical interference. That may require shielding around the cable, or around portions of the mechanical assembly.

Configuration: The following configuration changes are necessary from a default setup:

conf set aux1.pins.0.mode 2   # spi_cs
conf set aux1.pins.1.mode 1   # spi
conf set aux1.pins.2.mode 1   # spi
conf set aux1.pins.3.mode 1   # spi
conf set aux1.spi.mode 2      # ext_as5047

Discussion: After the hardware has been constructed and the above configuration changes made, use and performance is identical to the onboard AS5047P configuration.

4: Sine / Cosine

Some absolute encoders emit their output in the form of two analog signals, the “sine” and “cosine”. For a given angle, each signal emits an analog voltage biased about a non-zero operating point based on its namespace. If the sine and cosine are 3.3V level, or can be downshifted using a resistive divider to fit within that range, then they can be used with moteus.

Hardware: Like with the hall effect example, for moteus r4.8/11, this requires that a Pico-SPOX connector be populated on the ENC blank pads (TE 5-1775444-6) or that a cable be directly soldered to the pads. A cable must then be constructed to attach the encoder to the mating Pico-SPOX connector (Housing: Molex 0874390601, Terminal: Molex 0874210100). The sine and cosine channels must be connected to one of:

  • ENC pin 4 / K / aux1 pin 1
  • ENC pin 5 / I / aux1 pin 2
  • ENC pin 6 / O / aux1 pin 3

Any two of those may be used. The ground signal should be connected, and the 3.3V signal can be used to provide up to 100mA of power to the target encoder.

Configuration: The following configuration changes are required from a default setup:

conf set aux1.pins.1.mode 8           # sine
conf set aux1.pins.2.mode 9           # cosine
conf set aux1.spi.mode 1              # disabled
conf set aux1.sine_cosine.enabled 1   # enabled

The next value must be calibrated. The raw values of the sine and cosine channel can be observed in telemetry as the encoder rotates through several revolutions. Take the midpoint, and use it below:

conf set aux1.sine_cosine.common 1692  # from hand-calibration

Calibration requires no special options.

Discussion: As a sine/cosine encoder is just an alternate way of connecting a rotor absolute position sensor, the basic properties are the same as for the onboard AS5047P encoder. However, the analog signal has less effective resolution, and more noise than a digital encoder. Thus, the audible noise may be greater for a given selected encoder bandwidth, and the control performance may be worse.

5: Quadrature / Index (ABI)

A lowest common denominator of absolute encoders is the quadrature and index interface. Here, two digital signal lines step through a quadrature relationship as ticks on the encoder pass, and a separate “index” line becomes positive at the 0 point of the encoder. This does provide absolute positioning, but requires that the motor possibly spin through an entire rotation at power on in order to discover what that absolute position is.

  • Torque: Yes
  • Velocity: Good
  • Position: OK

Hardware: Like with the hall effect example, for moteus r4.8/11, this requires that a Pico-SPOX connector be populated on the ENC blank pads (TE 5-1775444-6) or that a cable be directly soldered to the pads. A cable must then be constructed to attach the encoder to the mating Pico-SPOX connector (Housing: Molex 0874390601, Terminal: Molex 0874210100). The three signals can be connected to any of the following pins:

  • ENC pin 2 / C / aux1 pin 0
  • ENC pin 4 / K / aux1 pin 1
  • ENC pin 5 / I / aux1 pin 2
  • ENC pin 6 / O / aux1 pin 3

The ground signal should be connected, and the 3.3V pin may be used to provide up to 100mA of power to the target encoder.

Configuration: The following must be changed from default (assuming that the I pin is connected to K/ aux1 pin 1, and the two quadrature lines are connected to I / aux1 pin 2, and O / aux1 pin 3.

conf set aux1.pins.1.mode 7         # index
conf set aux1.pins.2.mode 4         # quad_sw
conf set aux1.pins.3.mode 4         # quad_sw

conf set aux1.spi.mode 1            # disabled
conf set aux1.quadrature.enabled 1  # enabled

Quadrature encoders have a rated cycles per revolution or counts per revolution. The counts per revolution is 4x the cycles per revolution, and is what moteus requires. The following configuration is suitable for a 1000 cycle per revolution or 4000 counts per revolution encoder.

conf set aux1.quadrature.cpr 20000
conf set motor_position.sources.0.type 3    # quadrature
conf set motor_position.sources.0.cpr 4000

Calibration requires no special options.

Discussion: This option can provide nominally the same performance as the onboard encoder with two caveats.

First, a “homing” step must be performed before control can begin. This means that actuating the motor must be possible without damage from the turn-on state, and not incompatible with the end application.

Second, it is possible for counts to be “missed” because of electrical interference. This can result in decreased performance or loss of control depending upon the severity. Proper routing and shielding of quadrature lines are essential to robust performance. The aux1.quadrature.error diagnostic field can be used to monitor for events that are consistent with electrical interference (it is normal to start at 1 at turn-on).

6: I2C Disambiguator

Often a motor controlled by moteus drives a gear reducer before eventually driving the output. In such cases, the onboard moteus encoder is not able to uniquely determine which sector the reducer output is located. One means of resolving that ambiguity is to place a low-rate absolute I2C based encoder on the output shaft.

Hardware: The only pins that support I2C on moteus r4.3/5/8/11 are the two data pins on the ABS port (a JST ZH-4 connector). A harness must be constructed to connect those two signal lines. On an unmodified board, 3.3V resistive pullups are hard-wired onto the board. If a 5V sensor is desired, either a boost regulator will be required to operate it from the ABS port power, or a separate 5V regulator will be necessary to power it.

  • ABS pin 1 – 3.3V
  • ABS pin 2 / aux2 pin 0 – SCL
  • ABS pin 3 / aux2 pin 1 – SDA
  • ABS pin 4 – GND

Configuration: Assuming an AS5048B encoder is attached, the following configuration changes are required:

conf set aux2.pins.0.mode 13        # i2c
conf set aux2.pins.1.mode 13        # i2c
conf set aux2.spi.mode 1            # disabled
conf set aux2.i2c.devices.0.type 1  # as5048
conf set motor_position.sources.1.aux_number 2
conf set motor_position.sources.1.type.i2c 7     # i2c
conf set motor_position.sources.1.i2c_device 0
conf set motor_position.sources.1.cpr 65536      # for the as5048
conf set motor_position.sources.1.reference 1    # output

Since two encoders must be correlated, the sign and offsets should be manually calibrated. Set them such that motor_position.sources.0.compensated_value in the diagnostics pane increases in the same direction for both and that it is 0 for both at the same position. Additionally, the gear box reduction ration needs to be set.

conf set motor_position.sources.1.sign 1           # hand-calibrated
conf set motor_position.sources.1.offset 0         # hand-calibrated
conf set motor_position.sources.0.offset 0         # hand-calibrated
conf set motor_position.rotor_to_output_ratio 0.5  # reduction ratio

Discussion: The control performance of this example is identical to that of the primary encoder in use, which here would be the onboard AS5047P. The advantage is that the absolute position of the output shaft is known exactly at power on, even with the reducing stage.

7: Onboard + Quadrature Output

For this example, a high resolution quadrature encoder is used in combination with the onboard encoder to improve the velocity and position tracking performance of the controller.

  • Torque: Yes
  • Velocity: Excellent
  • Position: Excellent

Hardware: Since the onboard encoder is being used, the quadrature encoder must be connected to the 2 pins on the ABS port (aux2) using a JST ZH-4 connector.

  • ABS pin 1 – 3.3V
  • ABS pin 2 / aux2 pin 0 – Quadrature A
  • ABS pin 3 / aux2 pin 1 – Quadrature B
  • ABS pin 4 – GND

Configuration: This configuration assumes that a 5000 cycle per revolution / 20000 count per revolution encoder is being used. For that, the following changes must be made:

conf set aux2.spi.mode 1             # disabled
conf set aux2.pins.0.mode 4          # quad_sw
conf set aux2.pins.1.mode 4          # quad_sw
conf set aux2.quadrature.enabled 1   # enabled
conf set aux2.quadrature.cpr 20000

conf set motor_position.sources.1.aux_number 2
conf set motor_position.sources.1.quadrature 3    # quadrature
conf set motor_position.sources.1.cpr 20000

Since multiple encoders are being correlated, we need to manually inspect the motor_position.sources telemetry view to identify if both sources increase as the output is moved in the same direction. If not, then the sign of the quadrature source should be -1.

conf set motor_position.sources.1.sign -1

Finally, the output position is configured to be derived from the quadrature signal, with a reference source from the onboard encoder so that it starts at the correct absolution position.

conf set motor_position.output.source 1
conf set motor_position.reference_source 0

Discussion: For this example, a quadrature encoder of much higher resolution than the onboard encoder was selected. Thus the resulting control performance has improved control at very low speeds, and improved position stiffness at any speed. The relative performance gain is directly proportional to the increase in effective resolution.

8: RLS AksIM-2

The RLS AksIM-2 is an encoder that reads a ring with magnetic tracks encoded on it to produce high resolution absolute position. It can be used when a hollow shaft is desired, or when high performance velocity control or high stiffness position control is required.

  • Torque: Yes
  • Velocity: Excellent
  • Position: Excellent

Hardware: moteus only supports the RS-422 asynchronous serial configuration of the RLS AksIM-2. The exact part numbers used in this demonstration are:

  • MB049SFF17BDNT00 – encoder
  • MRA049BG034DSN00 – magnetic ring
  • ACC049 – development cable

It requires the two data pins on the ABS port of moteus r4.3/5/8/11. To use it, a RS-422 level transceiver is required. One option which has been demonstrated to work is the MAX3490. Additionally, the AksIM-2 requires 5V power, this can be provided via a boost regulator driven from the ABS port or a separate 5V regulator.

  • ABS pin 1 – 3.3V
  • ABS pin 2 / aux2 pin 0 – RX
  • ABS pin 3 / aux2 pin 1 – TX
  • ABS pin 4 – GND

Configuration: The required configuration is straightforward. This example will leave the onboard encoder configured, but it could also be disabled if not used. It also assumes that the AksIM-2 has been configured ahead of time (either at the factory or using RLS tools) to 1000000 baud.

conf set aux2.pins.0.mode 3   # uart
conf set aux2.pins.1.mode 3   # uart
conf set aux2.spi.mode 1      # disabled
conf set aux2.uart.mode 1     # aksim2
conf set aux2.uart.baud_rate 1000000

conf set motor_position.sources.1.aux_number 2
conf set motor_position.sources.1.type 2        # uart
conf set motor_position.sources.1.cpr 4194304   # 22 bits (for any AskIM-2)

conf set motor_position.commutation_source 1
conf set motor_position.output.source 1

Discussion: This example provides excellent control performance both at very low speeds, high position stiffness, and gives a hollow shaft capability. The primary downsides are complexity and cost. An active adapter is required, and in single unit quantities, the AksIM-2 and an encoder ring is more than twice as expensive than a moteus itself.

9: iC-Haus iC-PZ

The iC-Haus iC-PZ is a reflective optical encoder that reads a marked code ring. It is high resolution and accuracy and supports a number of communication interfaces. The only one that moteus supports currently is the SPI protocol.

  • Torque: Yes
  • Velocity: Excellent
  • Position: Excellent

Hardware: Like with the hall effect example, for moteus r4.8/11, this requires that a Pico-SPOX connector be populated on the ENC blank pads (TE 5-1775444-6) or that a cable be directly soldered to the pads. A cable must then be constructed to attach the external AS5047P to the mating Pico-SPOX connector (Housing: Molex 0874390601, Terminal: Molex 0874210100).

Additionally, while the iC-PZ can provide 3.3V compatible I/O pins, and thus no SPI level translation is required, it also requires a 5V supply. Thus either a boost regulator is required to use the ENC port provided 3.3V power, or a separate 5V supply is needed.

All of the pins are required:

  • G: Ground
  • 3: 3.3V
  • C: SPI chip select
  • K: SPI clock

Configuration: The iC-PZ requires a moderate amount of provisioning before it will produce usable values. While it is possible to do so using only diagnostic mode commands with moteus, the process is involved enough that it is not recommended. Instead, this configuration assumes that the encoder has been provisioned using the iC-Haus tools first.

conf set aux1.pins.0.mode 2  # spi_cs
conf set aux1.pins.1.mode 1  # spi
conf set aux1.pins.2.mode 1  # spi
conf set aux1.pins.3.mode 1  # spi

conf set aux1.spi.rate_hz 6000000
conf set aux1.spi.mode 3           # ic_pz

conf set motor_position.sources.0.cpr 16777216   # 24 bits

Discussion: The iC-PZ provides the same benefits as the RLS AksIM-2 above in terms of improved control performance and hollow shaft capability. It still requires adapter hardware with moteus, although the required hardware is slightly less complex than with the AksIM-2. Also, it is lower cost, but does require a more complex provisioning stage and optical cleanliness.

10: Fixed Voltage

Sometimes you can’t fit an encoder into a design, but you still want low speed operation. For those cases, moteus can be configured to drive a brushless motor as if it were a stepper motor with micro-steps. This means that there is no velocity or position feedback, and the motor will constantly dissipate a roughly fixed power, even when stationary or unloaded.

  • Torque: No
  • Velocity: Poor
  • Position: Poor / Incremental Only

Hardware: No additional hardware is required.

Configuration: Auto-calibration is not possible in this mode, but the required configuration is minimal. The number of pole-pairs must be entered manually if the output is to be scaled correctly, and the fixed control voltage must be specified by the designer. A larger fixed voltage will dissipate more power but provide more holding torque.

conf set motor.poles 14   # the number of pole-pairs for the motor

conf servo.fixed_voltage_mode 1         # enabled
conf servo.fixed_voltage_control_V 0.3  # selected by the designer

Discussion: In most cases where you want to drive a motor in a stepper like manner, a dedicated stepper driver is more appropriate. However, if you already have multiple moteus controllers in your system, or you want to use the advanced trajectory controls, this might be a valid design choice.

11: GPIO

Any of the pins on any connector can be used for 3.3V digital input or output. Some pins can be used for 3.3V analog input, and some can be used for 5V digital input. A dedicated “function” on the aux port is not required, so any pins which are not otherwise needed for a function can be so configured.

Hardware: moteus does not require any specific hardware, although if you want to use a given digital input or output to do something, you will likely need to construct at a minimum a harness to connect it to one of the ENC or ABS ports.

Configuration: Each pin can be configured independently:

conf set aux2.pins.0.mode 14    # digital input
conf set aux2.pins.1.mode 15    # digital output
conf set aux1.pins.2.mode 16    # analog input

Discussion: Once configured, the digital readings from each pin are accessible in the auxN.pins diagnostic mode tree and by using the 0x05e or 0x05f register mode registers as 7 bit integer bitfields.

Writing to the digital outputs can be conducted with the diagnostic protocol as follows

aux1 out 33   # a decimal bitfield

or by using register mode registers 0x05c / 0x05d.

Finally, analog inputs can be read in the diagnostic tree at aux1.analog_inputs, and in register mode at registers 0x060 – 0x06c.


While the flexible I/O subsystem already enables a lot of new configurations for moteus, keep on the lookout as it makes even more things feasible in the future!

Flexible I/O: Sink configuration

This will be the final post describing the fundamentals of configuring the new flexible I/O system. There have been a number of previous posts (part 1, part 2, part 3). In this iteration, we’ll cover how to configure the sinks that consume the “source” encoder data. As a reminder, the block diagram of the I/O system looks like:


To perform commutation with field oriented control, moteus needs to know the relationship between the rotor and stator in the magnetic domain. With the addition of the new flexible I/O system, some of the configurable values associated with this remain as they were, where there are some new ones.

First, the number of poles for the motor is still at motor.poles, and whether or not to invert the ordering of the output phases is at motor.phase_invert. Similarly, the theta mapping table has the same semantics before and remains at motor.offset.

Newly added is motor_position.commutation_source which controls which 0 indexed source is used to drive commutation.

It is shown in the block diagram above, but not discussed here yet are the cogging compensation parameters. They’ll be covered soon, I promise!


The other primary purpose of encoder data within moteus is to act as feedback to the position control loop. For this, the source to use can be selected with:


Additionally, an offset and sign can be configured at this stage with motor.output.offset or motor.output.sign. A gear reducer can be configured by entering a non-unity value for


Values here will typically be smaller than 1. For instance, a 1:4 gear reducer would be 0.25.

Finally, to support disambiguation after a reducer with low-performance sensors, a second source may be configured for that purpose with:


The “reference source” will be consulted only at system power on after it is valid, in order to resolve the ambiguity resulting from the reducer. It can also be used if the output is incremental, but another absolute source is present that is lower quality or lower rate, to reference the output at power on.


There are no configurable values necessary to operate any of the new I/O system from the application level. There are new registers that can be used to read the position and velocity from each of the sources, whether or not they are configured for any sinks. Similarly, the digital input, digital output, and analog inputs can all be operated from the register or diagnostic protocol. All of these can be seen in the reference documentation.

Flexible I/O: Source configuration

In the last two posts (part 1, part 2), I started talking about the new, more flexible I/O subsystem for the open source moteus brushless motor controller. In this post, I’ll continue by describing what a “source” is, and how it is configured.

For reference, the block diagram showing how auxiliary ports, sources, and sinks are related is below:

Each “source” in the above diagram represents a single encoder. To the sinks it provides a position and velocity, along with various validity indications for that data. Each has three basic configuration components: where to get the raw data, how to transform that raw data, and the low-pass filter configuration. We’ll cover each in turn.

Source raw data location

The location where a source retrieves raw data is selected by picking the auxiliary port number to use (1 indexed, so 1 is auxiliary port 1) and which function within that auxiliary port the data should be retrieved from. For instance, if we want a quadrature encoder connected to auxiliary port 2 to feed into source 3, we would set:

motor_position.sources.2.aux_number = 2
motor_position.sources.2.type = 3

The full list of possible function identifiers can be found in the reference documentation, and tview presents a drop-down with names.

There are two other minor complications. The first, is that for I2C functions, a separate configurable value is used to select which I2C device should be used.


If an “incremental” raw source is used, like quadrature, then an index function can be used directly at the source level to allow the source itself to report a correct absolute reading. This is necessary if the source is to be used for commutation. It is configured by entering the 1 based auxiliary port number into:


Source transforms

The are a number of transforms applied in sequence, each of which may be configured:

  • Offset: This is an offset in count space to add to the raw value before any other transformation is performed.
  • Sign: This may be 1 or -1. If -1, then the value will be inverted.
  • CPR: This is the “counts per revolution” and must be configured. For auxiliary port functions which have a CPR configuration, this must match it.
  • Eccentricity Compensation: A table of 32 points that describe percentage offsets from a perfectly linear response. It can be generated through a separate calibration tool, and all 0’s performs no eccentricity compensation.

The nominal output after all transforms have been completed is an angular position value between 0.0 and 1.0.

Low pass filter

The PLL based low pass filter is the final stage in the source pipeline. It is configured through a single parameter to select the 3dB cutoff frequency:


If the encoder is intended to be used for either commutation or for output position control, this frequency should be at least as large as the mechanical bandwidth of the system and at least as large as the torque bandwidth for stable control. Higher values than that trade off audible noise versus control performance.

Reference frame

Each source can be configured to be in one of the “rotor” or the “output” reference frames. This controls whether the rotor to output ratio is applied when using the value for either commutation or output control.


Where 0 is used for rotor and 1 is used for output.

Nearly done with configuration

I’ll tackle the final part of configuration in the next post, the sinks that use this encoder data.

Flexible I/O: Auxiliary port configuration

In the last post, I covered the goals behind more flexible I/O support in the moteus brushless controller. This time, I’ll start to cover the configuration model that I implemented to make that support work. It is broken up into 3 distinct phases, auxiliary ports, sources, and sinks.

Slightly simplified I/O structure flow diagram

Auxiliary port pin configuration

To begin with, the available connectors and external pins on moteus are organized into “auxiliary ports”. For the moteus r4.3/4.5/4.8/4.11, the correspondence is that the external primary encoder connector, if present (r4.8 and newer), is “auxiliary port 1”. The ABS port and some on-board debug pads are “auxiliary port 2”. For each port, there are two levels of configuration, at the pin level and the function level.

At the pin level, each pin can be assigned to exactly one possible function from the available set. The possibilities include each of the possible encoder functions, as well as digital input output, or analog input. For each board, not all pins are capable of all functions, if an unsupported configuration is attempted, it just results in a runtime error. The current set of capabilities for the connectors available on moteus r4.11 looks like:

Some modes can be assigned to any pin no matter the board version. These modes are digital input or output, hall effect input, software based quadrature, and index input.

The pin configuration is selected by a single enumeration for each pin, located at aux[12].pins.X.mode. Additionally, a pull-down, pull-up, or open-collector can be configured for each pin at aux[12].pins.X.pull, although not all functions and pins support all pull-up values. For instance, analog inputs do not support pull-up or open-collector, and digital outputs support no pulls at all.

Note that the pin number in configuration is the “logical pin” number and is listed in the “Aux” column of the above table. It basically counts from 0 starting at the first pin available for input or output.

Auxiliary port function configuration

After the pins are configured, each of the possible functions on an auxiliary port has a configuration section which allows it to be enabled, and in most cases provides additional options as well. If a function is enabled, but pins with appropriate capabilities are not configured for that function, then a runtime error is generated.

The functions that are currently implemented and their configuration options are:

  • SPI: Whether an AS5047 or a iC-PZ encoder is present, and what bit-rate the SPI bus should operate at.
  • UART: The only supported type currently is the RLS AksIM-2. The baud rate and polling rate can be configured.
  • Quadrature: The counts per revolution.
  • Hall effect: A bitfield describing the polarity of the 3 hall effect inputs.
  • Index: None
  • Sine/Cosine: The neutral value can be configured as measured by a 4096 count ADC converter (this usually must be calibrated).
  • I2C: The I2C bitrate and mode can be configured, along with up to 3 different I2C devices. For each device, the address and polling rate can be selected. Currently available devices are the AS5600 or the AS5048A.

Notably, with the exception of I2C, only one instance of each function can be enabled for a given auxiliary port. Thus it is not possible to have two independent quadrature encoders connected to a single auxiliary port, even if the available pins would otherwise permit it. Digital input, digital output and analog input do not require a “function”, so as many of those may exist as desired.

Next up

In the next post, I’ll cover how the sources section of the above diagram is configured.

Beginnings of more flexible encoder and I/O support

The moteus controller, being a brushless servo drive, needs to use encoders to measure things like how the rotor is positioned relative the stator, and possibly output shafts that have passed through a reducing stage. The support for this has gradually expanded over time, but is still relatively limited as far as those things go. The available options are:

  • Primary encoder (used for commutation)
    • The onboard AS5047P
    • An external AS5047P
  • Auxiliary encoder (optional, for measuring the output shaft)

However, the moteus hardware has always been capable of more, both because the processor is a very capable one, and the exposed IO pins are relatively flexible. While looking at some future designs that incorporate even more IO options, I decided it was time to update the firmware to finally start taking advantage of that flexibility.

My goals can be broken down into two parts. First, what types of IO (input/output) to support, and second, how those inputs and outputs can be used.

I/O Support

For the types of encoders, there are first a number of “primitive” types that are somewhat like common interchange formats for encoders:

  • Quadrature – This uses two signal lines which cycle through a fixed pattern to indicate motion and direction.
  • Index – Here a single digital line is used to indicate when the rotating device is at a fixed position.
  • Sine / cosine – Two analog values report the sine and cosine of the angular position.
  • Hall effect sensors – Many brushless motors have 3 hall effect sensors permanently mounted inside which, while it cannot determine the position of the rotor relative to the stator fully, can determine the electrical phase configuration.

Second, there are a number of digital sensors which are relatively common in servo applications that are moderately convenient to interface.

  • iC-PZ – This is an absolute optical reflective encoder from iC-Haus which has relatively loose mechanical tolerances and very high resolution. It supports a number of protocols, multiple of which could work with moteus. Many diameter code rings are supported for hollow shaft applications.
  • RLS AksIM-2 – This is an absolute magnetic encoder with similar performance properties to the iC-PZ and also supports a variety of code ring diameters.

Third, the exposed pins of moteus could be used for application specific purposes:

  • Digital input – Some of the STM32 pins support 5V tolerant inputs, and all support 3.3V
  • Digital output
  • Analog input
  • Step/direction control
  • RC PWM/PPM input

How they can be used

There are a number of ways that encoders can be used, and it would be nice to be able to use some or all of them in different applications:

  • Electrical commutation – Determine the electrical phase for performing field oriented control.
  • Output position control – Provide feedback to the servo position and velocity control loop.
  • Disambiguate gear reducers – When an absolute sensor is present before a gear reducer, a separate sensor of lower quality can be used to determine in which sector of the output the reducer is currently in.
  • Application monitoring – In some cases, moteus may not need to use the value at all, but the application would like to monitor the position or rate of an encoder that happens to be colocated with a moteus.

Next steps

In the next post, I’ll take a look at the configuration model I implemented to support these goals in moteus!

moteus r4.11

Here’s yet another update to the moteus line, moteus r4.11!

r4.11 is electrically, mechanically, and software compatible with r4.3, r4.5, and r4.8.

This revision supports two alternate footprints for the CAN-FD transceiver to better support component availability and refines the power stage for the DRV8353 gate driver. moteus r4.8 was the first version to use the DRV8353 because of, once again, component availability issues. However, it was developed on a very abbreviated schedule. With r4.11 the EMI is much improved over r4.8 and r4.5, and the efficiency is much better than r4.8 at all input voltages and PWM frequencies.

PWM Frequency24V r4.1124V r4.836V r4.1136V r4.8
15kHz95% / 0.28W94% / 0.20W95% / 0.30W92% / 0.25W
40kHz93% / 0.35W88% / 0.25W90% / 0.38W85% / 0.30W
50kHz91% / 0.36W84% / 0.30W89% / 0.40W84% / 0.35W
moteus r4.11 and r4.8 thermal efficiency and idle power driving nearly stationary motor

The matching development kit will be available shortly, once the r4.8 developer kits sell out.

Velocity and acceleration limited trajectories

One of the oldest requested features for the moteus brushless controller has been a form of trajectory control beyond constant velocity trajectories. For most applications this is not an actual deal-breaker, because arbitrary trajectories can be approximated by piecewise linear constant velocity trajectories in the application layer. However, for many people, that is big hurdle to jump over to start with, and for some, it can actually limit application effectiveness because a fair amount of CAN bandwidth is required to achieve the high rate control necessary for smooth motion.

So, as of release 2022-04-07, moteus now supports velocity and acceleration limited trajectories out of the box. All devkits now come with human-eye-pleasing limits enabled by default, although bare boards leave it disabled as per board defaults. There are two parameters to control the feature:

  • servo.velocity_limit – The maximum velocity used when moving to a particular target goal
  • servo.acceleration_limit – The maximum acceleration used when moving to a particular target goal

Additionally, these limits can be overridden on a per-command basis, both using the diagnostic protocol “d pos” mechanism, and the register implementation.

Check out this video, then read below the fold for more details:

There are a few details to the implementation which help it fit better into the existing moteus control framework, while enabling some features that don’t exist elsewhere.

Special-valued limits

First, either limit may be “nan”, (the usual special value in moteus configuration-speak). In that case, no limits are applied. If both are “nan”, that results in identical behavior to what was previously used in moteus, in that a “d pos” command immediate attempts to achieve the desired position and velocity. If velocity is limited, but not acceleration, all such commands result in constant velocity trajectories to reach the target position, followed by continuing at the target velocity indefinitely. If acceleration is limited but not velocity, then the velocity will be continuous, but unbounded (except possibly by servo.max_velocity). If both are limited, then the velocity will be continuous and bounded.

Non-zero velocity targets

The target state is a position and a velocity. moteus will reach the position and be traveling at the desired velocity when it reaches it. After reaching the state, the controller will continue at that target velocity indefinitely, or until another command is received.

Idempotent commands

For many cases, it is still possible to construct commands such that they are idempotent. That means that you can send the same command over and over and it will not affect the semantics of the control. This is particularly easy if the destination state has a zero target velocity.

If the destination state has a non-zero velocity, then the command will need to change sufficiently far in advance to avoid “looping around”. If a command with a non-zero velocity is received after the machine has passed through the target state, then the only way to achieve it is to slow down, go back, and accelerate again so as to be moving at the proper speed. For applications like these, it is best to always have the target be at least several control cycles in the future so there is no risk of such looping.

Completion indication

It is now feasible, in many cases, to send moteus control commands very infrequently. A complete motion from stopped in one position to stopped in another can be initiated with a single command. So that the application code can advance in a timely manner, moteus now reports a “completion” indication that flags when the target position and velocity has been reached. This can be accessed as register 0x00b in register mode, or as servo_stats.trajectory_done in diagnostics mode.

Interaction with the “stop position”

Previously moteus had the “stop position” mechanism to emulate a constant velocity trajectory with a fixed endpoint. That mechanism is still present. While it has defined interactions with the acceleration and velocity limits, they are, shall we say, “not particularly useful”. Thus, it is recommended to not use the “stop position” feature combined with the velocity and acceleration limits unless you really know what you are doing.

Jerk limited trajectories

The implementation in moteus only supports limiting acceleration, not jerk. Thus the acceleration is allowed to be discontinuous. For an application where jerk limited trajectories are required, piecewise interpolation is still required. For accelerating phases, this can be trivially accomplished by slewing the acceleration limit override in each moteus command. For decelerating phases, or where precise control over position is required, a full fledged jerk-limited planner (like ruckig) will be required. The output from such a planner is probably easiest to feed to moteus as a piecewise series of constant velocity trajectories with no limits configured as before.

Solid model of wcubed Pocket NC vise

Some time ago I wrote about using the wcubed vise for the Pocket NC. While I don’t end up using it very often any more, mostly because I rarely work with rectangular stock, it can be useful from time to time. Unfortunately, it is no longer manufactured. In case anyone is interested in replicating it, I’ve taken at least a minimal stab at modeling it up based on measurements of my unit along with necessary hardware as picked from McMaster. I suspect the model should be good enough to get something that works.

Note, that there are a number of quirks and annoyances that you’d probably want to fix if you did make more of these:

  • The main drive screw is an imperial #8-32. Ideally you would use a metric one, as everything else on the PNC is.
  • The nut that you use to tension is annoyingly small and easy to strip.
  • There is no reason to have holes for only 2 alignment pins, it might as well have 4 at least.

The bigger problem is that as the jaws are aluminum, you can’t actually apply a lot of clamping pressure before stripping out the threads. But, as long as you keep that in mind, it is serviceable.

With those caveats out of the way, here’s the link!