Real-time bus

Overview

On this page:
Duplex | Frames | Timing | Cycles | Latency

Abstract

The real-time bus is designed to operate with low and predictable latency even during peak load conditions.

Duplex

There are two transfer directions,

Downstream (from PC to the modules)
Upstream (from the modules to the PC)

The upper bus levels (USB and FDX) are running full duplex and transmit up- and downstream simultaneously. The lower levels (HDX and EDG) are running half duplex and toggle between the two directions. Because of this, the net bandwidth of HDX and EDG is half compared to FDX. The FDX level is using the additional bandwidth for its real-time v-channels, which are only available on this bus level.

Any component requiring real-time v-channels must be on the FDX bus. Powerboards and Boosters are examples of components using real-time v-channels.

Bus Frame

RTB uses fixed sized frames with 1024 bytes each. The reason for large frames,

Atomic execution across multiple devices
Synchronous reporting of telemetric data
Less frequent arbitrations on half duplex busses (HDX & EDG)

With an average frame rate of 120[F/s], RTB transmits 120 [kB/s] downstream and simultaneously 120[kB/s] upstream, giving a total throughput of 240[kB/s] full duplex over the FDX bus. In addition, each bus cycle addresses multiple destinations simultaneously, providing an atomic execution characteristic across multiple devices. Currently a bus cycle addresses in parallel,

16 Powerboards & DCC generators
896 track sensors
3 accessory devices

FDX Frame

The FDX bus sits right below the USB layer and operates full-duplex sending data in both directions simultaneously (downstream / upstream).

Since USB runs at 3 Mbps and FDX runs at 2 Mbps, I created a downstream bandwidth oversubscription of 1.5:1 at the FDX layer. This ensures that the FDX bus always has data for downstream transmission and always enough bandwidth upstream.

HDX Frame

The HDX bus operates half-duplex reversing bus direction halfway in the middle. So the net payload is somewhat equal for both directions. In addition, the HDX bus provides asynchronous upstream event reporting capabilities.

EDG Frame

Likewise, the EDG bus operates in half-duplex mode and has similar event reporting capabilities. The main difference to the HDX bus is on the physical level.

Bus Timing

The RTB bus is tightly integrated with the DCC protocol. This is perhaps a unique feature of the bus. Each DCC command represents a “transaction tick” on the bus. This tick is also called a “bus cycle” or “generation tick”. To efficiently use the available bandwidth, the bus transmits large frames (1 Kbyte per frame). Typically 100-120 DCC commands are generated per second and due to the coupling, RTB transmits exactly the same number of FDX/HDX/EDG bus frames. In summary, RTB frame rate depends on DCC frame rate.

Precise micro timing is important for both FDX and HDX. For this reason, the microcontrollers connected to the FDX and HDX bus must have crystal oscillators (<= 50 ppm). At the beginning of each bus cycle, all bus participants are synchronized against the bus master time. By design, the EDG bus does not require this precise timing, allowing accessory modules to run on built-in RC oscillators.

Bus Cycles

All commands sent in a single RTB frame are executed simultaneously on all bus layers (FDX, HDX, EDG). In addition, results are also sent upstream simultaneously, regardless of which bus level they were executed on. This can be described as a bus transaction across multiple bus participants.

FDX

The FDX bus level implements a 4 stage instruction pipeline,

Fetch: Receive new frame from FDX
Exec: Execute command
Merge: Merge result data from HDX bus
Post: Return result on FDX

In one bus cycle all 4 stages execute simultaneously. Assuming a single bus cycle takes 10ms, a command roundtrip yields 40ms (Fetch + Exec + Merge + Post). Because this operates as a pipeline, every 10ms a command is executed.

HDX

The HDX bus level implements a 3 stage instruction pipeline,

Fetch: Receive new frame from HDX and relay the data to EDG
Exec: Execute command
Post: Merge HDX + EDG result and return result on HDX

The HDX bus does not have a Merge stage. Instead, it send its result upstream one step earlier, so it can be merged on the FDX layer.

HDX

EDG

The EDG bus implements a 3 stage instruction pipeline as well,

Fetch: Receive new frame from EDG
Exec: Execute command
Post: Return result on EDG

Like the HDX level, the EDG level also eliminates the need of a Merge stage. The upstream data from the EDG bus will be handled by cut-through-forwarding on the HDX bus level.

Latency

My RTB design delivers predictable, low-latency. This is a very important attribute of this layout bus as it ensures events being transmitted to and from the control station in-time.

There are three timing metrics defined,

Service Time (SvcTm): time it takes for a command to execute
Bus Time (BusTm): Service time plus the time for the data transmission
Response Time (RspTm): Bus time plus the queuing time on the PC

The absolute value of round-trip latency (40ms) is not the only measure of quality. The second important metric concerns the variation in latency. And here too we see a very stable horizontal stability characteristic.

During the initial layout discovery after a system start, frame collisions may occur on the bus, which negatively affects latency. Once the discovery processing has completed, the bus transitions to its operational mode and the frame latency stabilizes between 40 and 50ms.