The Software Defined Blocks project uses an ESP32 with sensors to emulate block activations for a train model railroad.

The desire for the MVP is to be designed around “modules”.

Modules

Each module should have:

• An init method called when the main starts.
• A start method called once all other modules have been initialized.
• A loop method called as part of the main loop.
• The loop method will return the max time it wants to sleep before being called again. The loop method may be invoked sooner than requested, but not later.
• Its own thread-safe message queue.

Each module is free to start a thread or do their process in the main loop.

The main loop will call all the modules’ loop methods, and sleep the minimum time requested by all modules (similar to a classic Arduino sketch main loop).

Care should be taken to avoid having two modules use the same hardware resource -- e.g. the same GPIO pin, or the same I2C controller, etc. One way to ensure this would be to have a global “hardware resource allocator”, e.g. an object which “owns” all hardware resources and panics if a resource is reused without being freed first. This seems overkill in the context of the phase 1 MVP.

Module Manager

Modules are registered in a global ModuleManager singleton.

The module manager is also responsible for maintaining a thread-safe message queue per module.

Each module has a unique identifier code name which will be 2 characters long (see below).

Message Queue

Inter-module communication is done by posting a message on another target module via the manager rather than directly. Modules regularly pool their own message queue.

Although we could use a notification object to wake up a thread when a message is posted, I do not anticipate having to do that in the first version.

The goal of the message queue is to simplify the need for synchronization if two threaded modules call each other, and to simplify the dependency between modules. We want to avoid one module calling methods from another module directly.

The downside of a message queue is that it is an inefficient mechanism to share large amounts of data by copy, which we posit is not needed in phase 1. This would not work to share image buffers, but it’s enough to share e.g. pointers.

Data Store

The message queue is also a poor way to share frequently changing data between modules, as well as to handle configuration data.

Thus a suggestion is to have a singleton global dictionary as a Data Store. Some keys are transient data, whereas others are backed up to NVR.

Whereas it makes more sense for each module to have its own message queue, the data store is a global singleton. Accessors must be thread safe as they can be used by any threaded module.

The data keys are strings, and one character in the string key indicates whether the value should be automatically loaded/saved into the NVR.

The data keys strings should be interned to speed up access, which may mean enforcing a size limit. A potential key string encoding would be something like:

• 2 characters for the module name.
• 1 character indicating whether the value should be automatically loaded/saved into the NVR.
• 7 characters max for the actual key.

Note that since this is Python, the values themselves can be anything, including object references or dicts, etc. However for these values that must be backed to the NVR, we’ll need to check that API and conform to its limitations.

In the first version, the data store will be limited to storing strings and long/ints, which can both easily be saved in the NVR.

The store is potentially accessible by many threads. We want to ensure consistency in reads & writes, not to mention that dictionaries/maps typically do not have thread safe updates. Thus we expect to have to use locks around read/writes.

To reduce locking contention, one option is to have a lock per target module since all the keys “belong” to a specific module.

Using a global dictionary is a bit more expensive than writing just global variables. However it is deemed an adequate compromise to prevent strong module interdependence.

That means modules should consider store accesses to be expensive. As an example, we can anticipate that the wifi / http module would update the sensor threshold configuration by writing a maximum of a handful of values, then send a message to the adequate module to reload their configuration; that module would read values once from the store and keep them in their own object attributes.

We probably want an API that allows a module to read & write multiple values with a single synchronized access, even possibly a get/condition/update access.

Block Logic

The core of the project is defining blocks, and the conditions that trigger them.

We need to elaborate on the possible conditions here.

We have 2 options here:

• Use a dedicated module that has the configuration data, reads the state of sensors from the store, and then updates the block state in the store.
• There is no dedicated module; instead each sensor module has its own logic to update block state in the store.
• In both cases, a separate JMRI JSON/MQTT module reads that block state from the store, and sends the JSON/MQTT to JMRI when a change is detected.

We’ll start with a dedicated “block module”.

The configuration consists of output blocks (JMRI system names).

For each block, an input is configured with its own data. For a ToF sensor, that data is the distance threshold, and whether the block is occupied or free when the sensor is below that threshold. To be clear, that means the ToF module only needs to export distance information, and the block module is the one reading the sensor(s) distance and computing the state of the block. That also means the trigger logic will be depending on the type of sensor, and that logic lives in the block module rather than the sensor module.

We may want to include debounce capabilities in there in case the measurements are noisy.

Foundations:

• Module Manager
• Message Queue (accessed via Module Manager).
• Data Store.

Expect modules:

• Display.
• Owns the OLED + its I2C driver.
• Read store for variables to display (e.g. a simple location code + value).
• Wifi + HTTP.
• Owns the wifi, handling reconnections.
• Owns sending a web page over it.
• Read store for states to display on that web page.
• On configuration change, write to store + notify modules.
• ToF sensor.
• Manages i2c sensors.
• Optional: filter/smooth the readings, discard spikes.
• Regularly reads sensor values and updates them into the store.
• Block Module.
• Regularly reads store for sensor states.
• Updates blocks states in store.
• Notified by Wifi / HTTP if configuration has changed.
• Notify JMRI JSON module when there’s a change.
• Notify JMRI MQTT module when there’s a change.
• (we’ll support both protocols upfront)
• JMRI JSON/MQTT Module.
• Manages its own socket to the JMRI server.
• Notified by Wifi / HTTP if configuration has changed.
• Notified by Block module if blocks states have changed.

---