A few months ago, I changed the automation to use the new Conductor 2 scripting engine as the default. It worked very nicely for a while, and in the last few weeks the behavior had degraded: the mainline automation was ending up in error recovery almost every day. The passenger train would be running fine, yet would end up with the automation in error.

There’s an interesting side effect here: One of the points of Conductor 2 is to have error recovery, being able to recover when things go wrong. Thus from an outsider’s point of view, the automation was “working” as in “doing something”, although from my point of view it was operating in a degraded running mode.

Eventually I had enough, reverted the automation computer to run the Conductor 1 script, while I investigated the issue to find the root cause.

First issue was figuring out what was not working, ending with the automated route in error recovery mode? I realized I simply did not have enough information in the logs to understand that after the fact.

That led me to enhance my logging: log error cases, log the root cause of an error, and log how long each sensor is triggered. I updated the logging, let it run for a while, and then captured one typical error:

14:56:20.416 D 8312 : Horn

14:56:21.117 D 8312 : -12

14:56:34.925 S S/NS771 B321 : ON

14:56:34.926 B S/NS773 B330 : TRAILING after 22.56 seconds

14:56:34.926 B S/NS771 B321 : OCCUPIED

14:56:35.696 S S/NS773 B330 : OFF

14:58:15.785 S S/NS771 B321 : OFF

14:58:35.012 R Sequence Mainline #3 Passenger (8312) : ERROR Sequence Mainline #3 Passenger (8312) current block {B321 [NS771]} suddenly became non-active after 120 seconds.

14:58:35.012 R Sequence Mainline #3 Passenger (8312) : ERROR

14:58:35.012 R Sequence Mainline #3 Passenger (8312) : ERROR Sequence Mainline #3 Passenger (8312) current block {B321 [NS771]} still occupied after 120 seconds.

One of the cornerstones of maintaining any automated process, especially daily automated running of little HO-sized trains on a public music model railroad, is to have dashboards to track the status of the automated processes and being able to see at a glance whether they are healthy.

For example this is part of a dashboard that I look multiple times a day:

My dashboards have a few moving parts:

• I use the GA Measurement Protocol to send custom pings using my own Analytics.java in Conductor / RTAC.
• Google Analytics receives the “page views” and the events in a “web” property.
• Looker Studio is used to build the dashboard.

I have detailed in a previous post how I create my “car/cab ride” videos: a Mobius Maxi 4K is placed on a flat car, and either pulled or pushed by the train using a custom “3D-printed rod” draw-bar connector:

The only issue with this approach is the extremely visible rod on the camera, even after cropping a 1080p out of the 4K video. I’ve had many comments that it’s quite distracting; I fully agree; it’s been bugging me too:

I tested Conductor 2 with script 53 while the rest of the team was cleaning the layout and it worked reliably.

Thus I enabled it as the default for the automation computer.

Basic recovery was also tested and worked in the “ideal” conditions.

Known cases where recovery will not work:

• Branchline on virtual block B830.
• PA/FR on flaky block B321 not registering the train stopped.
• Any train stopped across two blocks and activating both blocks.

The plan for B830 is to convert it to a real block sensor.

The plan for B321 is to change the sensor to another type than the BD20 with hopefully better sensitivity -- I’m especially interested in trying the new ATOM DCC Block Detector (CKT-BD1) from Iowa Scaled Engineering.

Dealing with the train stopped and activated two blocks will need more work, either on the Conductor engine side or on the script side. I need to think more about how I want to handle that one. One option is for the recovery script to be able to mark 2 consecutively occupied blocks as “occupied + trailing” since that’s really the state they are supposed to be in. Then the normal engine behavior will take care of it properly.

Now I understand the issue that was affecting the automated freight train; it's quite interesting and worth elaborating.

First, as a recap, the freight route is a shuttle composed of 2 blocks. The train starts on say block A, travels to the consecutive block B, stops, and reverses to block A. The sequence is very simple:

Step / State Block A                  State Block B                Description
1- [ A  active   occupied  ]   [ B inactive   empty    ]        Start from A
2- [ A inactive  trailing  ]   [ B  active   occupied  ]        Crosses from A to B
2- [ A inactive  trailing  ]   [ B  active   occupied  ]        Stops & Reverses
3- [ A  active   occupied  ]   [ B inactive  trailing  ]        Crosses from B to A
4- [ A  active   occupied  ]   [ B inactive   empty    ]        Stops back on A and resets

Note how although we have 5 logical steps, there are only 4 different steps from the sensor/block occupancy point of view -- there is no difference whether the engine is running on block B or stopped and waiting till it reverses.

I’m still trying to deploy Conductor 2 and it’s clearly not ready yet. I keep finding issues, some expected, some not.

The Freight automation has been an interesting source of such unexpected issues. It’s a simple reversing shuttle automation across two blocks, yet it has proven more challenging than the complex Passenger and Branchline automations -- these two actually perform quite well. Yet the “simpler” Freight one does not.

The most notable “bug” here is with the Freight automation: the engine starts on the first block, travels to the second block as expected…, and then instead of reversing the engine back to the beginning of the route, it decides the engine is back on the first block when it’s clearly still on the second block. At that point the automation panics because the engine is obviously not back at the starting point. The programming is entirely skipping half of the route automation, for no apparent reason. Why?

One of the issues which I was more or less expecting was dealing with the “flaky” sensor on B321, as I explain in detail on that post on the other blog. There’s some of that here, but there’s also something else entirely going on that I have not figured out yet.

Even after a bit of adjusting, the mountain sensors are still flaky, notably B321 and B370 with the Passenger train. I adjusted B321 for the Freight train but the newer PA engine is still borderline.

Of importance, these sensors detect current consumption. In the case of the reversing train, the sensor turns off when the train is stopped.

Here’s the new syntax:

        val T1 = throttle(1001) {

            onLight { b -> throttle.f0( b) }

            onSound { b -> throttle.f8(!b) }

            onBell  { throttle.f1(it) }

        } named "Throttle1"

        on { S1 } then {

            T1.light(true)

            T1.sound(true)

            T1.bell(true)

            T1.horn()

        }

Note that it seems quite annoying to have to place the “named ‘Xyz’” after the function block, so this syntax is also allowed:

Let’s implement virtual blocks. These have been part of the original design and I now have a need for them.

As a reminder, what is a virtual block and what is it good for?

Let’s look at the Branchline. It’s is a simple straight chain of blocks as shown in yellow on this map:

Because the track is curved and goes through tunnels, it looks complicated, yet it’s really a straight line:

• [ B801 parking ] ⇒ [ B820 station ] ⇒ [ B830 canyon ] ⇒ [ B850 tunnel ] ⇒ [ B860 reversing block ]

One thing not obvious on this map is that B830 has no track sensor associated with it. When I did the original sensor wiring back in 2017, I simply couldn’t identify the corresponding track feeders, and I suspect (or hope) there are actually two blocks in there because it’s a fairly long segment. That’s why I left a B840 in my naming scheme, for potential later usage.

But now the concept behind the Conductor 2 automation is for the engine to track on which block the train is supposed to be, and match that with the reality of the track sensors. That’s a problem in this case since after moving out of block B820 we have no idea… the track appears empty, and that’s a definite error condition. How do we solve that in the script?

Right now we load a single map via the Conductor script. That SVG map is displayed on the remote tablet and also on the automation computer’s main screen.

I want to change this to have different maps displayed in the Conductor Ui2 on the larger screen versus on the tablet’s remote status display. The original map data exchange format over KV Server was designed to allow sending more than one map to the tablet. The original design for RTAC was to have a list of map fragments in the tablet’s horizontal pager. Then users could just swipe to show different maps, each one highlighting one part of the layout. The initial implementation took the shortcut of displaying just one map. Same goes for the Conductor Ui.

As noted above, during the first test running “live” with JMRI, all routes instantly became in error as soon as the engine moved and activated the next block for BL or PA.

This is due to the simulator being too naive: it instantly changed from one block to the next one, in perfect sync. This made it “perfect” when I was debugging the engine. But unfortunately, that’s not how reality works: when an engine crosses a block boundary, both blocks become temporarily active since the engine bridges both blocks. And depending on the speed of the engine, this can be anywhere from a second to a handful of seconds.

It’s actually even a bit worse than that as even non-powered car wheels can bridge two blocks for a fraction of a second.