The Shay running at Randall after DCC conversion.

I recently got a Bachmann Spectrum Shay -- the “80-ton 3 truck Shay” #81901. The one I got is an unlettered black version, used, and in good condition. All gears work. It has an 8-pin NMRA / NEM652 connector, and it screams to be converted to DCC so we’ll discuss this effort here.

The Bachmann Shay before DCC conversion, as unboxed.

Initial Observations

Here are the initial observations:

In the latest version of Conductor 2, I’ve added “end-of-route statistics”. I’ve now accumulated enough data to start looking at it.

First, I needed to parse a JSON log file (a log file with one JSON entry per line per day per run), and I needed a way to convert that to CSV to dump it in a Google Spreadsheet. I briefly considered writing a short python script to do that, then realized that nothing can beat the awesomeness of sed, so I ended up with this:

I regret nothing. <insert appropriate meme>

I shall note that this expression purposely only selects log lines that have no errors, that is, successful runs that have completed with no issue. This will become relevant later.

Now let’s look at the collected data.

Freight Route

This is a simple shuttle route over two blocks: we’re moving from block B311 (the Stockton Station) to B321 (the Sonora Station), stopping there, and then back again to B311. The way I wrote the route stats aggregator, I add an index number when a block is repeated, so “B311.1” means “block B311 when leaving”, and “B311.2” means “block B311 when coming back”.

I have 98 runs for this one, and when we compute the stats we get a very nice result:

The latest fix for Conductor 2 using a virtual block for B504 is now installed and has been working very well.

The latest change I did to Conductor 2 is to compute end-of-route statistics. Each time a route has finished, we log the time spent on each block as well as the error status of the route. This is sent to a little server over a basic JSON RPC, and in the statistics/status page we get a quick overview of the latest route -- how many times we ran that route during the day, when the last one finished, and how many seconds the train spent on each block:

This also keeps track of all the recovery attempts.

Having the time per block should help me fine-tune the automation script later. We can see the time spent on B321 for example, and directly relate that to the observations made in the previous post.

Finally I also have the info in the server log, which is rotated daily, which means I could compute statistics later. I’m interested in getting the time spent on each block for long term analysis. Can we get trends out of it? How consistent are these running times? For example, could we detect erratic behavior, or do engines suffer some kind of loss of performance before they stop working?

A few months ago, I changed the automation to use the new Conductor 2 scripting engine as the default. It worked very nicely for a while, and in the last few weeks the behavior had degraded: the mainline automation was ending up in error recovery almost every day. The passenger train would be running fine, yet would end up with the automation in error.

There’s an interesting side effect here: One of the points of Conductor 2 is to have error recovery, being able to recover when things go wrong. Thus from an outsider’s point of view, the automation was “working” as in “doing something”, although from my point of view it was operating in a degraded running mode.

Eventually I had enough, reverted the automation computer to run the Conductor 1 script, while I investigated the issue to find the root cause.

First issue was figuring out what was not working, ending with the automated route in error recovery mode? I realized I simply did not have enough information in the logs to understand that after the fact.

That led me to enhance my logging: log error cases, log the root cause of an error, and log how long each sensor is triggered. I updated the logging, let it run for a while, and then captured one typical error:

14:56:20.416 D 8312 : Horn

14:56:21.117 D 8312 : -12

14:56:34.925 S S/NS771 B321 : ON

14:56:34.926 B S/NS773 B330 : TRAILING after 22.56 seconds

14:56:34.926 B S/NS771 B321 : OCCUPIED

14:56:35.696 S S/NS773 B330 : OFF

14:58:15.785 S S/NS771 B321 : OFF

14:58:35.012 R Sequence Mainline #3 Passenger (8312) : ERROR Sequence Mainline #3 Passenger (8312) current block {B321 [NS771]} suddenly became non-active after 120 seconds.

14:58:35.012 R Sequence Mainline #3 Passenger (8312) : ERROR

14:58:35.012 R Sequence Mainline #3 Passenger (8312) : ERROR Sequence Mainline #3 Passenger (8312) current block {B321 [NS771]} still occupied after 120 seconds.

One of the cornerstones of maintaining any automated process, especially daily automated running of little HO-sized trains on a public music model railroad, is to have dashboards to track the status of the automated processes and being able to see at a glance whether they are healthy.

For example this is part of a dashboard that I look multiple times a day:

My dashboards have a few moving parts:

• I use the GA Measurement Protocol to send custom pings using my own Analytics.java in Conductor / RTAC.
• Google Analytics receives the “page views” and the events in a “web” property.
• Looker Studio is used to build the dashboard.

I have detailed in a previous post how I create my “car/cab ride” videos: a Mobius Maxi 4K is placed on a flat car, and either pulled or pushed by the train using a custom “3D-printed rod” draw-bar connector:

The only issue with this approach is the extremely visible rod on the camera, even after cropping a 1080p out of the 4K video. I’ve had many comments that it’s quite distracting; I fully agree; it’s been bugging me too:

I tested Conductor 2 with script 53 while the rest of the team was cleaning the layout and it worked reliably.

Thus I enabled it as the default for the automation computer.

Basic recovery was also tested and worked in the “ideal” conditions.

Known cases where recovery will not work:

• Branchline on virtual block B830.
• PA/FR on flaky block B321 not registering the train stopped.
• Any train stopped across two blocks and activating both blocks.

The plan for B830 is to convert it to a real block sensor.

The plan for B321 is to change the sensor to another type than the BD20 with hopefully better sensitivity -- I’m especially interested in trying the new ATOM DCC Block Detector (CKT-BD1) from Iowa Scaled Engineering.

Dealing with the train stopped and activated two blocks will need more work, either on the Conductor engine side or on the script side. I need to think more about how I want to handle that one. One option is for the recovery script to be able to mark 2 consecutively occupied blocks as “occupied + trailing” since that’s really the state they are supposed to be in. Then the normal engine behavior will take care of it properly.

Now I understand the issue that was affecting the automated freight train; it's quite interesting and worth elaborating.

First, as a recap, the freight route is a shuttle composed of 2 blocks. The train starts on say block A, travels to the consecutive block B, stops, and reverses to block A. The sequence is very simple:

Step / State Block A                  State Block B                Description
1- [ A  active   occupied  ]   [ B inactive   empty    ]        Start from A
2- [ A inactive  trailing  ]   [ B  active   occupied  ]        Crosses from A to B
2- [ A inactive  trailing  ]   [ B  active   occupied  ]        Stops & Reverses
3- [ A  active   occupied  ]   [ B inactive  trailing  ]        Crosses from B to A
4- [ A  active   occupied  ]   [ B inactive   empty    ]        Stops back on A and resets

Note how although we have 5 logical steps, there are only 4 different steps from the sensor/block occupancy point of view -- there is no difference whether the engine is running on block B or stopped and waiting till it reverses.

I’m still trying to deploy Conductor 2 and it’s clearly not ready yet. I keep finding issues, some expected, some not.

The Freight automation has been an interesting source of such unexpected issues. It’s a simple reversing shuttle automation across two blocks, yet it has proven more challenging than the complex Passenger and Branchline automations -- these two actually perform quite well. Yet the “simpler” Freight one does not.

The most notable “bug” here is with the Freight automation: the engine starts on the first block, travels to the second block as expected…, and then instead of reversing the engine back to the beginning of the route, it decides the engine is back on the first block when it’s clearly still on the second block. At that point the automation panics because the engine is obviously not back at the starting point. The programming is entirely skipping half of the route automation, for no apparent reason. Why?

One of the issues which I was more or less expecting was dealing with the “flaky” sensor on B321, as I explain in detail on that post on the other blog. There’s some of that here, but there’s also something else entirely going on that I have not figured out yet.

Even after a bit of adjusting, the mountain sensors are still flaky, notably B321 and B370 with the Passenger train. I adjusted B321 for the Freight train but the newer PA engine is still borderline.

Of importance, these sensors detect current consumption. In the case of the reversing train, the sensor turns off when the train is stopped.

Here’s the new syntax:

        val T1 = throttle(1001) {

            onLight { b -> throttle.f0( b) }

            onSound { b -> throttle.f8(!b) }

            onBell  { throttle.f1(it) }

        } named "Throttle1"

        on { S1 } then {

            T1.light(true)

            T1.sound(true)

            T1.bell(true)

            T1.horn()

        }

Note that it seems quite annoying to have to place the “named ‘Xyz’” after the function block, so this syntax is also allowed: