The Passenger Train stopped working on the Mainline automation yesterday. The train was nowhere to be found on the remote cameras, yet JMRI detected the train’s presence on block B360 -- that’s the curve just before reaching Summit on the mountain.

Since I wasn’t in a position to go to the museum to investigate, I remotely updated the automation script over SSH to disable the Passenger train and only run the Freight train on the automation.

When I got there today, I found this:

That’s a new one as I had never seen a derailment at that spot before.

From there, every effort at fixing the situation pretty much went wrong one way or another.

First, I grabbed the train and brought it back to the Stockton Yard to set it up again for automation. Chatting with Orion, he suggested maybe cleaning the wheels would help. They didn’t seem otherwise dirty but why not. And actually I’m glad I did because I found there was something odd going on with the engine -- it vibrated when set in reverse. That’s not supposed to happen.

I decided to open the engine to look at the gears inside. These are a bit confusing to open and I was trying to remember how to do that when I remembered I wrote myself a specific page on doing just that: Shell Removal: HO Walthers Mainline SD70ACe.

The front truck does indeed vibrate but only when going in reverse direction. I suspect a stripped gear or something similar, which won’t be obvious to fix unfortunately. That seems like one more unwanted issue on these Walthers engines with which I had just issue after issue. I wouldn't expect Whalters to have spare parts although I'll ask.

This UP 8330 engine is thus out of service for now. I thus chose to use the twin engine UP 8312 for the automation. There was a reason why I pulled it off automation a few months ago though. I didn’t like the custom sound I programmed on the LokSound for UP 8312, and I preferred the one for 8330. I realize now I forgot to make that change, but that will be for another day -- it’s more than just a “trivial” sound change, as the program has very different acceleration characteristics for the motor, which requires me to entirely reprogram the automation script accordingly. Thus I’ll do that later when it’s not urgent -- that kind of work typically takes at least 4+ hours to get done right.

So fine, I reinstall UP 8312 in automation using the same engine profile that I programmed last time. It should work, and indeed at first it seems to work -- till the engine reaches Summit, reverse as expected, and… the automation fails while the engine is going down the mountain.

I check the automation logs and things look fine yet there was an error in one of the track sensors but nothing seems off. That’s wrong -- either there’s an error or there isn’t. After trying the automation 3 times, it works one time and fails twice, and I finally notice the pattern, this is subtle yet wrong:

15:54:05.335 D 8312 : Bell OFF 15:54:05.335 D 8312 : F1 OFF 15:54:05.335 D 8312 : Horn 15:54:06.036 D 8312 : -22 15:54:32.055 S S/NS775 B360 : ON 15:54:32.055 B S/NS775 B360 : EMPTY after 77.79 seconds 15:54:32.056 B S/NS776 B370 : TRAILING after 77.79 seconds 15:54:32.056 B S/NS775 B360 : OCCUPIED after 0.00 seconds 15:54:38.393 S S/NS776 B370 : OFF 15:54:44.104 D 8312 : -24 15:55:07.755 S S/NS774 B340 : ON 15:55:07.756 B S/NS776 B370 : EMPTY after 35.70 seconds 15:55:07.756 B S/NS775 B360 : TRAILING after 35.70 seconds 15:55:07.757 B S/NS774 B340 : OCCUPIED after 113.49 seconds 15:55:51.138 S S/NS773 B330 : ON 15:55:51.139 B S/NS775 B360 : EMPTY after 43.38 seconds 15:55:51.139 B S/NS774 B340 : TRAILING after 43.38 seconds 15:55:51.140 B S/NS773 B330 : OCCUPIED after 190.83 seconds 15:55:51.172 R Sequence Mainline #3 Passenger (8312) : ERROR Sequence Mainline #3 Passenger (8312) has unexpected occupied blocks out of {B330}: [<B360 [NS775]>] 15:55:51.172 R Sequence Mainline #3 Passenger (8312) : ERROR [...] 15:55:52.390 S S/NS775 B360 : OFF 15:55:52.391 D 8312 : F5 OFF

I’m kidding, I don’t expect you to understand that log. What’s interesting in this log is the line that is not there. So how about this view from Conductor 2, the software that controls the automation and monitors the track occupancy:

Here’s how to read the log: the train stops at Summit, which is block B370. That’s occupied, as expected. Then it reverses to go back down the mountain. First it enters block B360, whose sensor promptly turns on. After that, the train moves to block B340 (there’s no block B350, don’t ask). At that point, the block B360 should turn off as soon as the train is fully in block B340 and that’s precisely what does not happen -- there should be a line “B360: OFF” at timestamp 15:54:08” and it’s blatantly missing. Instead the block B360 stays on for a “long time” -- namely a whopping 1 minute and 20 seconds after the train has left the block!

The bottom line is that there’s something odd going on here. There is no “debounce” in JMRI that would force that sensor to stay on, and that seems like a fairly new behavior. The sensor itself is a regular NCE BD20 as shown in this post when I installed the CKT-BD1 (B370 has one, but B360 uses the legacy BD20). I also note that JMRI thinks that block B380 is occupied, when there’s nothing on that block either. That leads me to suspect something wrong in the Mountain Panel where the sensors are installed rather than a software problem.

Or maybe it’s a different electrical problem somewhere else. The reason I say that is because block B360 is one of the two blocks that magically register as occupied when the Richmond Yard is being used, even though they are supposed to be electrically isolated (and clearly, they are not). In this case, Richmond Yard was not in use, yet that’s an eerily similar situation.

To be continued.

Update: Next morning, another try… This time the “phantom block B360 stays on for one minute after the train leaves the block” does not occur at all. That may be an issue with sensor sensitivity in this case. The passive current-sensing sensors like the BD20 can sometimes get too sensitive and need to be adjusted.

---