Four days ago, a derailment of the Amtrak train occurred. This was left unattended for a while. The end result is that both trucks on one of the Amfleet coaches melted. I didn’t even have a remote idea this was possible. I did not get a chance to see the damage first hand nor how it occured. One possibility I can consider is that there was a short and it lasted long enough to somehow produce heat and melt the trucks.

Photos: J. Evans.

That left me speechless. Not for long though.

In the 2018-03-28 post “Error Conditions”, I explained the idea of detecting errors and stopping the automation/layout. It’s something I considered last year before the museum opened and the idea seemed way overkill back then.

So here’s a thought, or more exactly a few of them:

• Does sending an E-STOP command to the NCE Command Station actually kills power to the track, or does it just stop sending DCC packets?
• In the latter case, I should consider having a USB-controlled power strip feeding the boosters and the CS, or have relays on the output of the boosters.
• Monitor the NCE circuit breakers using the automation computer. When a short is detected for a specific duration on the automated route, kill off power via the relays.

Let’s remember that DCC circuit breakers are not designed to protect the equipment on the track. They are designed to protect the command station and the boosters from shorts. When they trip, they automatically reset a few seconds later and try to reapply power. This creates the characteristic repeated high-pitch “bzzz … bzzz … bzzz” sound one can hear when there’s a short e.g. in a yard.

Randall has four boosters and ten power districts each with their own circuit breaker:

• Booster 1 (Command Station): Valley 1 + Stockton Yard.
• Booster 2: Valley 2 + Napa + Town / Lodi
• Booster 3: Mountain 1 + Branchline / Richmond 2
• Booster 4: Mountain 2 + Bridgeport + SIA / Roundhouse.

For the automation, we currently only use the districts of Stockton Yard (Booster 1) and Mountain 1 (Booster 3). Later I might want to have more automation involving Valley 2 (Booster 2). Eventually if we’re to do JED one day, the full main line loop would involve Valleys 1+2, Mountains 1+2, e.g. all four boosters.

As far as cutting power, one option is a single USB-controller power strip that would cut power to the Command Station and all the boosters. The other option is individual relays, either before or after the circuit breakers. Before would require 4 relays, and after would require 10 of them at most (or at least as many as circuits involved in the automation).

The benefit of using more relays is more granularity and the potential for the automation to limit where it cuts the power to not disturb non-automated routes. What we really need are Valleys 1+2, Mountains 1+2, and Stockton Yard. That’s 5 districts.

So how could this all work? Just thinking aloud here:

• An 8-relay board. I’ve used a Sain-Smart one before with actual physical relays rated at 10 A / 250 V each. The relays are on by default on this board, which is perfect for our use case.
• An Arduino e.g. Nano might be enough. Needs 8 outputs to trigger the relays and 8 inputs to monitor the circuit breakers.
• USB connection from the automation computer to the Arduino to read the circuit breakers states and trigger relays as needed. No need for wifi in this case.
• Modification to RTAC / Conductor: define relay monitoring entries (simply a 1..8 index for the Arduino) and a time limit. If a circuit trips for longer than the time limit, trigger the relay to cut off power. Doing so will of course release the circuit breaker, so a rest period is needed with an exponential backoff (e.g. keep it off for 15 seconds, 30 seconds, 1 minutes, 2 minutes, etc.)

The other modification needed to RTAC / Conductor is the ability to have “expectations”, as explained before. This was going to be part of a larger rewrite where I would change the automation from a event state machine to a sequential state machine. However we can trivially retrofit the idea here:

• Overall the Amtrak automation takes 5 minutes to perform its back-and-forth route.
• For any given state, add an instruction in the script which indicates the expected next state. All that matters here is a block number. E.g. B320 leads to B330, which leads to B340, which leads to B360, which leads to B370, which leads to B360 etc.
• A global time limit would work probably just fine, e.g. 1 minute to reach the next expected block. It should be possible in the script to override that per block but overall that global time limit might just work enough for our small needs here.
• When an “expect next block” instruction is executed, it triggers a timer. If the destination block triggers, it resets the timer. If the timer expires and the block has not been activated, that means the train never reached the destination. Thus there’s a fault.
• On fault, send either an E-STOP or a per-train STOP command.
• On fault, also display a prominent warning on the RTAC tablets to inform staff of the condition of the automation: clearly indicate what stopped and why. We may want further notification systems such as sending an email or similar (understanding that this depends on the state of the wifi, which is fairly precarious in that concrete-laden basement).
• If the fault is not corrected within a specific time by a reset on the RTAC tablets, the power to the layout can be cut off using the relays in line with the circuit breakers.

The choice of an E-STOP versus a per-train STOP command needs to be examined. One variable to look into is whether issuing an E-STOP command via the NCE serial port requires a full NCE power cycle like it does when using the red emergency stop button on the NCE controllers. One would assume they would do the same thing, yet that remains to be validated.

One reasoning for a full layout-wide E-STOP is the case where other non-automated trains are being operated. On one hand, it would be best to stop them in case they happen to be on the same route as e.g. a derailed automation train. On the other hand, one would assume the operator would be paying attention (which in the past has been an illusory false predicate), yet at the same time it may stop trains totally unrelated to the automation (e.g., in a yard). However that’s a minor point and it would be fairly easy to make that a choice in the automation script.

Last but not least, we need to consider the human factor here. The layout is automated, which means that 4 days out of 5 it runs without any kind of supervision. The one person in charge is the museum’s receptionist, who has, as one can guess, many other responsibilities and is by definition not in the train layout room. The main receptionist is a very nice lady who’s very good at all the technical stuff I can give her and has repeatedly proven being able to make good decision calls. Being there often, she’s getting familiar with the way the trains work. The same cannot be said of the various receptionist volunteers that come and go on a weekly basis. We cannot expect them to be as proficient nor can we realistically increase their cognitive load by burdening them with what are essentially a bunch of technical idiosyncrasies. Instead we need to simplify their work, and that is to stop the automation when there’s a problem and teach them that it’s ok to keep it off rather than to try to fix a problem they don’t know how to fix. The problem is that they get the reverse message from their management or visitors who want to see the thing running and not turned off and are prone to encourage taking shortcuts to make it work again, without regards to consequences that they cannot foresee or understand. In a nutshell, we clearly asked them to shut everything off when there’s a problem, yet they are reluctant to do so. Having the automation automatically turn power off would ironically be a better solution.

In conclusion:

• The RTAC / Conductor change would have helped us get notified that a train had derailed. It would not have helped with the short or whatever condition that melted that coach truck.
• A relay combined with circuit breaker monitoring should have helped with a short to prevent damage to the equipment on the track.
• It is expected we might have to empirically fine tune various timing factors to make sure the protection is not too aggressive and does not impair normal operations.

---