The Randall Museum in San Francisco hosts a large HO-scale model model railroad. Created by the Golden Gate Model Railroad Club starting in 1961, the layout was donated to the Museum in 2015. Since then I have started automatizing trains running on the layout. I am also the model railroad maintainer. This blog describes various updates on the Randall project and I maintain a separate blog for all my electronics not directly related to Randall.
2024-07-11 - Branchline Automation Failing
Category RandallThe Branchline Automation has been failing for a week or two now. It seems to start ok, then fails mid-morning. Symptom is that the train drives back in recovery mode, whilst it’s already on the parked block. As a one-sentence recap, one of the main design points of Conductor 2 is to have fancy recovery abilities in the automation script to be able to re-align the trains when they are not at their expected locations.
Checking the logs last week, I could not figure why it failed in the first place. Let’s try again.
After monitoring the train for a little while this morning, I got this:
Thus the automation control sees 3 blocks occupied, whereas the train is visually stopped at the boundary between B820 (BL station) and B801 (BL parked). It goes without saying that only one block should be occupied here, or maybe 2 if the train is stopped at the boundary between 2 blocks.
Something interesting in this log… (Line starting with ⇒ are my annotations):
⇒ this is a normal run, we’re returning from B830 (the big canyon U curve):
10:43:49.680 S S/NS753 BLStation : ON
10:43:49.680 B S/B830v B830 : TRAILING after 74.28 seconds
10:43:49.681 B S/NS753 BLStation : OCCUPIED after 183.61 seconds
10:43:49.683 D 204 : -4
10:44:40.816 S S/NS753 BLStation : OFF
10:44:41.478 S S/NS753 BLStation : ON
10:44:55.509 S S/NS752 BLParked : ON
10:44:55.510 B S/B830v B830 : EMPTY after 65.83 seconds
10:44:55.511 B S/NS753 BLStation : TRAILING after 65.83 seconds
10:44:55.511 B S/NS752 BLParked : OCCUPIED after 320.57 seconds
10:45:05.562 D 204 : 0
10:45:13.781 R Sequence Branchline #3 Shuttle (0204) : IDLE
⇒ Engine parked normally. There’s a bit of “blinking” on the BLStation sensor but nothing we can’t handle.
⇒ Next run starts here:
10:50:21.031 R Idle Branchline #0 Ready : IDLE
10:50:21.032 R Sequence Branchline #3 Shuttle (0204) : ACTIVATED
10:50:21.032 B S/NS752 BLParked : EMPTY after 325.52 seconds
10:50:21.067 B S/NS752 BLParked : OCCUPIED after 0.04 seconds
10:50:21.067 R Sequence Branchline #3 Shuttle (0204) : ACTIVE
⇒ Engine at the expected place. BLParked is occupied, and from previous lines, all other blocks should be empty, thus the train starts its run:
10:50:30.885 D 204 : 4
…
10:50:55.272 S S/NS753 BLStation : ON
10:50:55.272 B S/NS752 BLParked : TRAILING after 34.21 seconds
10:50:55.272 B S/NS753 BLStation : OCCUPIED after 341.49 seconds
⇒ Engine moving from BLParked to BLStation…
10:50:59.263 S S/NS752 BLParked : OFF
10:50:59.263 S S/NS753 BLStation : OFF
⇒ Interesting, we have some “blinking” on the BL Station sensor again here… and then instantly everything breaks:
10:50:59.277 R Sequence Branchline #3 Shuttle (0204) : ERROR Sequence Branchline #3 Shuttle (0204) next block {B830 [B830v]} activated in 4.0 seconds. Current block {BLStation [NS753]} must remain occupied for at least 10 seconds.
10:50:59.278 R Sequence Branchline #3 Shuttle (0204) : ERROR
10:50:59.279 R Sequence Branchline #3 Shuttle (0204) : {"name":"Shuttle","th":"BL","act":2,"err":true,"nodes":[{"n":"BLParked","ms":34205},{"n":"BLStation","ms":4006}]}
10:50:59.281 B S/NS752 BLParked : EMPTY after 4.01 seconds
10:50:59.282 B S/NS753 BLStation : TRAILING after 4.01 seconds
10:50:59.282 B S/B830v B830 : OCCUPIED after 363.77 seconds
10:50:59.282 R Sequence Branchline #3 Shuttle (0204) : IDLE
10:50:59.284 R Sequence Branchline #3 Shuttle (0204) : {"name":"Shuttle","th":"BL","act":2,"err":true,"nodes":[{"n":"BLParked","ms":34205},{"n":"BLStation.1","ms":4006},{"n":"BLStation.2","ms":4010},{"n":"B830","ms":0}]}
10:50:59.286 B S/NS753 BLStation : EMPTY after 0.00 seconds
10:50:59.286 R Sequence Branchline #4 Recovery (0204) : ACTIVATED
OK so there’s some blinking on the BLStation sensor. And when moving forward, this seems to (incorrectly) trigger the virtual B830v block, e.g. we moved too fast the active block from BLStation to B830v.
So we end up in recovery, and this happens:
10:50:59.286 R Sequence Branchline #4 Recovery (0204) : ACTIVATED
10:50:59.286 B S/B830v B830 : EMPTY after 0.00 seconds
10:50:59.287 D 204 : 0
10:50:59.540 S S/B830v B830 : ON
10:50:59.541 B S/B830v B830 : OCCUPIED after 0.26 seconds
10:50:59.541 R Sequence Branchline #4 Recovery (0204) : ACTIVE
⇒ Recovery (incorrectly) deducts the train is on virtual block B830 and activates it, and tries to move it.
10:51:00.276 D 204 : -4
10:51:01.029 S S/NS753 BLStation : ON
10:51:01.229 S S/NS753 BLStation : OFF
10:51:01.891 S S/NS753 BLStation : ON
10:51:03.040 S S/NS753 BLStation : OFF
10:51:03.703 S S/NS753 BLStation : ON
10:51:05.050 S S/NS753 BLStation : OFF
10:51:05.615 S S/NS752 BLParked : ON
10:51:05.615 S S/NS753 BLStation : ON
⇒ Obviously the train is at the BLStation/Parked junction, and there’s a ton of blinking on that BLStation sensor.
10:51:05.625 R Sequence Branchline #4 Recovery (0204) : ERROR Sequence Branchline #4 Recovery (0204) has unexpected occupied blocks out of {B830}: [<BLParked [NS752]>]
10:51:05.626 R Sequence Branchline #4 Recovery (0204) : ERROR
10:51:05.628 R Sequence Branchline #4 Recovery (0204) : {"name":"Recovery","th":"BL","act":1,"err":true,"nodes":[{"n":"B830","ms":6085}]}
10:51:05.630 D 204 : 0
10:51:05.882 R Sequence Branchline #4 Recovery (0204) : IDLE
10:51:05.883 R Sequence Branchline #4 Recovery (0204) : {"name":"Recovery","th":"BL","act":1,"err":true,"nodes":[{"n":"B830.1","ms":6085},{"n":"B830.2","ms":6341}]}
⇒ We end up the recovery sequence thinking the train was only on B830.
I suspected the lack of recovery had to do with the virtual block -- if the train is found nowhere, recovery “deduces” it is in the virtual block and reverses it. This is a fairly logical assumption with ideal sensors, but it is wrong if we start with flaky sensors. Then if the train is somewhere else and wasn’t detected when stopped, it may be detected once it starts moving and now we have 2 active blocks, which is a definite recovery error.
There’s an attempt at a “2nd stage recovery” here in script 55 v11, line 1966:
The main recovery handles the “normal” cases of all the real blocks detecting the presence of a train on a single block (or a boundary). If that fails, we switch to a secondary recovery. That one (ab)uses an “idle” route rather than a “sequence” route: we run the train backwards till B801 (BL Parked) is active, with B820 (BL station) inactive. There’s also a 5-minutes timeout for that operation. That will not fix the root cause, but it should fix the recovery.
There are 2 root causes to fix:
- That B810 / BLStation sensor is now flaky. It used to be fine, but not anymore 😕
- The cause of the sensor working can be anything from the track leakage current having changed due to a scenery update, or due to a bad block feeder solder. It’s rarely the sensor itself which is at fault.
- It’s an NCE BD20 sensor. Either I need to solder an adjustment resistor to it, or upgrade it to one of the Iowa Scaled CKT-BD1 sensors as I have done on the Mountain Block Panel.
- I want to get rid of the virtual block.
- The proper way to do that is to finish wiring a real block sensor in that spot -- that was the eventual goal anyway, I just need to find more time crawling under the layout finding where the block boundaries are and where the power feeds are for this or these blocks.
The Iowa Scaled CKT-BD1 sensors are more fancy than the “legacy” NCE BD20 sensors. To adjust the sensitivity of the NCE ones, I need to solder a small variable resistor and then manually turn that “till it’s good” (easier said than done). The Iowa Scaled are more modern and have a little ATTiny85 processor on them. They are adjusted merely by pressing a button and waiting a few seconds. I like the NCE BD20 sensors because they are more compact, very simple in design; I still have a handful of these around. The Iowa Scaled CKT-BD1 sensors did not exist when I started automating the layout so I only currently use them for the track blocks which have proven to be very unreliable.