In the old Conductor 1, anytime the train sequence did not work as expected, the engine simply stopped everything, and we had to manually replace the trains as expected and restart everything.
Thus, when I redesigned the new Conductor 2, I added some “recovery” functionality -- the idea is that if the train sequence does not happen as expected, the automation engine should do something to try to recover the trains.

Unfortunately, the idea of “recovery” is basically to more or less predict the unpredictable and decide what the automation program should do in that case. Which means there are always cases that one never thought about because that “could never happen” and thus such cases are not handled. Today is one such case.

Orion and I have been slowly working on fixing the Branchline T324 turnout. That involves crawling through one of the mountain access holes, and then we need to reach the Branchline over the Mainline track. To make it more fun, that’s one of the tracks used by the Passenger train automation.

Twice I’ve seen the Passenger train go into error mode while we were trying to work on the Branchline T324 turnout. What was even more puzzling is that the Passenger was at Summit and then the Freight train just… started! Suddenly, on the same track, I had the Passenger train go into recovery and thus reversing towards the station whilst at the same time I had the Freight train leaving the station… Each going towards each other in the opposite direction. Not exactly a good scenario.

Hilarity ensues.

The first part of the puzzle was easy to find: when we crawl into the mountain access hole, we somehow activate the block B340. It’s not exactly clear why and how that happens, yet that sure is the result.

Here’s an excerpt from conductor-log-2024-10-12-08-40-03.txt:

16:42:22.404 R Sequence Mainline #3 Passenger (8330) : ACTIVATED

16:42:22.404 B S/NS787 B503b : EMPTY after 374.60 seconds

…

16:45:26.205 B S/NS775 B360 : TRAILING after 24.67 seconds

16:45:26.205 B S/NS776 B370 : OCCUPIED after 714.77 seconds

16:45:26.236 T @timer@4 : start:4

16:45:29.142 S S/NS797 AIU-Motion : OFF

16:45:30.255 T @timer@4 : activated

16:45:30.256 T @timer@17 : start:17

16:45:30.256 D 8330 : 10

16:45:30.506 D 8330 : Bell ON

16:45:30.507 D 8330 : F1 ON

16:45:32.764 S S/NS797 AIU-Motion : ON

16:45:39.914 S S/NS774 B340 : ON
⇒ Block B340 suddenly got activated when trying to work on the Branchline. Train 8330 is on B370 (Summit).
16:45:39.915 R Sequence Mainline #3 Passenger (8330) : ERROR Sequence Mainline #3 Passenger (8330) has unexpected occupied blocks out of <B370>: [<B340 [NS774]>]

16:45:39.916 R Sequence Mainline #3 Passenger (8330) : ERROR
⇒ Automation goes into error mode because an unexpected block suddenly gets occupied.
16:45:39.920 R Sequence Mainline #3 Passenger (8330) : {"name":"Passenger","th":"PA","act":2,"err":true,"nodes":[{"n":"B503b","ms":17873},{"n":"B503a","ms":10133},{"n":"B504","ms":19340},{"n":"B321","ms":55122},{"n":"B330","ms":22999},{"n":"B340","ms":33531},{"n":"B360","ms":24666},{"n":"B370","ms":13711}]}

16:45:39.923 D 8330 : 0

16:45:40.424 D 8330 : Sound OFF

16:45:40.424 D 8330 : F8 OFF

…

16:45:40.630 R Sequence Mainline #3 Passenger (8330) : IDLE

16:45:40.635 R Sequence Mainline #3 Passenger (8330) : {"name":"Passenger","th":"PA","act":2,"err":true,"nodes":[{"n":"B503b","ms":17873},{"n":"B503a","ms":10133},{"n":"B504","ms":19340},{"n":"B321","ms":55122},{"n":"B330","ms":22999},{"n":"B340","ms":33531},{"n":"B360","ms":24666},{"n":"B370.1","ms":13711},{"n":"B370.2","ms":14425}]}

16:45:40.637 B S/NS775 B360 : EMPTY after 14.43 seconds
⇒ There’s motion in the room and now the Freight train starts, since the Passenger one is no longer active.
16:45:40.638 R Sequence Mainline #4 Freight (1067) : ACTIVATED

16:45:40.638 B S/NS769 B311 : EMPTY after 311.24 seconds

16:45:41.140 D 8330 : F5 OFF

16:45:41.141 B S/NS769 B311 : OCCUPIED after 0.50 seconds

16:45:41.141 R Sequence Mainline #4 Freight (1067) : ACTIVE

16:45:41.175 D 1067 : Light ON

16:45:41.276 D 1067 : Sound ON

…
⇒ A second later, a delayed timer activates which returns the Passenger train towards the station.
16:46:17.295 T @timer@30 : activated

16:46:17.295 D 8330 : Bell OFF

16:46:17.295 D 8330 : F1 OFF

16:46:17.296 D 8330 : Horn

16:46:17.996 D 8330 : -10   ← This is the Passenger train

This “@timer@30” is internal name for “Delay_B370_Pause_Delay” -- the normal behavior at Summit is to pause then reverse after 30 seconds at speed 10:

onEnter {

   after (AM_Data.Delay_B370_Entrance) then {

       // Forward

       PA.forward(AM_Data.Summit_Speed)

       PA.bell(On)

   } and_after (AM_Data.Delay_B370_Forward_Stop) then {

       PA.stop()

       PA.horn()

   } and_after (AM_Data.Delay_B370_Pause_Delay) then {

       // Stopped

       PA.bell(Off)

       PA.horn()

       PA.reverse(AM_Data.Summit_Speed)

   }

}

So that’s one bug -- that means somehow this route is still executing even though the script has changed route. Instead, the timer should have been canceled by the route change, leaving the Passenger train stranded at the top of Summit.

As for the error/recovery, it’s perfectly reproducible in the simulator:

Script: [ML Recovery] PA start=false occup=2 total=2 [{B503b [NS787]}, {B503a [NS786]}, {B504 [NS784]}, {B321 [NS771]}, {B330 [NS773]}, <B340 [NS774]>, {B360 [NS775]}, <B370 [NS776]>]

Script: [ML Recovery] FR start=true occup=0 total=1 [<B311 [NS769]>, {B321 [NS771]}]

Script: [ML Recovery] Ignore Passenger, Activate Freight

The core issue here is that’s a bug: if a route is abandoned because it goes into error mode, any delayed timer blocks should be abandoned and should definitely not execute.

The way the recovery should have worked is as follows:

• The Passenger route is active and the train running.
• An unexpected block is activated  ⇒ This stops the train. The route is marked as being in error mode (This part works as expected).
• The route selection should ideally respect the 1-minute pause I have when switching trains (This part is missing.)
• The route selection should determine that the Passenger train is stuck at Summit, and consequently bring it back to the station. Once that is done, it can schedule the Freight train to leave.
• Or the route selection should determine that the Passenger train is stuck at Summit, determine that there is at least one spurious block occupied on the track (B340), and thus decide it is not safe to bring back the Passenger train to the station. Better leave it where it is. Thus it’s fine to schedule the Freight train to leave.

That last action is what’s happening, and it’s happening “as expected”. The only issue is that it happens instantly: block B340 gets activated, that stops the Passenger train, and instantly the route tries to “recover” and switches back to the Freight, because it that small millisecond the spurious block is still active. If there were a small pause in between, the block would not show activate anymore, and the Freight would not start instantly.

In essence, this is similar to the issue from the previous post: if there’s spurious block activation when trying to start, the program should delay and try again a few minutes later.

---