Back in December I was fighting with the spurious block activation of block B360. This would break the automation, and I would have no real report about it. Of course I already have a custom web dashboard that shows me the status of the automation and after-the-fact analytics -- what kind of automation would that be without such dashboards? Still I figured having a “realtime” notification on failure would be better than periodically checking my custom dashboard.

I could add some email-sending capabilities to Conductor 2, but that’s something I’d rather not do as it’s a bit of its own rabbit hole. The Debian Linux computer running Conductor 2 has no email capabilities, and I intend to keep it that way -- I treat it as what it is, namely an unsecured box in a public museum with fairly loose access, thus it’s basically a DMZ.

Instead what I want is for the emails to be generated by some server that I control. All I need is to get the data to that server. I could piggyback the errors on the JSON used by the RTAC status server. That would almost be too logical. Or I could publish it on the MQTT broker and then proxy it over to the email server. Well, if I’m going that route, what about RSyslog?

So that’s what I did, and I get this kind of beautiful emails:

From: pi@alfray.com To: self@alfray.com Subject: New Automation Error Date: Wed, 08 Jan 2025 11:00:11 -0800   Jan  8 11:00:01 consist fireman:  CONSIST ERROR:  10:57:38.342 R Sequence Branchline #3 Shuttle (0204) : ERROR Sequence Branchline #3 Shuttle (0204) current block <BLYouBet [NS760]> still occupied after 120.5 seconds.  10:59:38.892 R Sequence Branchline #4 Recovery (0204) : ERROR Sequence Branchline #4 Recovery (0204) current block <BLYouBet [NS760]> still occupied after 120.0 seconds.

Here’s how I implemented this:

On the automation computer, I have a cron job parsing the Conductor 2 logs every 5 minutes. These already have exactly the information I want to see, as shown above -- they look wonderfully cryptic to the uninitiated yet relay all the information I need. Usually I’d notice something wrong on the dashboard, and then check the logs over SSH. Thus having exactly that log delivered by email is ideal.

Obviously, the best way to collect the error log is a quick little Bash script:

#!/bin/bash LOGGER=/usr/bin/logger # Get the last log YMD=$(date +%Y-%m-%d) L=$( ls -1 --sort=time conductor-log-$YMD*.txt 2>/dev/null | head -n 1 ) # Find all errors grep -h ": ERROR " "$L"  > "current.txt" # Compare with the last "all errors" LOG=$( diff --new-line-format="@ %l @" --new-file --ignore-blank-lines --text "last.txt" "current.txt" | grep @ ) if [[ -z "$LOG" ]]; then exit 0; fi # No new error(s) today $LOGGER "@@ CONSIST ERROR: @${LOG}@" # Update "last" errors on success cp "current.txt" "last.txt"

This finds the most recent log file, filters using the error pattern, and compares it with the last saved error pattern file. If they differ, send the diff via the system logger. (Editor’s note: in my real bash script I use variables for every file name; I’ve just expanded and simplified them above for clarity.) Note the very specific usage of the “@” character in the diff output and the logger call -- I use that to trim the new-lines out of the log output and then will convert the “@@” pairs back to “\n” on the receiver side.

Then I just call that in a crontab every 5 minutes.

I already have the local rsyslog configured to send some of the logs I care to one of my servers -- modern Debian Bookworm doesn’t have rsyslog installed by default IIRC, so we need to install it first and configure it:

$ sudo apt install rsyslog $ view /etc/rsyslog.conf … *.info;mail.none;authpriv.none  @@dmz.alfray.com:1234

On the server side, all I need is to parse the logs collected by the receiving rsyslog. Thus we need the Debian RSyslog package again, and this time configured to store stuff in some specific log file:

$ sudo apt install rsyslog $ view /etc/rsyslog.conf … $template RemoteHost,"/tmp/logs/remote_%HOSTNAME%.log" if $hostname == 'consist' then -?RemoteHost & stop

You’ve got to admire that superb rsyslog syntax… I’ve written my fair share of custom config files and custom associated parsers over the years, so really I don’t mind (*) -- what matters is that it’s very flexible and documentation for it is well written. (* except that I’ve come to realize over the years, like everybody else, that there’s some value in sticking everything in JSON or YAML so I don’t really code custom parsers anymore… sort of.)

Now I’m ready to have this send me an email when the last error has changed -- we can parse that server-side log and generate an email by calling sendmail.

I thought about writing a systemd service daemon at first. Should it be in Rust, NodeJS, Java, Kotlin, or Go? Let’s not even mention Perl. Nah, Bash and a simple crontab job will get us there as usual:

#!/bin/bash sleep 10s # Check if log is new LOG=$(grep "@@ CONSIST ERROR:" "remote_rsys.log" | tail -n 1) if [[ -z "$LOG" ]]; then exit 0; fi PREV=$(cat "diff.txt" 2>/dev/null || echo "") if [[ "$LOG" == "$PREV" ]]; then exit 0; fi  # no new stuff   echo "$LOG" > "diff.txt" # re-create the newlines from @@ EXPANDED=$(sed 's/@@/\n/g' "diff.txt")   # Main new log cat > "msg.txt" <<EOF From: $FROM_ADDRESS To: $TO_ADDRESS Subject: New Automation Error   $EXPANDED   EOF /usr/sbin/sendmail -f "$FROM_ADDRESS" "$TO_ADDRESS" < "msg.txt"

I also stick that script invoked from a crontab every 5 minutes with the exception that the receiver side is offset to run 10 seconds after the sending side. That way I have at most a 5-minute delay before being notified of an automation error.

---