Linux anti-idle

When I SSH into some systems (such as an adtran appliance), they have an idle timer where if the user (me) doesn't enter a command in a set amount of time, the system will close the connection. It doesn't take much to keep the system alive – simply hitting Enter periodically will keep the session alive, but doing this automatically can be a bit troublesome. Adtran has a document covering a couple methods to capture lengthy debug logs.

Adtran recommended solution (1) - Use a script with terraterm.

sleep_count = 0
changedir "C:\Logs\"
setdir "C:\Logs\"
getdate logfile "%Y-%m-%d_%H-%M-%S.txt"
logopen logfile 0 1 1
sendln "term len 0"
sendln "debug voice verbose"
sendln "debug sip stack messages"
goto keepalive
:keepalive
  ; After 12 hours, switch to new log file
  if sleep_count =144 then
    logclose
    getdate logfile "%Y-%m-%d_%H-%M-%S.txt"
    logopen logfile 0 1 1
    sleep_count = 0
  endif
  timeout = 300
  sendln "show clock"
  waitevent 1
  sleep_count = sleep_count + 1
  goto keepalive
end

Adtran recommended solution (2) - Use SecureCRT, a paid product ($99), which offers an anti-idle feature that can run a command on an interval.

Linux-y solution with tmux

Start a named tmux session:  tmux new -s Adtran

Within the tmux session, tee SSH to a logfile: ssh SOMEWHERE |tee SOMEWHERE.log
After logging into the appliance, run the necessary debug commands.

Finally, write a small script to send Enter to the tmux session periodically.

#!/bin/bash

while true; do
  tmux send-keys -t Adtran Enter
  sleep 30
done

Epic - Full Auto tmux

This builds on the previous solution by rolling everything into a fully automated script. The whole point of this exercise was to gather logs from adtran appliances, and when a specific error was found (failure to register), gather some additional logs, and then exit.

#!/bin/bash
PASS=REDACTED
hosts=(
  192.168.1.21
  192.168.1.22
  192.168.1.23
)
# The bang iterates the array index rather than the value
for i in "${!hosts[@]}"; do
  # Start a detached tmux session "AT" for "Adtran tmux"
  tmux new -ds AT$i
  # Login to the adtran
  tmux send-keys -t AT$i "ssh ${hosts[$i]} |tee ${hosts[$i]}.log" Enter
  # It would probably be better to use expect or similar, but as long as
  # the connection stays online, just tweaking the timings should be fine
  sleep 3
  tmux send-keys -t AT$i "$PASS" Enter
  sleep 1
  # Escalate to a privliged shell
  tmux send-keys -t AT$i en Enter "$PASS" Enter
  sleep 1
  # Start the debug logs
  tmux send-keys -t AT$i "debug voice verbose" Enter
  tmux send-keys -t AT$i "debug sip stack messages summary" Enter
done

echo Have initialized all sessions. Starting the loop.
while true; do
  finish=false
  for i in "${!hosts[@]}"; do
    # This serves dual purpose of looking for the error, and sending
    # periodic data to the adtran to keep the SSH session alive.
    # Spaces at the end to grab the pagified report.
    tmux send-keys -t AT$i "show sip trunk-registration" Enter "  "
    if tail -32 ${hosts[$i]}.log |head -27 |grep No; then
      echo "${hosts[$i]} has SIP error(s). Grabbing Requested info."
      # The final debug info adtran support wanted.
      tmux send-keys -t AT$i "show sip resources" Enter
      finish=true
    fi
  done
  if $finish; then
    echo Because there were errors, closing all connections.
    for i in "${!hosts[@]}"; do
      # Exit the adtran SSH connection
      tmux send-keys -t AT$i exit Enter
      sleep 1 # Need to space out the exits
      # Exit the tmux session
      tmux send-keys -t AT$i exit Enter
    done
    # Exit this script
    exit
  else
    # Need to send keepalive within 5 minutes (300s).
    # It takes some time to iterate the hosts.
    sleep 200
  fi
done