When I SSH into some systems (such as an adtran appliance), they have an idle timer where if the user (me) doesn't enter a command in a set amount of time, the system will close the connection. It doesn't take much to keep the system alive – simply hitting Enter
periodically will keep the session alive, but doing this automatically can be a bit troublesome. Adtran has a document covering a couple methods to capture lengthy debug logs.
Adtran recommended solution (1) - Use a script with terraterm.
sleep_count = 0
changedir "C:\Logs\"
setdir "C:\Logs\"
getdate logfile "%Y-%m-%d_%H-%M-%S.txt"
logopen logfile 0 1 1
sendln "term len 0"
sendln "debug voice verbose"
sendln "debug sip stack messages"
goto keepalive
:keepalive
; After 12 hours, switch to new log file
if sleep_count =144 then
logclose
getdate logfile "%Y-%m-%d_%H-%M-%S.txt"
logopen logfile 0 1 1
sleep_count = 0
endif
timeout = 300
sendln "show clock"
waitevent 1
sleep_count = sleep_count + 1
goto keepalive
end
Adtran recommended solution (2) - Use SecureCRT, a paid product ($99), which offers an anti-idle feature that can run a command on an interval.
Linux-y solution with tmux
Start a named tmux session: tmux new -s Adtran
Within the tmux session, tee SSH to a logfile: ssh SOMEWHERE |tee SOMEWHERE.log
After logging into the appliance, run the necessary debug commands.
Finally, write a small script to send Enter
to the tmux session periodically.
#!/bin/bash
while true; do
tmux send-keys -t Adtran Enter
sleep 30
done
Epic - Full Auto tmux
This builds on the previous solution by rolling everything into a fully automated script. The whole point of this exercise was to gather logs from adtran appliances, and when a specific error was found (failure to register), gather some additional logs, and then exit.
#!/bin/bash
PASS=REDACTED
hosts=(
192.168.1.21
192.168.1.22
192.168.1.23
)
# The bang iterates the array index rather than the value
for i in "${!hosts[@]}"; do
# Start a detached tmux session "AT" for "Adtran tmux"
tmux new -ds AT$i
# Login to the adtran
tmux send-keys -t AT$i "ssh ${hosts[$i]} |tee ${hosts[$i]}.log" Enter
# It would probably be better to use expect or similar, but as long as
# the connection stays online, just tweaking the timings should be fine
sleep 3
tmux send-keys -t AT$i "$PASS" Enter
sleep 1
# Escalate to a privliged shell
tmux send-keys -t AT$i en Enter "$PASS" Enter
sleep 1
# Start the debug logs
tmux send-keys -t AT$i "debug voice verbose" Enter
tmux send-keys -t AT$i "debug sip stack messages summary" Enter
done
echo Have initialized all sessions. Starting the loop.
while true; do
finish=false
for i in "${!hosts[@]}"; do
# This serves dual purpose of looking for the error, and sending
# periodic data to the adtran to keep the SSH session alive.
# Spaces at the end to grab the pagified report.
tmux send-keys -t AT$i "show sip trunk-registration" Enter " "
if tail -32 ${hosts[$i]}.log |head -27 |grep No; then
echo "${hosts[$i]} has SIP error(s). Grabbing Requested info."
# The final debug info adtran support wanted.
tmux send-keys -t AT$i "show sip resources" Enter
finish=true
fi
done
if $finish; then
echo Because there were errors, closing all connections.
for i in "${!hosts[@]}"; do
# Exit the adtran SSH connection
tmux send-keys -t AT$i exit Enter
sleep 1 # Need to space out the exits
# Exit the tmux session
tmux send-keys -t AT$i exit Enter
done
# Exit this script
exit
else
# Need to send keepalive within 5 minutes (300s).
# It takes some time to iterate the hosts.
sleep 200
fi
done