Logging incident handler activity on the console
During incident handling, time is precious. I try to make myself take notes and communicate about the incident over logged channels like e-mail and IM - to the point that I think some team members have rules specifically for e-mail from me.
The detailed timelines that can be reconstructed from these notes are crucial when preparing post-mortem documents!
I have a simple philosophy: Fill what’s empty. Empty what’s full. Scratch where it itches Script the process wherever possible.
— Alice Roosevelt Longworth (and Grant Stavely)
Initially, I thought bash history could help.
# Bash settings
export HISTTIMEFORMAT="%s "
export HISTCONTROL=ignoredups
export HISTFILESIZE="9999999"
But what about the command output? There is a better way:
Firstly, and this is something that took getting used to, set up a fancy shell prompt to provide the log with context and time stamping:
--(grant@sensor)-(1/pts/1)-(17:32:33-UTC/28-Apr-08)--
--($:/nsm/)-
Gross! A two-line shell prompt! Trust me, it’s useful.
Then just add to your shell start-up:
# start script to log everything now!
exec /usr/bin/script -f /nsm/var/handlerlog/$USER.shell_log.` date +'%Y-%m-%d:%H:%M:%S' `.$$
Script will then log everything that prints to the console with the prompt providing context.