Good Old Syslog
When working on an intermittent problem that results in a device either rebooting on its own or a technician restarting usually results in loosing the log. Most system’s default system logs are stored in memory. You can usually configure devices to use some of its storage but gets to be a bit messy and complicated if you want to track a few devices.
That’s where good old syslog comes in. Syslog is a way to send a device’s log entries to a central device running a syslog collector for further analysis.
Some good syslog collectors will provide some cool features like email alerts on specific messages, the ability to run scripts and the ability to recognize non-standard syslog formats.
In this specific example we had a Cisco 2800 router intermittently reboot on its own or become non-responsive, requiring a technician to manually reboot it. When the device is unresponsive we couldn’t even get a login prompt with the console cable or SSH.
So I thought, maybe we can configure the router to send its system messages to a computer so we can see if there are any clues as to what happened before the router died or rebooted.
Now armed with this new information it looks like the router might have bad memory, no enough memory or an IOS memory leak.
Unfortunately I am currently working on this issue and will provide an update as soon as I figure it out.