SA Tenets – First, Do No Harm
This is a continuation of the “SA Tenets” series of posts in which I discuss my beliefs about System Administration and work in general
First, Do No Harm
Yes, stolen from the popular misquote of the Hippocratic Oath. It’s important for SAs to remember, too.
In the heat of fixing the latest production issue, it is important to keep in mind that it is possible to make things even worse. So take a moment to decide the best action, consider what you may be forgetting under the pressure, and slowly and methodically fix the problem.
I once was reprimanded for not having a backup of our most important server. It ran NIS, DNS, pretty much everything for our local LAN. So I mounted a second drive and started a “dd” from the blank disk, over top of the only copy of the very important data… I wasn’t fired, but I did learn an important lesson.
Reader Comments
and this goes along with “document everything”.. because when stuff is broken and you’re in a panic to fix it, being able to go over previously confirmed/established steps will help to make sure you don’t make a whoopsies :D
then again, not everything can be documented, like in your case of dd-itis, so yeah, hehe
Good kickoff.
Two things come to mind when it comes to do no harm:
1) Document the fix! This is, IMO, more important that the original documentation that should accompany the environment in the steady state. A fix @ 3am that isn’t written down, is very quickly forgotten, and may make the existing documentation wrong. Of course, if it’s a simple host reboot, this may not apply.
2) Harm has been done. Own up to it! We all make mistakes (being human..ain’t it great?). If the pooch is horked and you did it, own up to it, tell folks what happened, that you did it and are working on a fix, or explain what the fix is that has just been applied. If you are afraid to make a mistake, you are in the wrong profession.
Making the same mistake over and over again is a different problem for a different post.