What Makes Me Nervous

It’s not layoff rumors, or a superior that is too busy for me. It’s not the budget or the outages. It’s not the latest bit of busy work or the next rash of hackers.

That’s not what makes me really nervous.

What makes me really nervous is when the entire team goes long stretches – months – without making a single mistake.

Don’t Be “Hook-ish”

Years ago I had a boss who used to chide me for being “hook-ish”. When I finally asked him what he meant, he picked up a wooden pointer (for chalk boards and such – I said it was a long time ago) and described the parts. A meter-long wooden shaft about a centimeter thick. On one end was a foam point. On the other was a metal screw-in eye-hook. He explained that the point end is the end that got the hard work done, and the hook was used to store the pointer. Picking up and using the pointer was “dangerous” and hanging it up was “safe”.

He told me that I was being too safe – that most of the time I would pick the safe thing to do when given a choice. Which is not terrible, but also not what he wanted me to do. I explained that I was afraid of failing so bad that I’d be fired. He explained that I was not going to get fired for failing or for being hook-ish, but that I being too conservative was worse than being fired.

Fail Spectacularly

I would rather see an employee fail spectacularly – auger-in in a ball of flames – because she is attempting something new, difficult, and dangerous, than succeeding at doing the status quo.

New, difficult and dangerous tasks tend to be the ones with the largest payoff. I’m not saying you should start editing a running binary because that’s risky. I’m saying don’t shy away from the hard stuff just because you may fail.

Winning BIG means risking BIG – and I want my employees to win BIG. Sometimes that means BIG mistakes, and I’m OK with that.

Eat Your Own Dog Food

I have a coworker who hates that phrase – he thinks it’s too self-deprecating. He prefers “Eat Your Own Cooking”, but the sentiment is the same.

Take Pride in Your Work

Taking pride in your product extends past the production stage. You need really want to make great things. If you believe in what you are making, you will naturally want to be a consumer.

Be a Super Consumer

I’m not saying you should only use your own product – in fact just the opposite. You should use your competitor’s products too. Then you will know how each stacks up, and your own experience will teach you where your product needs to be improved.

No one trusts a skinny cook, so be sure to be a consumer of your own product!

Oncall Applies In Both Directions

Most products do not fail at convenent times. So most SAs will find themselves carrying a pager for an oncall shift. I have one major rule that says “oncall applies in both directions” and no one ever understands what I mean (so it probably needs a new name). Here is what I mean.

Oncall Is an Intrusion

Being paged to fix a problem at work is an intrusion on you private life. And in most cases it is considered “part of the job” and doesn’t garner any additional compensation. But it really is worth more than your normal work – it’s an emergency. And an interruption. And involves more work (finding a computer, finding a network, establishing a VPN, etc). Companies generally do not acknowledge this fact.

Intrusions Happen at Work, Too

If your work is allowed to intrude on your life, then your life is allowed to intrude on your work. Plumbers must be let in, packages received, recitals attended, and almost always during business hours. And if your job caused you to miss four hours of sleep last night, then I don’t really want you touching production machines – better to sleep in and come to work with a sharp mind.
It’s basic work/life balance – remind your boss (and HR) if (s)he balks.

Automate The Hell Out Of Everything

There is an old joke that goes like this:

• The junior SA can’t help you because she is too busy because nothing is automated.
• The journeyman SA can’t help you because she is too busy automating everything.
• The senior SA can’t help you because she is too busy playing a First Person Shooter.

I’m sure most senior SAs are watching graphs, writing docs, and learning a new skill, but that is not as funny.

Spend Time Now to Save Time Later

Any task that you have to repeat more than three times should be automated. And right now, before you have to do it again. You will quickly go insane repeating mundane tasks by hand, and you will NEVER be able to handle more than a handful of machines.

I think this is really important – and I’m not alone. Karl thinks so. So does Doug. Automating your tasks now is going to save you time later and shorten the time spent fixing issues in the middle of the night. Doug points out that 1000:1 host-to-SA ratios are within the reach of companies with good automation, and that is generally unheard of because so few have bothered to take the time.

If You Can’t Open It, You Don’t Own It

This one is borrowed from the MAKE crowd. The gist is to not let external parties dictate what you can and can not do with your resources.

You Don’t Own Closed Hardware or Software

If you buy a hardware solution and are unable to open it up and replace failed parts, then you don’t really own it. You are obligated to pay others for service, forever. If you lease software in binary form, then you don’t own it (both figuratively and legally). You will be at the mercy of the real owners, paying whatever they wish to charge. Want a real-world example? It was years ago, but read what happened to Revlon.

Bet Your Business on Resources You Own

If you want total control over your hardware and software then you must make your own. This is often not practical, but it is the most complete solution. You control every aspect, but now you need to employ quality developers and engineers.

Open-Source software is often a good compromise. You get access to all aspects of the hardware and software in exchange for sharing any changes you make with the rest of the community.

And it’s a lot harder for anyone to hold you hostage.

Manage the Interesting

So, you are collecting your statistics. Any you look at them every so often. Maybe you even meet to review them with your customers and bosses.

Don’t waste everyone’s time by reviewing graphs of stats that never change.

Spend your time managing the stats that are interesting. If someone asks “Why did that happen?” Or makes a comment like “That’s strange…” that is what you should be spending time managing.

This one seems very basic, but I have seen too many instances of people trying to demonstrate that all is well. Don’t do that more than once. Show what you are collecting, then only show it again if something is wrong. Going on for even a few minutes about how everything is wonderful is just a waste of time.

First, Do No Harm

Yes, stolen from the popular misquote of the Hippocratic Oath. It’s important for SAs to remember, too.

In the heat of fixing the latest production issue, it is important to keep in mind that it is possible to make things even worse. So take a moment to decide the best action, consider what you may be forgetting under the pressure, and slowly and methodically fix the problem.

I once was reprimanded for not having a backup of our most important server. It ran NIS, DNS, pretty much everything for our local LAN. So I mounted a second drive and started a “dd” from the blank disk, over top of the only copy of the very important data… I wasn’t fired, but I did learn an important lesson.

A tenet is a belief held as true by the members of a profession. Here is a list of my tenets, as I understand and believe them. None of these are my own creation – I have learned them from my respected bosses and coworkers.

I stand on the shoulders of giants as I share them with you.

1. First, Do No Harm
2. If You Can’t Measure It, You Can’t Manage It
3. Manage the Interesting
4. If You Can’t Open It, You Don’t Own It
5. Automate The Hell Out Of Everything
6. Oncall Applies In Both Directions
7. Eat Your Own Dog Food
8. Mistakes Are OK
9. Document everything
10. Hire Potential, Train Knowledge

What am I missing? What beliefs do you apply to your work?

Network!

Now is not the time to burn bridges. Swallow your pride, and check your anger. Your next job will most likely come from a lead from someone you know, so start networking. Get names, phone numbers and email addresses from everyone who will share, before you head out the door for the last time. Make sure your Facebook, LinkedIn, and other social networks are up to date with your contact info.

Your New Job

Congratulations! You now have a new job – to find a job. It’s full time, and hard work. If you are getting unemployment or got a severance package, then it’s even a (poorly) paying position. You’ll be getting up every day, and working toward finding a job. Your tasks will be varied – from drafting resumes and cover letters.

Resume(s)

Yes, that’s right – plural. You’ll be writing resumes for each of your applications. Customized for each position. As a hiring manager, I want to see that your are hungry for the job that we are discussing. I highly recommend using a resume hosting service.

Utilize Your Resources

Utilize any contractors at your old job? Give them a call and tell them you are looking. Did you get a placement service in your severance package? Take full advantage of their services. Belong to a professional group? Attend meetings and let them know you are looking for a job. Need some income or just need to fill that new hole in your work experience? Take a temp position in your field, or even just volunteer – no one said your resume can only list paid positions.

A coworker pointed out this article on host naming conventions because it caused such a controversy among the SAs. Go ahead and take a look now, just come back when you are done. So, here are some of the conventions I have used.

The Cute Theme

Is there anyone who has not had a host in the Star Trek themed domain? Choosing a fun theme for naming your hosts is good for entertainment value. Explaining that “grumpy.opssa.net is down” is good for a laugh, even if it’s not very descriptive. Letting your user’s name their own can be good for morale, though.

The Coded Name

In one company, we named each host “Host1234″ and just changed the number for each host. We kept a database of hosts, so we could lookup location, component info, even repair history.

At another we used the host’s serial number as the hostname. We would then record details about the host in a DNS “INFO” record.

The URL Name

A URL encodes the location of an object in the name of the object itself. A really long name. You can do the same general thing in your hostnames. Name the host with any combination of location, function, model or any other information you need. This technique can produce some long hostnames, so keep that in mind if it matters to you.

Or Chose Any Combination

Don’t forget that utilizing DNS “CNAMEs” you don’t have to choose just one technique. Even multiple domains can be used.

So, what technique do you use? Tell us in the comments.