Dealing With Unemployment

In her post titled “How to deal with unemployment in the face of holiday cheer” Penelope covers some views that I share about unemployment in today’s work environment. Please take a look.

and I’ll Add

Looking for work is a full time job. If you already have a job and are looking for a new one, you have some long days ahead of you. Here are some of the things that help the search:

• An online presence – Take control of what the Internet says about you.
• A strong digital resume – Make sure your resume is online and searchable. I like Emurse for resume hosting.
• A “Personal” card – Like a business card, but representing you, not your company. Either a paper one or a digital one. Or both.
• A social presence – Now is the time to be a social butterfly, both online and in real life. Be active on twitter, blogs, linked in and facebook as well as any in-person social situation that will bring you in contact with people.

What Makes Me Nervous

It’s not layoff rumors, or a superior that is too busy for me. It’s not the budget or the outages. It’s not the latest bit of busy work or the next rash of hackers.

That’s not what makes me really nervous.

What makes me really nervous is when the entire team goes long stretches – months – without making a single mistake.

Don’t Be “Hook-ish”

Years ago I had a boss who used to chide me for being “hook-ish”. When I finally asked him what he meant, he picked up a wooden pointer (for chalk boards and such – I said it was a long time ago) and described the parts. A meter-long wooden shaft about a centimeter thick. On one end was a foam point. On the other was a metal screw-in eye-hook. He explained that the point end is the end that got the hard work done, and the hook was used to store the pointer. Picking up and using the pointer was “dangerous” and hanging it up was “safe”.

He told me that I was being too safe – that most of the time I would pick the safe thing to do when given a choice. Which is not terrible, but also not what he wanted me to do. I explained that I was afraid of failing so bad that I’d be fired. He explained that I was not going to get fired for failing or for being hook-ish, but that I being too conservative was worse than being fired.

Fail Spectacularly

I would rather see an employee fail spectacularly – auger-in in a ball of flames – because she is attempting something new, difficult, and dangerous, than succeeding at doing the status quo.

New, difficult and dangerous tasks tend to be the ones with the largest payoff. I’m not saying you should start editing a running binary because that’s risky. I’m saying don’t shy away from the hard stuff just because you may fail.

Winning BIG means risking BIG – and I want my employees to win BIG. Sometimes that means BIG mistakes, and I’m OK with that.

Eat Your Own Dog Food

I have a coworker who hates that phrase – he thinks it’s too self-deprecating. He prefers “Eat Your Own Cooking”, but the sentiment is the same.

Take Pride in Your Work

Taking pride in your product extends past the production stage. You need really want to make great things. If you believe in what you are making, you will naturally want to be a consumer.

Be a Super Consumer

I’m not saying you should only use your own product – in fact just the opposite. You should use your competitor’s products too. Then you will know how each stacks up, and your own experience will teach you where your product needs to be improved.

No one trusts a skinny cook, so be sure to be a consumer of your own product!

Oncall Applies In Both Directions

Most products do not fail at convenent times. So most SAs will find themselves carrying a pager for an oncall shift. I have one major rule that says “oncall applies in both directions” and no one ever understands what I mean (so it probably needs a new name). Here is what I mean.

Oncall Is an Intrusion

Being paged to fix a problem at work is an intrusion on you private life. And in most cases it is considered “part of the job” and doesn’t garner any additional compensation. But it really is worth more than your normal work – it’s an emergency. And an interruption. And involves more work (finding a computer, finding a network, establishing a VPN, etc). Companies generally do not acknowledge this fact.

Intrusions Happen at Work, Too

If your work is allowed to intrude on your life, then your life is allowed to intrude on your work. Plumbers must be let in, packages received, recitals attended, and almost always during business hours. And if your job caused you to miss four hours of sleep last night, then I don’t really want you touching production machines – better to sleep in and come to work with a sharp mind.
It’s basic work/life balance – remind your boss (and HR) if (s)he balks.

Automate The Hell Out Of Everything

There is an old joke that goes like this:

• The junior SA can’t help you because she is too busy because nothing is automated.
• The journeyman SA can’t help you because she is too busy automating everything.
• The senior SA can’t help you because she is too busy playing a First Person Shooter.

I’m sure most senior SAs are watching graphs, writing docs, and learning a new skill, but that is not as funny.

Spend Time Now to Save Time Later

Any task that you have to repeat more than three times should be automated. And right now, before you have to do it again. You will quickly go insane repeating mundane tasks by hand, and you will NEVER be able to handle more than a handful of machines.

I think this is really important – and I’m not alone. Karl thinks so. So does Doug. Automating your tasks now is going to save you time later and shorten the time spent fixing issues in the middle of the night. Doug points out that 1000:1 host-to-SA ratios are within the reach of companies with good automation, and that is generally unheard of because so few have bothered to take the time.

If You Can’t Open It, You Don’t Own It

This one is borrowed from the MAKE crowd. The gist is to not let external parties dictate what you can and can not do with your resources.

You Don’t Own Closed Hardware or Software

If you buy a hardware solution and are unable to open it up and replace failed parts, then you don’t really own it. You are obligated to pay others for service, forever. If you lease software in binary form, then you don’t own it (both figuratively and legally). You will be at the mercy of the real owners, paying whatever they wish to charge. Want a real-world example? It was years ago, but read what happened to Revlon.

Bet Your Business on Resources You Own

If you want total control over your hardware and software then you must make your own. This is often not practical, but it is the most complete solution. You control every aspect, but now you need to employ quality developers and engineers.

Open-Source software is often a good compromise. You get access to all aspects of the hardware and software in exchange for sharing any changes you make with the rest of the community.

And it’s a lot harder for anyone to hold you hostage.

Manage the Interesting

So, you are collecting your statistics. Any you look at them every so often. Maybe you even meet to review them with your customers and bosses.

Don’t waste everyone’s time by reviewing graphs of stats that never change.

Spend your time managing the stats that are interesting. If someone asks “Why did that happen?” Or makes a comment like “That’s strange…” that is what you should be spending time managing.

This one seems very basic, but I have seen too many instances of people trying to demonstrate that all is well. Don’t do that more than once. Show what you are collecting, then only show it again if something is wrong. Going on for even a few minutes about how everything is wonderful is just a waste of time.

If You Can’t Measure It, You Can’t Manage It

“No news is good news” just isn’t valid in the Operations SA world. You need to be watching stats on every little process, at all times.

Which is impossible, of course.

So you need to have automated collection of stats on all your processes, and automated alarming when those stats step outside of the normal bounds. And each time you find and fix an issue, you have the potential to need to add yet another statistic with yet another alarm.

All too often a problem is found to be caused by a process that is not monitored enough, or at all. You probably don’t have too many stats, but too few – no matter how many you are currently collecting.

First, Do No Harm

Yes, stolen from the popular misquote of the Hippocratic Oath. It’s important for SAs to remember, too.

In the heat of fixing the latest production issue, it is important to keep in mind that it is possible to make things even worse. So take a moment to decide the best action, consider what you may be forgetting under the pressure, and slowly and methodically fix the problem.

I once was reprimanded for not having a backup of our most important server. It ran NIS, DNS, pretty much everything for our local LAN. So I mounted a second drive and started a “dd” from the blank disk, over top of the only copy of the very important data… I wasn’t fired, but I did learn an important lesson.

A tenet is a belief held as true by the members of a profession. Here is a list of my tenets, as I understand and believe them. None of these are my own creation – I have learned them from my respected bosses and coworkers.

I stand on the shoulders of giants as I share them with you.

1. First, Do No Harm
2. If You Can’t Measure It, You Can’t Manage It
3. Manage the Interesting
4. If You Can’t Open It, You Don’t Own It
5. Automate The Hell Out Of Everything
6. Oncall Applies In Both Directions
7. Eat Your Own Dog Food
8. Mistakes Are OK
9. Document everything
10. Hire Potential, Train Knowledge

What am I missing? What beliefs do you apply to your work?