Backups seemed to work fine from the replicas, so I went on vacation. The new Storage Area Network (SAN) arrived before I got back. The first crash occurred shortly after it was turned on. Three days before I returned, it ate a number of mailfiles and other user files. Things were almost recovered when I returned on Monday morning.

I spent Monday catching up. The speed of the replica servers hadn't improved; no more memory could be located for the old machines. The SAN dropped out for an hour in the afternoon and people were bumped to the backup servers. The SAN came back up and users quickly returned to their primary mail servers. Things seemed fine for the rest of the week.

Work on the new hardware continued. The servers hosting user home directories migrated successfully, except for printer queues. Servers with site-critical information not only crashed but had to be rolled back to the old SAN. Next in line for migration was mail. Mirroring to the new disks started. Things looked good. Everything still looked good on Wednesday when a firmware patch was applied. The system was restarted at noon. It didn't come back up for a week.

Messages piled up on both side of the corporate firewall because the routers couldn't find the network and the backup servers were too overloaded to respond. Users screamed about slowness. New employees couldn't use mail because their files hadn't been replicated before the primary servers went down.

Because Domino replication wasn't working, Names.nsf, with some configuration changes had to be copied across the network. I logged into the backup server bumping off someone else. Domino stopped. I restarted it. Domino stopped again as soon as I logged out.

The cause: the intern hadn't known to grant SYSTEM rights to run services. I hadn't thought to check. We didn't dare log anyone out of the other boxes.

It took most of the afternoon to get mail moving. Some messages were lost as systems bounced up and down. The department manager hovered by my cubicle entrance to prevent interruptions. Users continued to grumble about slowness but, unlike other parts of the network, mail was working.

Most of the files on most of the servers were recovered. File rights had to be rebuilt on some servers. Once printer queues were restored, printers all over site had to be restarted. Backups are still a problem. The mail replicas are still on old boxes. But at least there's an extra bit of insurance ready for the next upgrade.

DominoPower Contributing Editor Nancy Hand is primary Notes admin at a remote site for a large corporation. She earned both Novell and Microsoft certifications in network engineering before being introduced to Lotus Notes. The 3,000 users she supports constantly challenge Nancy to keep up with their creative missteps. With a background in art, she brings a different perspective to working with computers and their users. In the past, Nancy has worked in the fields of accounting, criminal justice, and museum display. To balance the challenges of the job, she continues to draw and sculpt between stabs at writing novels and designing knitware.