Web Design

Learning from the Rubble

I just had a website blow up on me.  Five months of work and a good deal of it suddenly gone, right as I was trying to launch.  Here's what I learned from the experience.

Version Control

I originally wanted to turn on git for this project, but the web host (Bluehost) wouldn't allow it.  Theoretically it's possible, but I kept getting "out of memory" errors.  Granted, it was a large project with a lot of files, but not that huge.  I suspect the host had pretty strict memory limits.  I sunk about an hour into this task and then gave up and moved on.  We had urgent tasks to attend to and this was a distraction.

But, of course, git is not a distraction.  Git is your workflow.  Git is important.  When youi break something, git is there to save you.  Generally speaking, git means your files are in more than one place.  So when a server explodes, it's okay because you've got all the data stored locally.  If I had insisted on getting git running, that single act would have saved me when the server blew up.

 

Do Your Own Backups

In fact, my client told me to do this.  I looked into the automated backups provided by the web host and concluded that they were sufficient: the web host provided automatic daily, weekly, and monthly backups.  I trusted them.

I shouldn't have.

When the site exploded and we needed to restore from backup, we discovered to our horror that the backups were missing a lot of files, especially inside subdirectories.  Well, this is Drupal.  There are a lot of subdirectories.  Most painfully, we lost most of the custom theme we'd been working on for five months, including the main CSS style sheet.

Again, I knew better than to rely on the web host.  Manual backups are a standard part of my workflow.  But in this case, we had easily a terabyte and a half of data that I would have had to download.  It was late, I was tired, and I wanted to put the site online and get on with my life.  So I cut a corner.  If I had done this one thing, I could have prevented this tragedy.

 

Don't Do Important Things When You're Tired

This saga played out over a two day period.  On both days, I didn't even start work until after 10 pm.  I'm not a night owl any more; it's hard to stay up late every night when you have to take the kids to daycare at 7 am.  A lot of web developers still live like college students, but that doesn't work for me any more.  Maybe that's a disadvantage, but you work with what you've got.

Naturally, I do plenty of work when I'm tired.  But in this case, I took dangerous risks at a critical stage of deployment.  While half asleep.

The late hour and my own fatigue contributed directly to the failure.  If it had been 10 am and if I had been well rested, I would have insisted on running a manual backup before trying to deploy.  But it was late and I was anxious to go to bed, so I cut corners.

Part of the reason I agreed to work like that was because the client requested it.  He was facing a deadline and didn't want to tell his clients that they would have to wait one more day.  I agreed to work under those conditions in order to help him meet his deadline.  I went against my own advice and now the site won't be ready for another week or two.

The lesson: When you're tired, go to bed.  If you miss the deadline, you miss the deadline.  You're setting yourself up for much worse harm if you push yourself too far.  You know yourself better than the client knows you, and you need to know when to call it a night.

 

Don't Use Bluehost

My client would disagree with this.  He's used Bluehost for years and never had this kind of problem before.  I know they're one of the more popular shared hosts for small Drupal sites.  But so far, my experience with them has been terrible:

  • Very low server resources have led to performance issues (even compared with other low budget shared hosts)
  • No git support
  • Backups failed when I needed them
  • Third tier tech support won't talk to you (that might be standard but still not a point in their favor)
  • Lousy support for top-level domains
  • Tool for managing sub-domains didn't work

To be sure, there are some nice things about Bluehost, but nothing that I couldn't have gotten with Hostgator or any number of other hosts.  I've been with Hostgator for years now and have been much happier with them.  I've built much more demanding sites than this, and have gone through much trickier deployments, and have never experienced this kind of problem.  I will never willingly use Bluehost again.

 

Stick To Your Guns

A lot of the above lessons ended with "I knew it was a bad idea but did it anyway."  The lesson here is: If you know it's a bad idea, don't do it.  The client and I were both eager to meet a deadline and we blinded ourselves to the risks of cutting corners.  I was only a subcontractor on this project; I had no vested interest in the outcome, aside from making the client happy.  To be sure, I've had a very positive relationship with this client for a long time, spanning multiple projects.  But that's all the more reason to stick to your guns.  I did him a disservice by ignoring my own advice.

Setting up a proper workflow - and sticking to it - takes a lot of extra time, and there is a strong temptation to just dive in and start working.  But being a mature professional means having the discipline to hold your horses back and lay the proper groundwork first.  I think "discipline" is the real key word here.  Be careful and deliberate, and disciplined, and you can save yourself from a lot of misery.

Unfortunately, I'm one of those people who has to learn things the hard way.  Now I'm donating my time to help the client repair the site; my mistakes have cost me time and money.  But I have learned my lesson.  Not just about setting up a proper workflow, but about being careful, deliberate, and above all discplined.  And now I have the experience to back it up.  That's what makes a good developer.