Deployment: July 20, 2008

If all went well, this would be the last deployment for a long time. We were pretty sure that we’d fixed all of our issues, and this release was simply to “turn on” all of the functionality for all users—now that it was working, we’d let them all use it.

01:30–01:35 (5 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:35–01:44 (9 minutes): We deployed the application.

01:44–02:19 (35 minutes): We Sanity Tested the application.

02:12–02:31 (19 minutes): The users did their Landing Tests. This was overlapped a bit with the Sanity Testing, because the person doing Sanity had to reboot her computer halfway through the process. However, based on the nature of the changes—i.e. that there were no changes, just a configuration change—we saw very little danger of Sanity failing, so we felt safe in letting the Landing Test start before Sanity was completely finished.

Everything worked fine.

Overall deployment: 01:30–02:31 (1 hour and 1 minute).

Deployment: July 11, 2008

This was sort of a continuation of the previous deployment. If all went well, we’d be able to kiss our recent troubles goodbye, and get back on to a regular schedule, without so many deployments all the time.

01:30–01:33 (3 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:33–01:40 (7 minutes): We made some configuration changes to some of the servers. (There was no “deployment” this time, per se, just configuration changes. But it still caused an outage, and so had to be controlled.)

01:40–01:55 (15 minutes): We performed our Sanity Testing. Everything was fine. Normally Landing Tests would follow, but in this case, since there were no changes to the application itself, it wasn’t deemed necessary.

01:55–01:58 (3 minutes): We verified with the user base that everything was back up and running.

Overall deployment: 01:30–01:58 (28 minutes).

Deployment: July 7, 2008

This was partially a bug-fix deployment, but it was also a configuration deployment. We had a number of configuration changes to make to our web servers, in preparation for some future changes to our network/physical architecture.

22:00–22:01 (1 minute): We all logged onto the conference bridge, and confirmed that we were ready to go.

22:01–22:28 (27 minutes): We made the appropriate configuration changes to our web servers.

22:08–22:38 (10 minutes): We down-graded the logging level on our application servers. Because of the recent troubleshooting we’d had to do, we’d increased the logging greatly, but we now felt it was time to decrease it, and stop chewing up so much disk space with our logs.

22:38–22:50 (12 minutes): We deployed the application.

22:50–23:11 (21 minutes): We performed our Sanity Testing. Everything was fine.

23:11–23:18 (7 minutes): The users did their Landing Test. Everything was fine. (It’s not usual that Sanity Test takes longer than Landing Test, but I guess the users had their testing down to a science, after so many deployments lately—many of which I missed, since I was on vacation.)

23:18–23:22 (4 minutes): We switched back on everything that had to be switched on, and verified with the user base was completely back up and running.

23:22–23:55 (33 minutes): Just as we had all signed off on the deployment, and were about to hang up, one of the users claimed that there was an issue. We spent this time re-testing the functionality, but there was no issue. (Error between keyboard and chair.)

Overall deployment: 22:00–23:55 (1 hour and 55 minutes).

Deployment: June 3, 2008

Apparently the bug fix deployments spilled over out of May, and into June, because here I was doing another one. All preliminary thinking had this being another quick one, though.

22:00–22:06 (6 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

22:06–22:57 (51 minutes): We deployed the application, and made some configuration changes required by the vendor of the application server software we use. (To help them debug a problem we’re having.)

22:57–23:23 (26 minutes): We Sanity Tested the application. Everything was fine.

23:23–00:05 (42 minutes): The users did their Landing Tests. Everything was still fine.

Overall deployment: 22:00–00:05 (2 hours and 5 minutes).

Deployment: May 27, 2008

Yet another bug fix deployment. (This is why I’d been telling people that May was going to suck—deployments left right and centre, for bug fixes. There’s a reason that you should never, ever, ever, rewrite your code base from scratch. But ours is not to question why…) Luckily, we could do this one without any kind of outage, so we got special permission to start sooner than usual: 10:00 PM, instead of 1:30 AM like usual.

22:00–22:05 (5 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

22:05–22:47 (42 minutes): We deployed the application.

22:47–23:13 (26 minutes): We Sanity Tested the application.

23:13–22:50 (37 minutes): The users did their Landing Tests.

Everything worked fine. There were some issues, but minor ones.

Overall deployment: 22:00–22:50 (1 hour and 50 minutes).

Deployment: May 23, 2008

Another bug fix deployment. We weren’t happy about it, because there were some big bugs that we knew still existed, but unfortunately we were having trouble reproducing some of the issues, so it was decided to deploy it into Production, and see if we could get some logs to help us diagnose the problem. That being said, we had fixed a whole bunch of other bugs, so it wasn’t necessarily a complete waste of time. It just irked us, because we knew that we’d be back in a couple of days, doing it again.

0130–01:34 (4 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:34–01:36 (2 minutes): We shut down the application servers.

01:34–01:43 (9 minutes): We executed our database changes.

01:36–01:40 (4 minutes): We made some configuration changes to our web servers.

01:43–01:55 (12 minutes): We brought the application servers back up, and deployed the application.

01:55–02:12 (17 minutes): We Sanity Tested the application.

02:12–04:06 (1 hour and 54 minutes): The users did their Landing Tests.

Everything worked fine. At least, everything that we expected to work fine worked fine—we still had the issues that we were expecting to have.

Overall deployment: 01:30–04:06 (2 hours and 36 minutes).

Deployment: May 16, 2008

This was a simple “bug fix” or “service pack” deployment, to fix some of the issues discovered during the last deployment. Because there was no DBR this time, we expected the deployment to be quicker than usual, and because there was no new functionality, just bug fixes, we also expected the Landing Tests to be quicker than usual.

Quite a number of steps in this deployment are overlapped with each other. We were doing a lot of things at once, and we had to get some log files while we did the deployment, to troubleshoot some issues that we’d been having. This was nothing to do with the deployment itself, but we wanted to do it while we were there. (There were a lot of log files, so this step is overlapped with much of the rest of the deployment; the log files were gzipped and tarred while everything else was going on.)

01:30–01:33 (3 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:33–01:34 (1 minute): We performed a minor configuration change, to turn off the redirection functionality. (See the last deployment for information on that.)

01:34–02:40 (1 hour and 6 minutes): We gathered the log files from the production servers.

01:42–01:52 (10 minutes): We redeployed the old version of the application.

01:52–02:05 (13 minutes): We did our Sanity Testing of the old version of the application.

01:52–02:40 (48 minutes): We adjusted the logging level on the new version of the application. (This wouldn’t normally take this long—it’s a simple configuration change—but again, the person making the change was also busy tarring and gzipping the old log files at the same time. Also, we had to have some discussions about what settings to use.)

02:10–02:15 (5 minutes): The users did their Landing Testing for the old application. (Everything was successful.)

01:52–02:45 (53 minutes): We redeployed the new version of the application. (Again, this took longer than normal because of the overlapping steps above—mostly the gathering of the log files.)

02:45–03:07 (22 minutes): We did our Sanity Testing of the new version of the application.

03:07–03:21 (14 minutes): The users did their Landing Testing for the new version of the application. (Everything was successful.)

Deployment: May 11, 2008

This was our biggest deployment. Maybe not the most important—release 1 would have qualified as the most important—but with this release, we were taking our application and migrating it over to a completely new technology. That meant a new platform, new servers, new software… new everything. It also meant lots of testing.

00:00–00:11 (11 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

00:11–00:55 (44 minutes): We began the backup of our database, before executing the DB scripts.

00:11–00:33 (22 minutes): Concurrently with the database backup, we began configuring our web servers.

00:33–01:00 (27 minutes): We began some configuration of our new application servers.

00:55–01:30 (35 minutes): We executed the DB scripts.

01:30–01:50 (20 minutes): We deployed the application to the application servers.

01:50–02:20 (30 minutes): We did our Sanity Testing.

02:20–06:40 (4 hours and 20 minutes): The users performed Landing Testing. There were issues, but they were all deemed minor enough that we could leave the deployment in place.

06:40–07:10 (30 minutes): We turned on the first “phase” of users. For this rollout, since it was so big, we actually decided to roll the users out in phases; it’s a web-based application, and all users go through the old application, which decides what “phase” the user is supposed to be on; if the user is in a “current” phase, s/he gets redirected to the new application’s URL. So we had to verify not only that the right users were on the right phase, but also that users on “non active” phases were still getting the old version of the application (i.e. that they were not getting redirected).

All in all, the deployment went about as I’d expected it to. I knew that there would be a lot of testing to do, and I’d expected us to find some issues—you can’t rewrite your entire code base in a new technology and not expect to find issues, even late in the game—and I was relieved that none of the issues were deemed serious enough to roll back the deployment.

At this point, we had to monitor the application, and see how it handled the load we were throwing at it.

Deployment: May 5, 2008

May is going to be a rough month—I’m doing deployments three out of four weekends this month. This one was sort of a preparatory deployment; the real functionality would be put into production the next week, but this one was getting ready for it.

01:30–01:33 (3 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:33–01:34 (1 minute): We shut down the application servers.

01:34: We decided that we didn’t need to back up the database, before executing the database scripts, since a back-out of the deployment wouldn’t require backing out the database changes. (The scripts were only creating new objects, that could sit in the database unused in the event of a back-out, no modifications to existing objects were being made.)

01:34–01:38 (4 minutes): We executed the database scripts.

01:35–01:40 (5 minutes): I verified the output logs from the DB scripts, to verify that everything ran successfully.

01:40–01:43 (3 minutes): We restarted the application servers, and deployed one of the two applications we needed to deploy.

01:43–01:44 (1 minute): We deployed the other application.

01:44–02:26 (42 minutes): We did our Sanity Testing. We actually could have finished our Sanity Testing much earlier, but all of the people who needed to do Landing Tests were in their cars on their way to the office, and not ready to do their tests yet, so we just kept testing, until they got in.

02:15–02:36 (21 minutes): We did our Landing Tests. (This actually overlapped with the Sanity Test; the main users who we needed didn’t get into the office until after others did. So we did some Landing Tests, but the really important tests didn’t happen until later in this block of time.) Everything tested fine, which made the deployment a success.

In the end, it actually turned out to be a good thing that the important testing was delayed. It allowed us to get some of the minor testing—which would have been the bulk of time spent—out of the way, and we didn’t have to waste extra time at the end of the deployment.

Deployment: March 16, 2008

This should have been a pretty normal deployment. We were introducing a new system, along with a re-deployment of the usual application I work on, and we weren’t expecting it to take too long.

01:30–01:35 (5 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:35–01:40 (5 minutes): We shut down the application servers.

01:40: We decided not to bother with a database backup, because even if we had to backout the deployment, we wouldn’t be restoring the database, in this case, so we saved ourselves 30–40 minutes by not backing it up. (This is normally a dangerous decision to make; however, the changes we were implementing in this case could have gone in standalone, so even if we backed out the application, we wanted the database changes to stay in.)

01:40–02:00 (20 minutes): We executed our database scripts.

02:00–02:10 (10 minutes): We restarted the application servers.

02:10–02:15 (5 minutes): We deployed the “new” application, and did a quick sanity test. The test worked, but we realized we had a minor configuration problem. (I’d made a typo in the URL for a web service. Small mistakes can make for big problems…)

02:15–02:40 (25 minutes): We paged the person who could fix the problem, and waited for him to respond. (It took a while because his pager was on vibrate mode, and it took a few pages before it was able to wake him up. That’s a common occurrence when you’re trying to get hold of someone at 2:30 in the morning.)

02:40–2:55 (15 minutes): The person logged on, and fixed our problem for us. Still slightly ahead of schedule, at this point.

02:55–03:20 (25 minutes): We had another configuration problem, and had to troubleshoot that. (Someone else had made another typo! Apparently it was contagious.) This one required us to re-compile the application. (It was only a configuration change, but still required a re-build.)

03:20–03:45 (25 minutes): I couldn’t believe it, but we had yet another problem. Our new Oracle package seemed to be having issues, even though it had compiled correctly in the database. (Apologies if you didn’t understand that sentence; I couldn’t muster up the motivation to type out a post explaining what an “Oracle package” is, when I wrote this.) We had to get the DBA back on the bridge, to help us fix it. By the time we’d fixed this problem, we were 5 minutes behind schedule.

03:45–03:50 (5 minutes): We deployed the “main” application, which depends on the “new” application. This brought us back ahead of schedule (by 5 minutes).

03:50–04:05 (15 minutes): We performed our Sanity Testing. Still 5 minutes ahead of schedule.

04:05–05:30 (1 hour and 25 minutes): Landing tests were performed. At this point, I stopped paying attention to the schedule; Landing Tests always go long, with this team. They do a lot of testing. But since it’s the last thing we do, before sign-off, it’s not an issue if this part goes long. That being said, though, we were only about 10 minutes behind schedule, which is not too shabby.

At the end of the day, we deployed with one minor bug. (Can we never have a bug-free release?!?) However, it seemed to be a data integrity issue, and the only accounts that exhibited the behaviour are accounts that we’ve used heavily in the past, for testing. So it’s quite possible that we won’t actually see the issue “in the wild.” (Famous last words…)