serna Deployments

Deployment: March 21, 2009

2009-03-22T02:44:00.000-04:00

This should have been a fairly typical deployment. We were putting in a new J2EE application, with a database change. The minor difference between this one and my normal deployments is that this wasn’t an update, it was a brand new J2EE application; so if we screwed up, and it wasn’t properly deployed on time, we wouldn’t have to roll back to a previous version.

22:00–22:01 (1 minute): We all logged onto the conference bridge, and confirmed that we were ready to go.

22:01–23:33 (1 hour and 32 minutes): The DBA ran the database scripts. There were some errors, so this took a lot longer than expected, but she managed to work through the problems and get everything done.

22:01–22:34 (33 minutes): We configured the J2EE cluster (connection pools, security, etc.). We then had to wait for the database scripts to be finished, to move on to the next part of the deployment.

23:33–02:30 (2 hours, 57 minutes): We deployed the J2EE application. Again, there were difficulties, which were hard to troubleshoot, because they had to do with some internal workings of our vendor’s software, rather than our own code. We were getting errors between the vendor’s software and their database objects. After troubleshooting as much as we could, we called it a night.

Overall deployment: 22:00–02:30 (4 hours and 30 minutes). We left the bridge knowing that there was a good chance we’d need to rebuild the environment from scratch; it was quite possibly a configuration issue, but the likelihood of being able to track it down was slim. We tentatively planned to retry the next night, schedules (and permissions) pending.

Sanity/Landing Tests Defined/Explained

2009-03-22T00:32:00.001-04:00

I use the terms Sanity Test and Landing Test a lot, so I should define what I mean by them—others might use these terms in different ways, but this is how I and my team use them.

When we’re doing a deployment, there is the technical team, which is doing the actual work of the deployment, and troubleshooting any technical issues, and there are the business users, who want to make sure that the application meets their standards before they hand it over to the end users.

The Sanity Tests are done by the technical team. Once we’ve deployed the application, and everything is up and running, we quickly run through it to make sure that it really is up and running. As the name implies, this really is just a quick sanity check, to make sure that everything is up; it’s not meant to be exhaustive.

The Landing Tests are done by the business users. Once the technical team has given the go ahead that the application is up and running (as verified by sanity testing), the business users go ahead and verify that the application is working according to their specifications. As with the sanity tests, landing tests are not meant to be exhaustive; the exhaustive testing should have been done long before the deployment! (We have a phase of testing called User Acceptance Testing, or UAT, which is where the exhaustive testing should have been done.) The purpose of the landing tests is for the business to ensure that everything is running, and that’s it. If a bug is discovered, it must be one of two things:

An environmental issue, where the production environment is different than the other environments and it’s causing a problem, or
A gap in the UAT, where something wasn’t tested properly

We do our best to plan for the former. We get very annoyed by the latter, but to err is human.

If either the sanity tests or the landing tests fail, we do our best to try and fix the problems right then and there, but that’s not always possible—especially at 2:00 in the morning. So when there are issues, it’s up to the business to decide if these issues are “show stoppers,” meaning we have to abort the deployment and roll back to the previous version of the application, or whether the issues are minor enough that they can live with them.

Deployment: March 19, 2009

2009-03-19T02:27:00.001-04:00

Not a complex deployment, this time. A simple J2EE EAR file to be updated, with no database scripts to worry about or configuration changes to make.

01:30–01:43 (13 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go. It took longer than it was supposed to; some people joined late.

01:43–02:02 (19 minutes): We un-deployed the old version of the EAR file, and deployed the new version.

02:02–02:08 (6 minutes): We did our Sanity Testing, and ensured that the app came back online successfully.

02:08–02:16 (8 minutes): We waited for the business to join the conference call, to do their Landing Tests; we were a bit ahead of schedule, so they hadn’t been expecting to join the call this early.

02:16–02:25 (9 minutes): The client did their Landing Testing, and signed off that everything was working as it was supposed to.

Overall deployment: 01:30–02:25 (55 minutes).

[Mini] Deployment: September 20, 2008

2008-09-20T02:02:00.001-04:00

This was actually just a simple database script, to update one of our data tables. No structural changes, just data. But someone decided to treat it like a full deployment anyway, complete with the conference call and everything. So…

01:30–01:37 (7 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go. It took a bit longer than we’d expected, because we weren’t sure if one particular person was going to join or not.

01:37–01:37 (about 30 seconds): We backed up the database.

01:37–01:41 (4 minutes): We executed the DB scripts.

01:41–01:48 (7 minutes): We did our Sanity Test—mostly just checking the logs that the scripts worked fine, as well as a quick glance at the application, to verify that the data was showing up correctly.

01:43–01:53 (10 minutes): The client performed their Landing Test. (Further tests that the data was showing up correctly in the application…) We let them start before the Sanity was done, in this case, since all indications were that things had worked fine, and we were just waiting on an email to show up in my Inbox to verify the scripts.

Overall, we were done ahead of schedule, and everything worked well. Total deployment: 01:30–01:53 (23 minutes).

Deployment: September 11, 2008

2008-09-11T05:28:00.001-04:00

This was a bug fix deployment, to fix the issue discovered with the previous deployment.

01:30–01:34 (4 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:34–02:01 (27 minutes): We backed up the database, and executed the database scripts. This took longer than usual, because the DBA accidentally ran the scripts on the wrong database, the first time.

01:34–01:56 (22 minutes): We un-deployed the old version of the application, and shut down the application servers.

02:06–02:19 (13 minutes): We deployed the new version(s) of the application(s). (Some were actually still deployed from the last time, and just had to be re-activated, while others—the ones with bug fixes—actually had to be redeployed.)

02:19–03:04 (45 minutes): We conducted the Sanity Test. During Sanity, we discovered an issue unrelated to our deployment—one that had been part of the application for the previous four months.

03:00–05:23 (2 hours and 23 minutes): The client did their Landing Tests. (This was overlapped with the Sanity Test a bit.) Some additional issues were discovered, but they were all deemed minor enough that we didn’t need to back out. (The most serious one turned out to not be a bug with our application, but with a back-end system that we depend on. So the bug fix will be for them, not for us.)

Overall deployment: 01:30–05:23 (3 hours and 53 minutes).

Deployment: September 8, 2008

2008-09-11T00:40:00.001-04:00

No timeline with this one.

There had been an issue with the previous deployment. Every once in a while, the CPU on the application servers was getting up close to 100%, and the Support Team had to reboot them to keep the application alive.

Unfortunately, rolling back wasn’t working; we tried un-deploying the application, and re-deploying the old version, but the new version seemed to be cached on the application servers. So we had to call the application server vendor, and get help. (It turned out we had to undeploy, remove the temp directory, and then re-deploy the old version.)

Deployment: September 7, 2008

2008-09-07T06:42:00.001-04:00

This was a slightly more intricate deployment than usual, because there were a few moving parts. We had not one but two sets of database changes to go through, plus three separate applications to deploy and/or re-deploy on the application servers. Plus, because of our new “global delivery model,” we had a DBA in India who would be making our database changes, which was new. So I was hoping it wouldn’t be a bad deployment, but I was a bit worried.

00:00: As I was getting ready for the deployment, I logged into my computer, and noticed an email from the DBA in India, dated twenty-four hours prior to the deployment, saying that he wasn’t able to join the bridge, but would I please call his cell phone when we were ready for him to start. Not a good sign. I looked in the database, and found that the changes had already been executed (presumably twenty-four hours earlier than they were supposed to be done). This could have been very bad, but we got lucky, and it turns out that the database changes didn’t break the existing application; the old version of the app was able to run for the last twenty-four hours with the database changes.

01:30–01:50 (20 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go. This took much longer than usual, because we had to confirm with the DBA what had actually been done, and what he hadn’t. It turned out to be even more confusing than I’d thought: out of the two DB changes we needed, one was partially done, and one was not done at all.

01:50–01:52 (2 minutes): We shut down the application servers.

01:52–02:21 (29 minutes): We had the DBA finish the database changes.

02:21–02:24 (3 minutes): We brought back up the application servers

02:24–02:42 (18 minutes): We deployed the updated and new applications.

02:42–05:00 (2 hours 18 minutes): We did our Sanity Testing. We found an issue with one piece of functionality, and determined that it might have been due to a back-end system, not ours. We spent some time troubleshooting it, and ended up not being sure which system was the culprit.

03:40–05:00 (1 hour 20 minutes): The client did their Landing Tests. This was overlapped with the troubleshooting we did, for the functionality that wasn’t working.

05:00–05:30 (30 minutes): One of the back-end systems that we depend on went down for scheduled maintenance, and we had to wait for it to come back up before we could go back to our testing.

05:30–06:20 (50 minutes): We finished testing, and then called it a wrap. There was still an oustanding issue—the one we weren’t able to troubleshoot—but it wasn’t a show-stopper.

Overall deployment: 01:30–06:20 (4 hours and 50 minutes).

Deployment: July 20, 2008

2008-07-20T02:40:00.001-04:00

If all went well, this would be the last deployment for a long time. We were pretty sure that we’d fixed all of our issues, and this release was simply to “turn on” all of the functionality for all users—now that it was working, we’d let them all use it.

01:30–01:35 (5 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:35–01:44 (9 minutes): We deployed the application.

01:44–02:19 (35 minutes): We Sanity Tested the application.

02:12–02:31 (19 minutes): The users did their Landing Tests. This was overlapped a bit with the Sanity Testing, because the person doing Sanity had to reboot her computer halfway through the process. However, based on the nature of the changes—i.e. that there were no changes, just a configuration change—we saw very little danger of Sanity failing, so we felt safe in letting the Landing Test start before Sanity was completely finished.

Everything worked fine.

Overall deployment: 01:30–02:31 (1 hour and 1 minute).

Deployment: July 11, 2008

2008-07-11T02:04:00.001-04:00

This was sort of a continuation of the previous deployment. If all went well, we’d be able to kiss our recent troubles goodbye, and get back on to a regular schedule, without so many deployments all the time.

01:30–01:33 (3 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:33–01:40 (7 minutes): We made some configuration changes to some of the servers. (There was no “deployment” this time, per se, just configuration changes. But it still caused an outage, and so had to be controlled.)

01:40–01:55 (15 minutes): We performed our Sanity Testing. Everything was fine. Normally Landing Tests would follow, but in this case, since there were no changes to the application itself, it wasn’t deemed necessary.

01:55–01:58 (3 minutes): We verified with the user base that everything was back up and running.

Overall deployment: 01:30–01:58 (28 minutes).

Deployment: July 7, 2008

2008-07-08T00:00:00.000-04:00

This was partially a bug-fix deployment, but it was also a configuration deployment. We had a number of configuration changes to make to our web servers, in preparation for some future changes to our network/physical architecture.

22:00–22:01 (1 minute): We all logged onto the conference bridge, and confirmed that we were ready to go.

22:01–22:28 (27 minutes): We made the appropriate configuration changes to our web servers.

22:08–22:38 (10 minutes): We down-graded the logging level on our application servers. Because of the recent troubleshooting we’d had to do, we’d increased the logging greatly, but we now felt it was time to decrease it, and stop chewing up so much disk space with our logs.

22:38–22:50 (12 minutes): We deployed the application.

22:50–23:11 (21 minutes): We performed our Sanity Testing. Everything was fine.

23:11–23:18 (7 minutes): The users did their Landing Test. Everything was fine. (It’s not usual that Sanity Test takes longer than Landing Test, but I guess the users had their testing down to a science, after so many deployments lately—many of which I missed, since I was on vacation.)

23:18–23:22 (4 minutes): We switched back on everything that had to be switched on, and verified with the user base was completely back up and running.

23:22–23:55 (33 minutes): Just as we had all signed off on the deployment, and were about to hang up, one of the users claimed that there was an issue. We spent this time re-testing the functionality, but there was no issue. (Error between keyboard and chair.)

Overall deployment: 22:00–23:55 (1 hour and 55 minutes).

Deployment: June 3, 2008

2008-06-04T00:07:00.001-04:00

Apparently the bug fix deployments spilled over out of May, and into June, because here I was doing another one. All preliminary thinking had this being another quick one, though.

22:00–22:06 (6 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

22:06–22:57 (51 minutes): We deployed the application, and made some configuration changes required by the vendor of the application server software we use. (To help them debug a problem we’re having.)

22:57–23:23 (26 minutes): We Sanity Tested the application. Everything was fine.

23:23–00:05 (42 minutes): The users did their Landing Tests. Everything was still fine.

Overall deployment: 22:00–00:05 (2 hours and 5 minutes).

Deployment: May 27, 2008

2008-05-27T23:54:00.001-04:00

Yet another bug fix deployment. (This is why I’d been telling people that May was going to suck—deployments left right and centre, for bug fixes. There’s a reason that you should never, ever, ever, rewrite your code base from scratch. But ours is not to question why…) Luckily, we could do this one without any kind of outage, so we got special permission to start sooner than usual: 10:00 PM, instead of 1:30 AM like usual.

22:00–22:05 (5 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

22:05–22:47 (42 minutes): We deployed the application.

22:47–23:13 (26 minutes): We Sanity Tested the application.

23:13–22:50 (37 minutes): The users did their Landing Tests.

Everything worked fine. There were some issues, but minor ones.

Overall deployment: 22:00–22:50 (1 hour and 50 minutes).

Deployment: May 23, 2008

2008-05-23T04:10:00.001-04:00

Another bug fix deployment. We weren’t happy about it, because there were some big bugs that we knew still existed, but unfortunately we were having trouble reproducing some of the issues, so it was decided to deploy it into Production, and see if we could get some logs to help us diagnose the problem. That being said, we had fixed a whole bunch of other bugs, so it wasn’t necessarily a complete waste of time. It just irked us, because we knew that we’d be back in a couple of days, doing it again.

0130–01:34 (4 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:34–01:36 (2 minutes): We shut down the application servers.

01:34–01:43 (9 minutes): We executed our database changes.

01:36–01:40 (4 minutes): We made some configuration changes to our web servers.

01:43–01:55 (12 minutes): We brought the application servers back up, and deployed the application.

01:55–02:12 (17 minutes): We Sanity Tested the application.

02:12–04:06 (1 hour and 54 minutes): The users did their Landing Tests.

Everything worked fine. At least, everything that we expected to work fine worked fine—we still had the issues that we were expecting to have.

Overall deployment: 01:30–04:06 (2 hours and 36 minutes).

Deployment: May 16, 2008

2008-05-16T03:39:00.001-04:00

This was a simple “bug fix” or “service pack” deployment, to fix some of the issues discovered during the last deployment. Because there was no DBR this time, we expected the deployment to be quicker than usual, and because there was no new functionality, just bug fixes, we also expected the Landing Tests to be quicker than usual.

Quite a number of steps in this deployment are overlapped with each other. We were doing a lot of things at once, and we had to get some log files while we did the deployment, to troubleshoot some issues that we’d been having. This was nothing to do with the deployment itself, but we wanted to do it while we were there. (There were a lot of log files, so this step is overlapped with much of the rest of the deployment; the log files were gzipped and tarred while everything else was going on.)

01:30–01:33 (3 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:33–01:34 (1 minute): We performed a minor configuration change, to turn off the redirection functionality. (See the last deployment for information on that.)

01:34–02:40 (1 hour and 6 minutes): We gathered the log files from the production servers.

01:42–01:52 (10 minutes): We redeployed the old version of the application.

01:52–02:05 (13 minutes): We did our Sanity Testing of the old version of the application.

01:52–02:40 (48 minutes): We adjusted the logging level on the new version of the application. (This wouldn’t normally take this long—it’s a simple configuration change—but again, the person making the change was also busy tarring and gzipping the old log files at the same time. Also, we had to have some discussions about what settings to use.)

02:10–02:15 (5 minutes): The users did their Landing Testing for the old application. (Everything was successful.)

01:52–02:45 (53 minutes): We redeployed the new version of the application. (Again, this took longer than normal because of the overlapping steps above—mostly the gathering of the log files.)

02:45–03:07 (22 minutes): We did our Sanity Testing of the new version of the application.

03:07–03:21 (14 minutes): The users did their Landing Testing for the new version of the application. (Everything was successful.)

Deployment: May 11, 2008

2008-05-11T07:14:00.001-04:00

This was our biggest deployment. Maybe not the most important—release 1 would have qualified as the most important—but with this release, we were taking our application and migrating it over to a completely new technology. That meant a new platform, new servers, new software… new everything. It also meant lots of testing.

00:00–00:11 (11 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

00:11–00:55 (44 minutes): We began the backup of our database, before executing the DB scripts.

00:11–00:33 (22 minutes): Concurrently with the database backup, we began configuring our web servers.

00:33–01:00 (27 minutes): We began some configuration of our new application servers.

00:55–01:30 (35 minutes): We executed the DB scripts.

01:30–01:50 (20 minutes): We deployed the application to the application servers.

01:50–02:20 (30 minutes): We did our Sanity Testing.

02:20–06:40 (4 hours and 20 minutes): The users performed Landing Testing. There were issues, but they were all deemed minor enough that we could leave the deployment in place.

06:40–07:10 (30 minutes): We turned on the first “phase” of users. For this rollout, since it was so big, we actually decided to roll the users out in phases; it’s a web-based application, and all users go through the old application, which decides what “phase” the user is supposed to be on; if the user is in a “current” phase, s/he gets redirected to the new application’s URL. So we had to verify not only that the right users were on the right phase, but also that users on “non active” phases were still getting the old version of the application (i.e. that they were not getting redirected).

All in all, the deployment went about as I’d expected it to. I knew that there would be a lot of testing to do, and I’d expected us to find some issues—you can’t rewrite your entire code base in a new technology and not expect to find issues, even late in the game—and I was relieved that none of the issues were deemed serious enough to roll back the deployment.

At this point, we had to monitor the application, and see how it handled the load we were throwing at it.

Deployment: May 5, 2008

2008-05-04T02:51:00.001-04:00

May is going to be a rough month—I’m doing deployments three out of four weekends this month. This one was sort of a preparatory deployment; the real functionality would be put into production the next week, but this one was getting ready for it.

01:30–01:33 (3 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:33–01:34 (1 minute): We shut down the application servers.

01:34: We decided that we didn’t need to back up the database, before executing the database scripts, since a back-out of the deployment wouldn’t require backing out the database changes. (The scripts were only creating new objects, that could sit in the database unused in the event of a back-out, no modifications to existing objects were being made.)

01:34–01:38 (4 minutes): We executed the database scripts.

01:35–01:40 (5 minutes): I verified the output logs from the DB scripts, to verify that everything ran successfully.

01:40–01:43 (3 minutes): We restarted the application servers, and deployed one of the two applications we needed to deploy.

01:43–01:44 (1 minute): We deployed the other application.

01:44–02:26 (42 minutes): We did our Sanity Testing. We actually could have finished our Sanity Testing much earlier, but all of the people who needed to do Landing Tests were in their cars on their way to the office, and not ready to do their tests yet, so we just kept testing, until they got in.

02:15–02:36 (21 minutes): We did our Landing Tests. (This actually overlapped with the Sanity Test; the main users who we needed didn’t get into the office until after others did. So we did some Landing Tests, but the really important tests didn’t happen until later in this block of time.) Everything tested fine, which made the deployment a success.

In the end, it actually turned out to be a good thing that the important testing was delayed. It allowed us to get some of the minor testing—which would have been the bulk of time spent—out of the way, and we didn’t have to waste extra time at the end of the deployment.

Deployment: March 16, 2008

2008-03-16T05:56:00.001-04:00

This should have been a pretty normal deployment. We were introducing a new system, along with a re-deployment of the usual application I work on, and we weren’t expecting it to take too long.

01:30–01:35 (5 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

01:35–01:40 (5 minutes): We shut down the application servers.

01:40: We decided not to bother with a database backup, because even if we had to backout the deployment, we wouldn’t be restoring the database, in this case, so we saved ourselves 30–40 minutes by not backing it up. (This is normally a dangerous decision to make; however, the changes we were implementing in this case could have gone in standalone, so even if we backed out the application, we wanted the database changes to stay in.)

01:40–02:00 (20 minutes): We executed our database scripts.

02:00–02:10 (10 minutes): We restarted the application servers.

02:10–02:15 (5 minutes): We deployed the “new” application, and did a quick sanity test. The test worked, but we realized we had a minor configuration problem. (I’d made a typo in the URL for a web service. Small mistakes can make for big problems…)

02:15–02:40 (25 minutes): We paged the person who could fix the problem, and waited for him to respond. (It took a while because his pager was on vibrate mode, and it took a few pages before it was able to wake him up. That’s a common occurrence when you’re trying to get hold of someone at 2:30 in the morning.)

02:40–2:55 (15 minutes): The person logged on, and fixed our problem for us. Still slightly ahead of schedule, at this point.

02:55–03:20 (25 minutes): We had another configuration problem, and had to troubleshoot that. (Someone else had made another typo! Apparently it was contagious.) This one required us to re-compile the application. (It was only a configuration change, but still required a re-build.)

03:20–03:45 (25 minutes): I couldn’t believe it, but we had yet another problem. Our new Oracle package seemed to be having issues, even though it had compiled correctly in the database. (Apologies if you didn’t understand that sentence; I couldn’t muster up the motivation to type out a post explaining what an “Oracle package” is, when I wrote this.) We had to get the DBA back on the bridge, to help us fix it. By the time we’d fixed this problem, we were 5 minutes behind schedule.

03:45–03:50 (5 minutes): We deployed the “main” application, which depends on the “new” application. This brought us back ahead of schedule (by 5 minutes).

03:50–04:05 (15 minutes): We performed our Sanity Testing. Still 5 minutes ahead of schedule.

04:05–05:30 (1 hour and 25 minutes): Landing tests were performed. At this point, I stopped paying attention to the schedule; Landing Tests always go long, with this team. They do a lot of testing. But since it’s the last thing we do, before sign-off, it’s not an issue if this part goes long. That being said, though, we were only about 10 minutes behind schedule, which is not too shabby.

At the end of the day, we deployed with one minor bug. (Can we never have a bug-free release?!?) However, it seemed to be a data integrity issue, and the only accounts that exhibited the behaviour are accounts that we’ve used heavily in the past, for testing. So it’s quite possible that we won’t actually see the issue “in the wild.” (Famous last words…)

Deployment: March 3, 2008

2008-03-03T00:41:00.001-05:00

We had the conference call, regarding the previous deployment, and it was decided that we were confident enough in the original database scripts to proceed with the deployment, at 20:00. We would be using a different DBA.

20:00: We all logged onto the conference bridge, and confirmed that we were ready to go.

20:00–20:20 (20 minutes): We shut down the application servers for both apps.

20:20–20:30 (10 minutes): We backed up the back-end databases for both applications.

20:30–20:50 (20 minutes): We executed the database scripts for Application 1.

20:50–21:20 (30 minutes): We deployed Application 1.

20:50–21:25 (35 minutes): We executed the database scripts for Application 2, in parallel with the deployment of Application 1.

21:20–21:25 (5 minutes): We did sanity testing of Application 1, which turned out fine. (Only 5 minutes because sanity testing for this particular application is pretty quick.

21:25–23:20 (1 hour and 55 minutes): The client did their Landing Test of Application 1, and everything looked good.

21:30–21:45 (15 minutes): We deployed Application 2.

21:45–22:00 (15 minutes): We did sanity testing of Application 2, and everything looked good.

22:00–00:40 (2 hours and 40 minutes): The client did their Landing Test of Application 2. There were some minor issues discovered, but none were show-stoppers. They signed off on the deployment, and we all went to bed.

The last time we’d deployed this, one of the back-end systems we depend on went crazy, and we had to back out. Many of us were thinking negative thoughts, worrying about having to do the same thing this time, as our heads hit our pillows…

Deployment: March 2, 2008

2008-03-02T06:37:00.001-05:00

This was a retry of the previous deployment, which was backed out.

For clarity, I’ve decided to start using military notation for the times, so that there’s no confusion between, e.g., 12:00 midnight and 12:00 noon. (I’m still rounding times to the nearest 5 minutes, though, for clarity.)

00:00–00:10 (10 minutes): We all logged onto the conference bridge, and confirmed that we were ready to go.

00:10–00:30 (20 minutes): We made some configuration changes to the Application 1 servers, and shut down the application servers for both Application 1 and Application 2.

00:30–01:45 (1 hour and 15 minutes): We backed up the databases, just in case. This took longer than usual, because the DBA had to back up one of the databases to his own machine, instead of doing it on the server. (It’s normally done directly on the server, which avoids the backup having to travel across the network, but he didn’t plan ahead to get the Unix passwords for the boxen in question.)

01:25–03:20 (2 hours and 55 minutes): We made the modifications to the database for Application 1. (This overlapped with the task above, as the database for Application 2 continued to be backed up.) Again, this took much longer than it should have—it took just under an hour and a half, last time—because the DBA had to run the scripts over the network, instead of from the server.

03:20–03:25 (5 minutes): There were issues with the database scripts, for Application 1. It turns out that the DBA modified the scripts, without telling anyone, before running them, but the modified scripts ran into issues. Because we were not confident that the database was left in a valid state, it was decided to roll back, and restore the database from the backup.

03:20–04:30 (1 hour and 10 minutes): During this time, the team had a roundtable conversation to discuss whether we could still put in Application 2, even though Application 1 was rolled back. The problem is that the new version of Application 1 had never been tested against the old version of Application 1. It was decided that the risk was low—the nature of the changes being made indicated that the different versions of the applications should work together—but risk is risk, and if they weren’t tested, there could be a chance of errors being undiscovered. It was finally decided to leave both applications as-is—that is, not try to deploy Application 2 without Application 1—and revisit the deployment in a day or two.

04:30–05:10 (40 minutes): We brought the applications back up. Luckily, based on the nature of the database changes for Application 1, it was decided that we didn’t need to restore the database; all we had to do was undo some configuration changes, on the application servers, and then bring those servers back online.

04:40–04:50 (10 minutes): The business came onto the bridge, and said that they’d prefer to try the deployment again, rather than roll back. Just to be safe, it was decided to continue with the rollback, and do a quick sanity test, so that there would be some version of the application online, while the decision was made. We also needed to chase the DBA team, and find a new DBA to perform the work.

05:10–05:20 (10 minutes): We all did a sanity/regression test, to verify that the applications were back up and running. They were fine.

05:20–05:30 (10 minutes): Dead air, as we waited for the DBA team to get back to us.

05:30–06:30 (1 hour): Another DBA joined the conference call, and was walked through the existing dilemma. He asked for some time to look over the situation—including what was changed, and why. Presumably, the results of his investigation would give the rest of us the warm and fuzzy that we can proceed to try again.

The result of this discussion was that the DBA(s) would try to recreate the errors we got, to prove that it was the modified scripts that caused the problems. We would reconvene at noon, to go over the findings, and talk about rescheduling the deployment.

Deployment: February 21, 2008—Backout!

2008-02-21T16:30:00.001-05:00

After a [mostly] successful deployment, some of the back-end systems that we depend on went crazy. The system was down for most of the morning. It was decided to roll back everything that went in during the night. (That included not just our application, but numerous other applications that had also deployed.)

12:00 (noon)–12:30 (30 minutes): We opened the bridge, and waited for people to join. Since this was an ad-hoc bridge, being opened at the last minute, it took longer than usual to round everyone up, and get them on the bridge. (Some of them may still have been sleeping, since we were exactly twelve hours after the original deployment.) Unfortunately, we weren’t able to get the DBA who had done the original deployment, but we had two other DBAs helping us.

12:30–12:45 (15 minutes): We shut down the application servers for both applications, and rolled back some of the configuration changes necessary for Application 1. (See the description of the original deployment, for what we were deploying, and a mention of “Application 1” vs. “Application 2.”)

12:45: At the last minute, the DBA who had done our original deployment showed up. This gave me a sense of relief, because it’s always easier for someone to back out her own work than for someone else to do it—even with good, detailed instructions.

12:45–1:20 (35 minutes): The DBAs rolled back the database changes. This is why the backups taken during the deployment were so crucial; they simply deleted the appropriate schemas, and re-imported the dump file taken during the backup. The DBA who had done our original deployment restored one of the databases, and, to shorten the outage time, another of the DBAs restored the other one. (All this time, the application was down, and users were unable to access it, so every second mattered.)

1:20–1:45 (25 minutes): The old version of Application 1 was re-deployed.

1:45–3:00 (1 hour and 15 minutes): Sanity Testing commenced for Application 1, and failed; the application didn’t come up correctly. It was believed to be a problem with the deployment procedure, so it was un-deployed, and re-deployed again. Unfortunately, the problem still remained. It was eventually determined that the application was deployed successfully, but we had a problem with a tool called Java Web Start—the Admin Tool for the application uses Java Web Start, and because we were using a new version of the Admin Tool, JWS didn’t re-download the old version.

As of 3:00, we were rolled back to our previous state—with no idea when we would next be attempting the deployment.

Deployment: February 21, 2008

2008-02-21T05:48:00.001-05:00

This was a problematic deployment before we even got started. First of all, it was a full moon—and not just any full moon, a blood-red full moon. (Isn’t it a bad omen to be doing a deployment when the moon is turning to blood?)

But that wasn’t what I was referring to. It was a problematic deployment because we were dependent on three separate systems, for the deployment, and two of those systems were themselves dependent on a number of other systems being deployed.

There was a lot of confusion about which system had to be deployed before which other system, or which systems had to be installed on the same night, etc. Unfortunately, one of the systems—one of the ones that were two or three removed from my system—had issues, so they weren’t going to be able to deploy before us. It was decided that we could deploy anyway, though; our system would be able to gracefully handle the missing back-end system, simply displaying an error message at the appropriate spot, and when they were able to deploy the other system, our system would just magically start working. (Unfortunately, it meant that I’d still have to take part in the other deployment, to do some testing, which means that I’d have another sleepless night coming up.)

Also, we ourselves were deploying not one, but two applications. I’ll call them Application 1 and Application 2.

Once we finally got to the point where we were ready to deploy, here’s how it went down.

12:00 midnight: We all logged onto the conference bridge, and confirmed that we were ready to go.

12:05–12:10 (5 minutes): We shut down the application servers for both Application 1 and Application 2. To save time, we also made some necessary configuration changes, during this time, for Application 1. (It was a last-minute change to the procedure; we realized that if we’d waited until it was scheduled, it would have meant an extra reboot in the process. As much planning as you try to do ahead of time, you just can’t control when you’ll have a good idea.)

12:10–12:30 (20 minutes): We backed up the back-end databases that we depend on. Since we had to make some changes to the schemas, it’s always a good idea to back them up, first, in case you have to rollback the deployment.

12:25–1:50 (1 hour 25 minutes): We made the appropriate modifications to the database for Application 1. (The backup of this database finished before the other one, which is why this step starts before the end of the previous step.) Unfortunately, while the database scripts were executing, the database ran out of space in the “temp” tablespace. So the DBA had to page someone who does more low-level support of the database, to get the issue resolved, before the script could be run.

12:30–12:50 (20 minutes): We made the modifications to the database for Application 2. Again, you’ll notice that this step overlaps with the previous one; because the first change involved a long-running DB script, the DBA was able to make the modifications on the second database, while she waited for the long-running script to finish. There were a number of things happening at the same time, but I trusted the DBA to know how much she could do at once, and how much had to be done sequentially.

1:50–2:10 (20 minutes): We brought back one of the application servers for Application 1, and deployed the new version of the application.

2:10–2:20 (10 minutes): One more reboot, to be safe, after the deployment of Application 1.

2:20–2:25 (5 minutes): We did our Sanity Test of Application 1, and everything looked good.

2:25–2:35 (10 minutes): We brought back up the application servers for Application 2, and deployed the new version of the application.

2:35–3:10 (35 minutes): We did our first round of Sanity Tests for Application 2. Unfortunately, one of the back-end systems—one of the ones that was supposed to be up and running—was down, because of some emergency maintenance. (Replacement of a network card, or something.) Nobody bothered to inform us that this was happening, so we were taken by surprise, when we ran our test and it didn’t work. It was decided to Sanity Test as much as we could, ignoring that part of the application, and then do the Landing Tests (still ignoring that part of the application.) When it came back up, we’d revisit Sanity Testing, to verify the functionality.

3:10–5:30 (2 hours and 20 minutes): We did our Landing Testing, for the business to verify that the functionality is up and running. Because of the back-end system’s outage, they ended up Landing Testing some functionality before the Sanity Test was done. Issues discovered:

Character encoding issues with one back-end system that we connect to; French characters were not showing up properly in the UI.
A web app that we link to was down, during the deployment. (Not a big deal.)
We deployed the wrong version of Application 1, and had to re-deploy it. When it was redeployed, it was fine.

At the end of the day, we were only left with the issue regarding French characters. It was decided that the application could be left in production as-is, and we’d have to look into a fix later on, when we could look more closely at it. (It seems to be related to character encoding issues with some XML being sent between applications. If you’re interested in character encoding issues in XML, I can recommend a great book, with some very handsome faces on the cover…)

Deployment: December 23, 2007

2007-12-23T02:07:00.001-05:00

The last deployment was a success. Well… sort of. We added in some new functionality which was pulling additional information from a back-end system. Unfortunately, that system wasn’t able to handle the extra load. And, because of the Christmas season, all of the systems were being taxed more heavily than usual, so there was a danger that this extra load would be enough to start causing crashes.

So it was decided to do an “emergency deployment,” and remove the functionality. The back-end system would be going through a change, the month after, which would make it able to handle this load, so the next version of my app would have the functionality reinstated.

It sucks that we had to do a deployment on the weekend before Christmas—when I was supposed to be on holidays—but that’s the way it goes, sometimes.

1:30AM: We all logged onto the conference bridge, and confirmed we were ready to go.

1:35–1:40: There were no database changes, this time around, so all we had to do was redeploy the application itself. We did so.

1:40–1:50: The application was back up, and we did a quick Sanity Test, to ensure it was working. The Sanity Tests passed.

1:50–2:05: The client did their Landing Tests. Again, the tests passed.

Deployment: December 2, 2007

2007-12-02T22:41:00.001-05:00

Finally. We finally got this thing deployed. After all of the false starts and number of times the release was deferred, it seems anti-climactic to have such a short post for this release, but the fact is, when we finally got a chance to deploy this thing, it went without a hitch.

1:30AM: We all logged onto the conference bridge, and confirmed we were ready to go.

1:35: We shut down the two applications that we had to deploy, for this release—the “front-end” app and the “back-end” app.

1:40: We backed up the database for the front-end app, and began the deployment of the back-end app. I can’t stress enough the importance of having a good, solid deployment plan, so that you can execute tasks in parallel like this, and not worry about losing track of who’s doing what!

1:45: The backup finished for the database, so our Database Analyst (DBA) began executing the new database scripts.

1:45: As the DBA executed the DB scripts, the back-end app was taking a bit longer than expected to come back up.

1:50: The back-end app came back up, and the DBA finished executing the scripts. We did our Sanity Test for the back-end app.

2:00: Sanity tests for the back-end app passed. We now began the deployment of the front-end app. (Because it depends on the back-end app, we had to ensure that the back-end app was up and running properly, before bothering to deploy the new version of the front-end app.)

2:05: The front-end app finished deploying, and we began our Sanity testing. At this point, we were about an hour ahead of schedule.

2:20: Sanity testing finished. We now got the clients to begin their Landing tests. We actually had to call some people, and get them to join the bridge early, since we were still ahead of schedule.

2:20–5:00: We performed Landing tests. We turned up two defects, but they were deemed minor enough that we could leave the release in, until a fix could be found.

Deployment: December 1, 2007

2007-12-01T15:07:00.001-05:00

The investigations into the back-end system have completed, and they believe the problems were caused by a problem with the hardware load balancer, for the back-end system. They’re making the change on the morning of December 1^st, which means that we’re being deferred yet again.

Assuming that all goes well with the changes to the load balancer, we’ll go in Saturday night/Sunday morning, meaning December 2^nd. We’ll have a go/no-go call at 5:00 PM Saturday afternoon, to make the decision.

And just to make everything even more fun, the email servers were down all day Friday, so updates couldn’t be sent via email. We were all waiting around to see what would happen, but nobody was able to send updates.

I’m almost afraid to ask what else can go wrong with this release.

“Load Balancer” Defined/Explained

2007-12-01T15:04:00.001-05:00

For high-availability systems, we usually want to cluster our servers. That is, instead of having one, very powerful server, we might want to split the processing between two or more servers. Requests can be processed by any of the servers in the cluster. This way, if any of the servers crashes, the other servers can handle the load, until the broken server is fixed.

However, most client applications can’t deal with a cluster; they need one place to go to, to get requests processed. So in order to enable clustering, there usually needs to be a load balancer put in place. The client applications only know the address/location of the load balancer, and the load balancer takes care of forwarding those requests to the servers in the cluster.

Depending on your needs, you may use a software load balancer, or a hardware load balancer. A software load balancer is simply a program running on an existing server, whereas a hardware load balancer is a dedicated networking device, which does nothing but balance traffic between different servers.

Deployment: November 30, 2007

2007-12-01T14:37:00.001-05:00

The investigations into the back-end system’s crash were inconclusive. Still a no-go for our deployment. Again, maybe it’ll happen Friday night/Saturday morning, but otherwise, it’ll be Saturday night/Sunday morning.

Deployment: November 29, 2007

2007-12-01T14:35:00.000-05:00

The back-end system were were depending on deployed successfully on Tuesday morning, so we were scheduled to go in Wednesday night/Thursday morning. Everything was set, and we’d had all of our go/no-go meetings.

Unfortunately, the same back-end system crashed Wednesday afternoon. We had to cancel, pending investigation into what caused the crash.

Assuming that all went well, we’d go in the next day.

“Source Code Repository” Defined/Explained

2007-11-23T13:30:00.000-05:00

When developing software, the source code is your most valuable asset. A Source Code Repository is a special database where the source code is stored.

Source Code Repositories—also called version control software—usually have the following types of features:

The ability to check out and check in code. When you check code out, it means that you’re locking that piece of code, so that other developers know not to work on it (or can’t work on it) until you’re done. When you’re finished with it, you check it back in. (Sometimes repositories are set to allow multiple people to check a file out at the same time; in this case, the changes have to be merged together, as explained in the bullet below.)
The ability to compare different versions of a file, and merge versions together. Comparing allows you to see what has changed, between the code you have on your computer and the code that’s checked into the repository. Merging allows you to take changes from multiple versions of a file, and combine them all together into one final version of the file.
The ability to label a particular version of the code. (Different repository products have different naming conventions for this; they don’t all call it a “label”.) Normally, the software you’re developing will go through multiple iterations; you’ll have version 1.0 of the software, and then 1.1, or maybe 2.0, etc. With each version of the software the code base changes. However, if you label your code, from time to time, you can always go back and get the code as it was at any particular point in time. e.g. “get me the code as it was when I labelled it ‘1.0’.”

There are a number of vendors who provide source code repositories. Some of the ones you’ll see most commonly are CVS, Subversion (SVN), and, in the Microsoft world, Visual SourceSafe.

Deployment: November 2007 (almost)

2007-11-23T09:49:00.001-05:00

This was going to be an unusual deployment, because it was scheduled for a Wednesday night, instead of a Saturday night. This worried me, because I was worried about having to work on Thursday, and I was coming down with a cold.

It was also going to be a long deployment; there is another system we depend on, that had to deploy along with us, but their deployments are much longer than ours. So we would have had to shut down our system, and then wait for a couple of hours, until they were done, and then continued on with ours.

As luck would have it, though, their deployment was deferred. Their source code repository crashed, and they spent days trying to recover from it. So we’re rescheduling for this coming week. And the good news keeps getting better: we’re hoping to deploy the other application on Monday night, and my application on Wednesday night—so we won’t have such a long deployment.

Deployment: September 2007 (Take Two)

2007-09-24T09:30:00.001-04:00

This was a second attempt on the deployment written about earlier. Since we were convinced we’d fixed the problems discovered during the first deployment, we decided to tackle this attempt exactly the same way.

Of course, there was one major difference, this time: I decided to take this one from home. On the previous attempt, they’d shut down the power in the building, forcing me to come home; this time, they were going to shut down the phones and the network. But luckily, I knew ahead of time, this time, so I was able to plan ahead to come home, instead of being forced halfway through.

10:00PM: We’re scheduled to start at 10:00PM, but there’s an accident on the 401, which prevents me from getting home in time for the beginning of the deployment. Luckily, I really only have to log on for a couple of minutes, verify that everything is going smoothly, and then log back off until midnight. So I log onto the conference bridge from my car, on my cell. The “database split team” begins their work—at this point, they’re backing up the database, in case of rollback—and I log back off again. I have just walked into my house, at this point.

10:00–12:00: I start watching another movie, to kill the time. The Spy Who Loved Me, this time. (Still on my 007 kick.) I also make some tapioca pudding, since I’m home, and spend some time preparing my “work environment”—setting up a phone with a long cable (since I don’t know how long my cordless phone batteries will last), getting a couple of cordless phones handy (since I don’t like my “corded” phone), and preparing some things to drink.

12:00AM: I log back on. The backup was completed successfully. We shut down the Application Servers for “Application 1”, so that they can proceed with the database split. And then I log back off the bridge, since there won’t be any further activity until 12:45. (For a reminder of what “Application 1” and “Application 2” are, see the post from the first attempt.)

12:05–12:45: More movie.

12:45: I log back onto the bridge.

12:45–12:55: We sit around on the bridge, wondering where in the world everyone is. We then decide to proceed without them, for the time being—the clock is ticking, after all.

12:55: We shut down the App Servers for “Application 2”.

1:00–1:01: We back up the database, for “Application 2”.

1:01: After the quickest backup in history, we begin the database changes for “Application 2”. (We did double check, of course, to make sure the backup was successful; when a backup is that quick, you have to wonder if it really backed up at all…)

1:05: We’re informed that half of the members of the client team won’t be showing up. Yes, you read that right: They’re just not coming. (Since we’d already done it once, I guess they got bored with the whole thing…) We’re told, of course, that we can call them, if anything goes wrong. (How magnanimous.)

1:10: The database changes are finished. We bring the App Servers for “Application 2” back online, so that the deployment of that application can begin.

1:10–1:35: We deploy “Application 2”.

1:35–1:40: We conduct our Sanity Test. Our testing is positive, which means that we’ve fixed the first issue we had problems with, on the last deployment. (Phew!) The client also jumps on the application, to start testing, but I give him a verbal slap on the wrist for it—I’d prefer us to finish our testing, before handing it over to the client. After my testing is done—which only takes a couple of minutes anyway—I give the client the go ahead to do his testing.

1:40: At this point, we’re ready to begin the deployment for “Application 1”, but, fortunately and unfortunately, we’re about an hour ahead of schedule. It’s fortunate because it means there is a chance of getting to bed early; it’s unfortunate because the people we need for the next phase of the deployment aren’t on the bridge—in fact, they’re probably sleeping, since they aren’t expecting to be needed, yet.

1:40–1:45: We call them on their home numbers, but they sleep through the calls. This is a fairly normal event, for late-night deployments; the human body is used to being asleep, at this time. So we take it in stride, and simply keep trying, maintaining our good humour. We finally get hold of them, and they join the bridge.

1:45–2:05: We deploy “Application 1”.

2:05–2:25: We do our Sanity Testing. This is a bit longer than we usually take, for this particular application—we can usually run through it in 10–15 minutes—but I think some of us are being extra careful. The good news, though, is that everything is working fine—meaning that the second problem we’d had, last week, is also fixed.

2:25–3:40: The client does their Landing Testing, and everything goes well. This is always the nerve-wracking part for me; there is so much chatter on the bridge, and every time someone asks a question—“Is it supposed to work like this?” or “How come this page is taking so long to load?”—I get nervous that we’ll have a bug that’s a “show stopper”. (In other words, a bug that’s serious enough to force us to back out the application.) I guess this is where I earn part of my salary, though; being able to think on my feet in the wee hours of the morning, answering questions, and deciding when a problem is serious, and when it is just a transient blip. (e.g. a link in the application that appears not to work, but it turns out it’s because the link is pointing to another application that’s also deploying right now.) You don’t want to be wrong, and assume something is transient if it’s not, or else you’ll spend the next week fielding calls from the Help Desk, with your cell phone glued to your ear. On this morning, though, we don’t have any bugs, serious or otherwise.

And, at 3:45AM, we have another successful deployment under our belts.

Deployment: September 2007

2007-09-19T14:06:00.001-04:00

This was a more complex deployment than usual. There were three major systems involved:

One of the back-end databases we talk to is being split into two databases, for performance reasons.
The “main” application, which is web-based, was being upgraded
A second back-end application was also being upgraded

Something like this:

(where, as always, I’ve greatly over-simplified this diagram)

I was worried about the database split; in our test environments, we had a lot of problems with it, because it’s a very complex thing to do; we kept encountering various permissions that needed to be re-created, and that type of thing, so I was worried that there would be additional permissions we hadn’t thought of before, that we wouldn’t discover until we deployed in production. In addition, the work for this split was being done by a separate team, whereas my own team was responsible for the upgrades to “Application 1” and “Application 2”. Whenever more than one team is involved in a deployment, coordination becomes an issue. Or rather, a potential issue—I shouldn’t be so pessimistic.

Because of the database split, which takes a long time to complete, we had special permission from the call centre to extend our outage, and start the deployment earlier than usual (12:00AM instead of 1:30).

Here’s how it went down:

10:00PM: I logged onto the conference bridge. The “database split team” was beginning their work at this time, to do some backups (in case of failure, later on). I simply logged on the bridge, verified they were good to go, and logged back off again.

10:00–12:00: To kill time until my part of the deployment was scheduled to start, I watched a movie on my laptop. The Man With the Golden Gun, in case you’re interested—I’ve been on a James Bond kick, lately.

12:00: I logged back on the bridge. Everything was still going smoothly; the database backups went quicker than they’d been anticipating, so they were just waiting for us to shut down our application, so that they could move on to their next piece.

12:00–12:05: We shut down our application.

12:05: The “database split team” began their next phase of the deployment, which was the actual work of splitting the databases into two. (You’ll note that I’m purposely not giving any details about this…) I wasn’t involved in this piece, so I had another hour or so to kill.

12:05–1:30: I finished watching my movie, and farted around doing some other stuff. Probably playing with Ubuntu.

1:30AM: I logged back onto the bridge. Again, everything was going smoothly; they’d completed their database work, and were ready for us to move forward.

1:30–2:00: We completed the deployment for “Application 2”, including the Sanity Test, with some minor issues. Or so we thought. Testing wouldn’t be complete, however, until “Application 1” could be tested, since “Application 1” is dependent on “Application 2”.

2:00: The power went off, in the building. (Let me repeat that: The power went off, in the building.) My laptop kept going, but my external monitor went off. I wasn’t overly worried, since they have backup power generators; I figured that the lights would probably come back on soon. And the networking infrastructure must be powered by the generators, because my network connections were still fine.

2:15: Someone went down to the security desk, and found out that this was a scheduled power outage. (“We communicated it to all of the appropriate channels…”) The power was scheduled to stay off until 5:00.

2:30–3:00: I knew that my laptop batteries wouldn’t last until 5:00, so I made the decision to drive home, and continue the deployment from there. (Assuming that VPN connectivity would be up and running…) During the 30 minute drive home, I stayed on the conference bridge, on my cell phone.

2:30–3:00: While I drove home, we completed the deployment for “Application 1”. (When I say “we”, I mean the people who did the actual work; luckily, I personally am not the one doing the work of a deployment.) Sanity Testing indicated that everything was working—except for connectivity to “Application 2”.

3:00–6:00: We—the technical team—continued troubleshooting the connectivity issue between “Application 1” and “Application 2”, while we let the client begin their Landing Test. (That is, they could test everything except the pieces of “Application 1” that require connectivity to “Application 2”.) Their testing confirmed our Sanity Test results; everything worked except for the connectivity between the two systems.

6:00: At 6:00, we had to make a decision: We needed something up, by 7:00AM, so we either had to be confident that we could fix the system(s) within the next hour, or make the decision to roll back, since a backout takes about an hour. But then the client stepped in and granted us another hour—meaning that we could stay down until 8:00—to continue testing.

6:00–6:50: We continued troubleshooting, until… we eventually ran out of ideas. Around this time, there was talk of extending our window for another hour, but we decided not to bother; we just didn’t have anything else we could think of to test. So we reluctantly made the decision to roll back.

7:00–8:15AM: We rolled each of the three systems back. (That is, we “re-joined” the two databases together, and rolled “Application 1” and “Application 2” back to their pre-deployment states.) It took a little longer than we’d anticipated, so we were 15 minutes late getting back up and running, but we were so disappointed at having had to roll back in the first place that the 15 minute delay was the least of our worries.

So, as it turns out, the database split, which I’d been worried about, went very smoothly. Frankly, I hadn’t been expecting any problems with “Application 2”, which turned out to be a problem for us.

As I write this, the deployment has been re-scheduled to be re-attempted this weekend (a week after the original attempt). We’ve fixed the issue which prevented “Application 1” and “Application 2” from talking to each other, and tested the changes, and are now confident that we’ll be able to get it in, this time.

There was talk of splitting the release into two deployments; one for the “database split”, and one for “Application 1” and “Application 2”. But it was eventually decided that

It would be too much work to re-jig “Application 1” to work with the database split, without incorporating all of the other changes that were supposed to be released as part of this deployment
We were confident enough that we’d solved our issues that we figured it was worth the risk to do it all in one shot, again

So there will be another post here, probably next week, to outline how the second attempt goes, for this deployment.

“Deployment” Defined/Explained

2007-09-18T14:45:00.001-04:00

I guess I should define the term deployment. (Unfortunately, I couldn’t find a definition in the Jargon File, so I’ll have to tackle it myself.) I’m a software developer, and I deal mainly in web-based business systems. (In other words, web sites used by a company, internally, to do its work. Most of the stuff that I do isn’t “public” or customer-facing.)

For example, I’m currently working on a system for a call centre; it’s a website that brings up customer information, and allows you to perform transactions on behalf of that customer—perform billing transactions, do technical troubleshooting, etc. etc.

Because I use web-based technologies, for my applications, the structure is usually something like this:

The users’ browsers connect to the web server, where the application resides, and the application, in turn, accesses other back-end systems or databases. Actually, that diagram is simplified; the structure is usually a bit more like this:

(Even this diagram is simplified, but of course, I can’t give out technical details about the actual systems I’m working on.)

Usually, servers are clustered together, to help handle the volume of calls to the application, and to provide redundancy—that is, if one server goes down, the other server handles the traffic until the broken server can be repaired.

When there is a new version of the application, to introduce new features, or fix bugs, it has to be installed on the Application Servers. (Sometimes changes have to be made to the Web Servers, too.) There is also usually work to be done on one or more of the back-end databases that the application uses for its data. Since the systems I work on are web-based, there is typically nothing to install on any users’ workstations; they use their web browsers to access the application, so as soon as the application is updated, they start seeing it immediately.

A deployment is when we un-install the current version of the application, on those servers, and re-install a new version. Typically, for these types of applications, a deployment is a tricky process, involving many steps, that have to be performed in a proper sequence. Certain parts of the deployment can’t be completed until other things have completed successfully. It needs to be planned out in advance, in detail, and, where I work, we typically create a deployment plan, where we detail these steps.

And, because a deployment is so complex, a deployment plan needs to include a backout, or rollback, strategy. If you start a deployment, and for whatever reason it doesn’t work, you need to be able to go back to the version of the application that had been working, before you started the deployment.

As an example, here’s what a typical deployment is like for me, these days:

Because the application I’m currently working on is for a call centre, which is up and running 24/7, our deployments have to be scheduled for times when the call centre is the least busy; for us, this means deployments happen Saturday nights/Sunday mornings, beginning at 1:30AM. During these wee hours on a Sunday morning, the call centre isn’t too busy, so the few people who are working can get by without our application for a while.
- But we do have to get the application back up and running by 7:00AM Sunday morning, when they start to get a bit more busy. So our deployment window is 1:30–7:00 AM.
Our first step, when doing a deployment, is to shut the application down, so that the users can’t use it while we’re working.
- There are times, during the deployment, when the application will be in an “inconsistent state”—meaning that some pieces are up and running, and others are not. It can cause problems when people are using the application in such a state.
- In our specific case, shutting down the application means shutting down the Application Servers. Depending on the technologies used, this can be quick, or it can take a few minutes.
- For other applications, this might be accomplished by shutting down the load balancers.
  
  Or, better yet, one might redirect the load balancer to another version of the application, which simply says that the application is down.
Once we have shut the application down, we would deploy any appropriate changes to back-end systems; in many cases, this might be changes to a database, but in more complex deployments, we may have to sit and wait, at this point, while another back-end system performs their own, complex, deployment.
Once all of the back-end systems are back up and running—and tested, if necessary—we bring the Application Servers back up, and begin deploying the “main” application code.
Once the application is back up, we do a sanity test. This means that my team, who developed the application, does a quick test, to make sure that it’s properly back up and running.
Once the sanity test is done, we would hand the application over to our client, who would do a landing test. This is a more thorough test, where they specifically test each new piece of functionality in the application, as well as regression testing existing parts of the application, to make sure it still works.
- Typically, of course, I would also be performing these tests on my own. If there are issues, I want to know as soon as possible.
If everything goes well, this is the end of the deployment. Once all of the testing has passed successfully, we can go home and go to bed.
If things go wrong, however—if some of the testing fails—we then have to decide how to proceed. Normally, we will do some troubleshooting on the spot; if it’s an issue we can fix immediately, we do so. Sometimes we can, and the deployment still ends successfully.
- In some cases, even when we troubleshoot the issue, we’re not able to solve the problem. There can be numerous reasons for this; perhaps it is a complex problem, that requires research, or perhaps we just run out of time, before we have to have the application back up and running for the users.
- In cases where we’re not able to solve the problem, we eventually have to make a decision to back our changes out, and put the old version of the application back in place.

And this is how a deployment works for the application I’m currently working on. Deployments for other applications could be much simpler, or much more complex.

Deployment Blog

2007-09-18T12:32:00.001-04:00

I’ve threatened to do it, in the past, and I’m finally making good on my threat: I’ve started a “Deployment Blog”. (It’s… er… it’s the blog you’re reading right now.) I do a lot of deployments—about one a month, these days—so it’s time I started blogging about them. As with the many other blogs I’ve started, I’m sure nobody will find this blog interesting but me. Only a fellow nerd/geek could possibly find this interesting, and even then, why would they want to read about my deployments? They’ve got their own to worry about.

Still, I’ve started it, and I’ll stick with it. For a while.

For those who don’t read my main blog, here are some links to posts from the past, when I mentioned previous deployments:

An intellgent, well-thought-out post covering the state of serna’s life at 11:30 on a Sunday night—one of those posts where the title sucks you in, only to be disappointed by the post itself
The Long, Long Weekend—the first deployment I really wrote about
My Deployment—a deployment post which is more detailed than usual
Deployment—Let’s try that again, shall we? and Deployment was successful. You can relax now.—this was a second try at the previous deployment
Deployments—a general post about what my Saturday night deployments are like
Another Deployment—a post where I had the idea for starting a blog about deployments, and then came to the conclusion that I never would. Shows how much I know.

So from now on, when I have a deployment, I’ll write about it here. (Also, I will put up a post which describes what a deployment is, for those who aren’t familiar with the term.)