“This is just a standard management technique that has been used by personnel supervisors since the days of.. Ho Lu, grand emperor of the Wu dynasty.”

broderic on September 19, 2005

This is assignment 1 due during week 2, and it is related to this post for school. Download it from here in order to print it out.

In the early 1990s, a piece of telephony software written by the DSC Communications Corp. caused some phone outages across the United States. The root cause of the problem was traced back to a typing mistake. The bug occurred in three lines of a million line program. At the time, Frank Perpiglia, Vice President for technology and poduct development at DSC had said that the intended change to the software was very minor, and that it had been determined that the usual rounds of testing of the software was not needed. In hindsight, that was a bad idea.

Yet, to put the blame on insufficient testing is to ignore the other factors which contributed to the software bug. In fact, I would say that the problem really resides in the policies and procedures that allowed the developers to release their software with less than adequate testing. A symptom of this lack of quality-focused policies can be found in the cavilier comment from Perpiglia, “We had a small modification to make a small change. We felt that the change itself did not require three to four months of testing.” It was sheer folly to accept the risk of not testing the change.

First, let’s focus on DSC’s software configuration management policy, if they had one. A change was going to take place on their software. Was there a need for the change? Was the fix to address a bug or was it relatively minor? From Perpiglia’s statement, it seems that DSC deemed the fix to be a minor, inoccuous change. It probably was also somewhat necessary. I can’t believe that a simple patch of less than several lines of code would be released at the whims of the company. From this viewpoint, did software configuration management accept the necessity to change the software and at whose discretion?

Secondly, let’s take a look at this issue from the software developers position in relation to their processes and procedures. To make the change requires at least a minimal set of informal tests. Was there a procedure for code/peer review? Even in an informal setting with several peers and domain experts maybe the flaw could’ve been spotted. It literally was a typo that was the problem. At the least, was there a procedure to conduct unit tests on the change? Perhaps a unit test could’ve caught the flaw early exposing the unit under testing as not performing correctly. A set of procedures for the developers would’ve been helpful in spotting the problem.

Finally, focus should fall on the system test group. Risk was taken to not test any of the software, because running a full set of qualification tests was time consuming. Still, could they have developed a reduced set of tests to ensure that the change would not produce any bugs? They knew the functional area affected, they could’ve run a smaller set of tests specifically focused on that small requirements which were affected. Did they have sufficient tests for that functional area or was the bug the type where the code changes in one part of the program and wholly unrelated piece fails? Sometimes the a small set of tests would not uncover the bug, and the full complement of qualification test is needed to discover it.

In summary, the problem with the DSC software could be traced back to problems with the policies and procedures that were in place. They were lacking, because it allowed them to assume the risk that an insignificant change to their software would not produce any anamolies. DSC neither planned for the change, reviewed the change, nor adequately tested the change. It was not just the lack of test time that did them in, but a lack of focus on policies and procedures that would’ve helped them focus on delivering a quality product.

“But yes, she did steal my high school sweetheart.”

broderic on September 17, 2005

Dreams. Man, I wish they could be real. But I guess that’s why they’re dreams.

Tossing and turning this morning. I had one that was so life-like it was not even funny, and like the majority of them I wish it were real.

It starts in a physics class in the University of Miami. I am taking it with the girl who’s become an icon to me, I. She is the first girl I have had a crush on, and probably the first girl who retruned the feelings back. She drives plenty of my feelings on what a girl should be like. She was smart, the smartest in grade school. She wore glasses. She was blonde, blue eyed. I was able to talk to her. We were friends throughout grade school. I often wonder where she is now. I think this dream comes from the depths of my psyche looking for her…

First day of class, I am amazed that after 20 years we end up in the same class. She does as well. I try to act cool and not think about the implications. Class is taught. I get the first homework assignment. I finish it up.

Next class is a movie presentation or at least looks like it will be one because the seats are like a movie theatre. She sits in front. I sit down next to her perusing my homework smiling at her. The professor comes in and gets ready to lecture. I., she turns my way and we converse. Small talk. Inane banter. Yet, out of the blue she says, “What happened between us?” She was referring to a point in grade school where I liked her, she knew I liked her, but I never had the balls to express that to her. Plus, she also had a feeling there, too. Like a moment, which passed, and we both never made the connection. Perhaps, I was in the friend zone…

“What happened between us?” I go cold at those words. They were why I sat next to her. I wanted to make the move, and yet, she says them. I don’t remember what I said but they were to the effect that I don’t know. She goes onto say that we had something. That there could’ve been a there there. Then she explains her end, “Kimmie, (one of her grade school girlfriends) said you weren’t interested in me. She said to move on.”

My heart breaks upon hearing this. The feelings I had for her come swelling to the surface. I am speechless. She did like me! At the time, I should’ve said something! It could’ve been…

The professor begins his lecture.

I say with anticipation, “Can we talk about this after class?” She nods. We listen to the lecture. The class poses and answers some questions. I look at her and she at me. I think of everything I want to say. I can’t wait for the class to end to find out the big reason why we never were.

I wake up wishing for the answer. It has been 20 years since I have seen I. I occasionalyl have dreams of her. I would like to know how she is doing. I wish that we would meet cute. I think that that too is a dream.

“Steven Seagal? That fat fraud is a disgrace to the martial arts.”

broderic on September 14, 2005

Transporter 2. Not as bad as it would’ve been. It was the best comedy of the fall. Unfortunately it wasn’t supposed to be. There were a lot of unbelievable things happening in this movie. And you’ve seen them all in the commercial. Except for the way he disarmed a bomb with a hook, a wooden bridge, and an Audi. If you saw Diamonds Are Forever you’ll know the stunt.

Jason Statham is being considered as the new Bond. That would not be such a good idea. He should be the new Seagal.

3 of 5 stars.

“Guess how many pieces of gum Beth chewed last year. “

broderic on September 14, 2005

In September school starts. This semester I am only taking one class, CS780 Software Reliability and Testing. I’ll be posting some notes about the class and some homework here. Join in the discussion to help me get an A.

First up is an assignment about DSC Communication Corp. and problems with telephony code.

Notes on the paper, “Testing Time Cut on Software Linked to Phone Outages.” Balt. Sun , Leslie Cauley:

  • They took the risk that they didn’t need to test the changes. What risk management did they have in place? What processes and procedures allowed them to accept the changes?

  • Would the changes have been caught in the first place? Did they have tests to catch this type of problem? Did they have regression testing? Would their test plan have caught this issue?

  • It was rather costly issue: 200 engineers on a bug hung!

  • How old is the system in use?

  • There was a mention of the regulation of the industry prevented them from acting quickly to identify the problem. Why is that an issue? Can’t the Bells ask for better testing from their suppliers? Insist on it in their contracts?

Links to more information:

  1. comp.risks digest forum post about the root cause.

  2. A Google Search for any information about this problem.

  3. More about the liability issues facing the company.

  4. More about the problem

If you have any help, please post it in comments.

“Sir, Siskel and Ebert are two people.”

broderic on September 11, 2005

The latest DVD that I bought was the full run of Undeclared. It’s another TV show on DVD that I like. I found it hilarious. Seth Rogan rocks. It is seventeen 30 minute episodes of hilarity.

Unfortunately, I can’t see the last three. The third DVD of the set doesn’t load on my DVD player or TiBook. I think the DVD is blank. I get error 13! How do you exchange these things anyhow? O well.

I recommend the show highly. It would’ve been the second TV show DVD that I watched all episodes of. Following in the grand tradition of News Radio.

“I wanted a house just like Xanadu, but without a dorky name.”

broderic on September 9, 2005

I am really getting lazy with my movie reviews. Here’s a quick roound up of the last few I saw.

40 Year Old Virgin – 3 of 5 stars. Could’ve been 20 minutes shorter seems to be the running complaint this summer. Don’t see it with your mom. Even if you’ve been laid it will make you feel like your the titular character.

The Constant Gardner – 3 of 5 stars. A challenging movie for the summer. First, the accents where somewhat hard to decipher that it made it hard to catch some of the plot points. Are they speaking in English?

Red Eye – 3 of 5 stars. Sweet 90 minute movie! Not enough of Rachel McAdams running. The script writes itself — See these people. They’ll be part of the plot soon. Yadda. Yadda. Yadda.