Bill Roper (billroper) wrote,
Bill Roper

Data Is Good

We have been flailing around and chasing an elusive bug in our software for a couple of months now. We've tried a number of things to fix it, declared victory, and then discovered that the bug isn't fixed at all, because we've not been sure at all how to cause things to fail. It's one of those cases where you try this and you try that and sometimes the system just falls ill, while other times it will just keep running.

Of course, one of the things that you do when chasing down these things is add more and more data to your logs in the hope of getting a critical piece of data that will explain everything. And over the weekend, the fellow who has been digging into this most recently said, "Hey, look at this."

So I did. And I replied, "I don't know where you're getting that data from, but the error message can't possibly be generated by the code that you are executing."

And after letting it roll around the back of my brain for a few hours, I realized that I had figured out the problem. There was a bug in the code that left old error messages attached to threads in the thread pool under certain circumstances; when one of those threads holding an old error message was dispatched, things started going wrong.

So, you see, once I had the critical bit of data, I was able to make the correct inference to fix the problem.

Of course, it turned out that there was a bit more to the problem than that, but it all related to similar behavior that was easily fixed once the data allowed me to come to the right conclusion.

It's all sort of fascinating. With incomplete data, you flail around and arrive at all sorts of incorrect conclusions, wasting time in the most amazing ways. When you finally have sufficient data -- that is to say, enough data to allow you to figure out what has happened -- you can get to the correct conclusion.

There is a lesson there somewhere.

When I have enough data, maybe I'll figure out exactly what the lesson is...
Tags: musings, work

  • Swept

    The weather was relatively cool today, so Gretchen and I went out and spent some time trying to get the garage into a state where it could be cleaned…

  • Everybody's Got a Recipe For Dumplings

    These things happen in our household. Gretchen and I were driving home from lunch and listening to Radio Margaritaville when our discussion turned…

  • Taking a Break

    I am taking a break. Not immediately, but I have scheduled some time off from work, because I have not taken more than a day or so off of work since…

  • Post a new comment


    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded