Archive | software development RSS for this section

Looking forward to learning Scala

Some people learn one programming language and stick with it forever.

I’m not one of them. I really enjoy learning new programming languages. When you learn different languages you learn different ways of approaching programming problems. When a new language is conceived, the language designer has captured techniques and methodologies that have worked well for them and are, in some sense, sharing them with you.

Each programming language promotes a particular perspective or “world view”. Each language is capable of expressing concepts in different ways. In fact, some concepts may not even be expressible in a given language. Depending on the type of problem you’re trying to solve, one language may be better than another in the sense that it makes easy what you need to solve. It’s a wonderful opportunity to broaden your perspective on problem solving and analysis.

I haven’t had the time to learn a new language in a while, so I’m looking forward to learning Scala. The first I heard of it was when Dave Thomas blogged that Twitter should move from Ruby to Scala to help it scale better (they did). We’ll see if we should move any if Frontier over to Scala (in particular, our scheduling engine). Maybe we’ll write our next product in Scala. Who knows. In any case, it’ll be fun to learn something new. I’ll let you know how it turns out.

Multi-level Vision

I remember watching a TED talk a while back where the speaker described how to represent data from different scales at the same time. The example had a picture of a house superimposed onto a street map superimposed onto a city map superimposed onto a country map, etc. It was hard to see all of this information at once, but occasionally you’d get flashes of how everything fit together.

Software architects have to do this all the time

One of the most important architectural skills is seeing through all of the layers of a piece of software at the same time. I don’t think you can be an effective architect without this. An architect has to know how a change in one layer will affect all the layers in a system. Often, something that makes sense in one layer is a terrible idea in another.

What makes this even more challenging is that each of these layers may involve different technologies. Not only must an architect have the vision to see potential problems, but they must do so in a way that holds multiple paradigms and structures in place simultaneously while threading multiple paths through all the layers. Occasionally, while doing this, an architect will see everything fall into place and have a momentary, deep insight into the entire system. The challenge then becomes capturing something coherent enough to act on before the insight vanishes.

It requires lots of practice and experience to develop this kind of multi-level vision. I don’t think you can learn how to do this casually—it’s something that you have to work at for a long time. How long? I think the current rule of thumb is for about 10,000 hours.

Pulling Threads

Son #2 was wearing a shirt that was embroidered with “Istanbul!” (my parents bought the shirt for him on one of their vacations). The thread making up the “I” in Istanbul came undone and he was pulling on it, causing the “I” to disappear. We told him to stop because he would start erasing the whole word.

It’s the same in software

One of the software teams I used to manage talked about the danger of “pulling threads”. In any piece of software that’s been around long enough, there are bound to be parts that are just a little bit off. There might be an extra parameter that you shouldn’t need, or there might be some extraneous code somewhere. You might think to yourself, “Well, why don’t I just fix this right now since I’m looking at it?” That’s a good attitude, but sometimes pulling on these “loose threads” will cause the entire module to unravel. Often, you may find that the thread runs through several modules and fixing this will involve touching a bunch of files.

The danger in pulling on threads in your spare time is that you won’t have enough spare time to finish. Inevitably, you’ll have a half-done pile of work that isn’t fit to check in and that doesn’t provide any visible value.

If you want to pull on threads, tie them to features

The best way to pull on threads is to associate them with features. If you can let features drive things, you’ll pull on the threads that matter. You’ll end up with cleaner code that does something new. It’s really hard to justify work that only (potentially) reduces maintenance.

Pre-emptive thread pulling

Sometimes you can prevent future thread pulling with a little thought today. One of the features that people have asked for in my company’s product is the ability to log notes against effort entries. On the surface, this is easy to implement. You just add a notes field to an applied effort entry and there you go. However, I’ve been hesitant to do this because I know this would create a thread that would have to be pulled at some point.

How do I know this? It’s because I know the data architecture of Frontier, and I know this isn’t the right way to solve this problem here. The right way involves architectural changes. It’s a little more involved, but the result will be something much cleaner and much more powerful. So that’s what I’ve spent the past week doing–pulling this thread before it exists.

Don’t do too much pre-emptive thread pulling

Too much pre-emptive thread pulling, though, leads to overdesigned software. You’ll never get it done. You’ll spend too much time analyzing cases and sub-cases, and inevitably, one of the cases you hadn’t considered will be the one that turns out to be important…and many of the cases you did consider will turn out to be threads that will need to be pulled some day 🙂

A Place to Hang Your Hat

A few years back, I posed this question to my software team at the time: What is software architecture? People answered this in a variety of ways. The answers ranged from the very dry “software architecture is the set of objects in a system and the relationships between them” to the very vague “software architecture is the shape of the software”. I was disappointed that we weren’t able to come up with anything better than that. I had hoped that we’d find a principle that could guide our day-to-day work.

Maybe I was asking the wrong question

Thinking back on this now, I think a better question would have been What is good software architecture? I imagine that we’d come up with things like “it runs efficiently” or “it minimizes the amount of required code”. I think this is part of it, but, it doesn’t really capture the essence of good software architecture.

My Answer Is…

I suppose you can already guess: “Good software architecture has a place to hang your hat”. When you’re working in software that’s well architected, you always know where things go. There’s a place for everything, even when it doesn’t exist yet. When software is well architected, you can even move the place where you hang hats, and everything works fine. In fact, if you need to move things around, you’ll often fix things that weren’t even problems yet. The software stays tidy. You don’t have a bunch of stuff lying around or on top of things or sticking out of places—there’s a place for everything.

When software is poorly architected, you’re not sure where things go. In fact, no one’s really sure where things go. People start doing the same things in different ways. Everything mostly works, but you’re scared to change anything. When you need to do something new, it can take a long time, because everything is a mess. You’ll see hats all over the place.

Getting to Good Software Architecture

There’s something subjective about good software architecture, something almost autocratic. Because there are so many ways to do things in software, there are so many decisions to make. It goes combinatorial really fast. Unless you find some way to narrow the field of choices, you’ll waste a lot of time arguing about options, and your group will spin its wheels.

The best way to winnow down options and get good software architecture is to have a good software architect, someone who’s worn a lot of hats, and who’s tried hanging them in lots of places. Someone who’s opinionated, but not overly so. Ideally, you build a good team around a good architect. The best way to do this is to have a good software manager, someone who’s worn a lot of hats, and who’s tried hanging them in lots of places 🙂

Debugging with the Scientific Method

One of the best techniques I’ve run across for debugging software is the Scientific Method.

The Scientific Method for Debugging

It’s actually pretty simple, but I’m always amazed at how quickly it can bring you to the root of a problem. Basically, it goes like this:

  1. Enumerate the evidence you’ve seen regarding the issue.
  2. Before changing any code, take a guess at what might be happening. It doesn’t have to be a good guess—it just has to be something you can test. This is called your hypothesis.
  3. Come up with a way to test your hypothesis. You want something that can give you a “yes” or “no”.
  4. If your hypothesis is correct, you’ve found the issue. Fix it and move on. If your hypothesis is incorrect, log this fact as the next piece of evidence and then come up with a new hypothesis based on all of the evidence, including this.

For a typical issue, it takes me 2 to 3 hypotheses to get to the root cause. For tougher issues (especially those that involve recursion or graphs), it takes me 5 to 7. In either case, it’s a great way to prune off areas to search. It is by far the most effective way of debugging that I’ve ever found. I’ve often noticed that issues typically requiring a day (or more) of debugging can be dispatched in an hour or so using this technique.

Modification using TDD

One powerful modification is using test-driven development with the Scientific Method. This is particularly useful when it takes many steps to duplicate the issue using the application. It’s even more useful when duplicating the issue requires you to delete and re-add data.

Here, the first step is to duplicate the issue in an automated test. This is the hard part, but the part that will save you the most time. Once you have this going, the tests of your hypotheses will be the same as your TDD tests. You add tests to check your hypotheses until you’ve proven your hypothesis correct. A bonus is that once you’re done, you’ll have some regression tests in place that actually exercise something that broke.

Believe it or not, it’s kind of fun to debug this way. You feel like a scientist wearing a white lab coat and jotting notes into a log book (I assume you think that’s fun 🙂 ). You set up experiments to see if you can trap bugs. They can be surprisingly evasive, but the scientific method always catches them in the end. Bwahahaha!