An Interview with Watts Humphrey, Part 28: Early TSP Trials and the Teradyne Team Launch
This interview was provided courtesy of the Computer History Museum.
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Early TSP Trials
Humphrey: As I said, we’d
gotten the PSP in place. It became clear that people really weren’t able to use
it. So I had to build something that ordinary folk-- not zealots like me --
could actually use on the job. So that, I decided, was a TSP -- a team software
process -- and I wanted to put something together. So starting in about January
’96 -- remember I talked about our process development process? I pulled out my
process development process, and I started to develop a TSP process
I got it together
during the spring and summer, and I was trying to figure out where to use it,
and the people at Embry-Riddle said, “Hey, let’s us do it.” So what they wanted
to do was to use it for a team of four or five of their graduate students who were
going to do a team project. They’d all been PSP trained, now, so they were
qualified.
I put together the
process and had it all defined and ready to go, but unfortunately, in
September, when the project was to start, I had to be in
Our processes, by the
way, have things called scripts. It’s what I call an operational process. The
process isn’t a big document thing that has paragraphs. It has steps in a
script which you follow. I’ve got copies or versions of them in my textbooks
and stuff. But it’s got a series of steps, and if you read what it says in the
steps -- let me pull up a couple here -- they’re very direct. They just tell
you what to do, step by step. So it’s not complicated at all. Let me pull up a
process: “Schedule planning template: Enter the following.” I mean, it starts
off with a purpose. “The purpose is to record the estimate and actual hours
expended by a calendar period, and relate the task plan value of the calendar
schedule, plan value earned, value stuff, calculated adjusted planned and
earned value.” And then, “General: Expand this template or use multiple pages
as needed. Complete it in conjunction with the task planning template.” Then I
go, “Enter your name, date, week number, when the project started. Enter a week
number, typically starting with one. For very small projects, it may be more
convenient to use days instead of weeks.” Then the next one.
So basically it goes
through -- this is the instructions for the template, and then the actual
process script starts with entry criteria, and then planning. “Step one: Produce
or obtain a requirement statement. Use the probe method to estimate total new
and changed lines of code, required time and the prediction interval. Complete
the size estimate template. Use the probe method to estimate the development
time required.” So it goes through steps, very straightforward, and we have
scripts like that for all the steps. The reason I had to do that is kind of
interesting.
And think about it
this way: when you tell somebody to produce a plan for a program, most
programmers won’t have the vaguest idea what you’re talking about. So they’ve
got to go off and sit down and figure out, “How do I make a plan?” They don’t
know. They have got to kind of figure it out. Even though it’s not very
complicated, they don’t know; they don’t have a process; they don’t know the
steps. So they’ve got to figure out step by step what to do, and then they do
it. So that’s an enormous waste of time. It’s fairly obvious. We know how to do
it, so we just give them a script and say, “Here, just do this.” So that’s what
we’ve done, and it works extremely well. So that’s what I sent to the folks at
Embry-Riddle. How do you do a TSP project, and that’s what I had done. When I
developed the process, I developed all the forms, the form instructions and the
scripts for every step of what they had to do.
It took me several hundred hours of work. I mean,
it’s a lot of work, and it is non-trivial. There’s a big, big pile of stuff. When
I got back from
And this is sort of
what you’d expect programmers to do. But they got a little bit along like that
and they realized they were totally lost. They had no idea where they were. And
that, by the way, is where most programmers are most of the time. They don’t
know where they are on the project and what to do next. So they basically said,
“Okay, we better go back.” And so they did. They basically said, “Okay, we are
going to follow the process.” So they did, and they did a marvelous job. They
went through and they produced a system. Unfortunately, it never got put in
production because one thing or another happened over there. It was a system to
manage the flight line at Embry-Riddle. They were going to develop a flight and
instructor scheduling system, and they put together the requirements. They went
through the design. They did the whole thing. But the process basically gave
them an anchor, so they knew where they were, what they should do next.
It was quite helpful,
but they had a problem getting started; their plan wasn’t very good; lots of
stuff. It was pretty obvious from that that they needed guidance when they were
starting the project. We couldn’t just send it to them and say, “Go start.” We decided
to call it a “launch” of a project. And we needed to have somebody to coach the
project. It occurred to me -- I talked about my wrestling coach -- really high
performance teams need coaching. So we decided we were going to have TSP
coaches. We originally thought of them as coaches for the launch, so they would
basically launch the project, and then the team would just go do it. So we went
and did a couple of industry teams. I think we did a couple up at Harris, up in
upstate
I kept running into
this, by the way, in organizations. You get halfway through and they get a
re-organization, and you get a new manager. The continuity in business is so
appalling that you literally can’t keep stuff going. In most of the places
where we worked -- a fairly high percentage -- you get something started and
then the sponsor moves on and something else happens, and it’s dead. It’s gone.
You’ve wasted an enormous amount of effort. And it happens all the time. It’s
just astounding. And it can’t be just us. It’s got to happen on everything
going on in industry. So it’s kind of frustrating. I don’t know if it’s just
the
The Teradyne Team Launch
But in any event, I
was the launch coach for a number of projects that we launched. I remember
doing one at Teradyne out in
And they were
basically trying to find line failures, and it was for the telcos
in
The opening meeting
is with senior management. We bring in marketing people who will describe what
the customer wants, the customer situation. And the reason for this is
interesting and that is that a lot of the decisions you make when you're
producing a system are tradeoffs. And they're tradeoffs of what you can do
technically and what the business can afford and what the customer wants. And
so the people who are really making those tradeoffs in very many cases are the
programmers. Nobody else is involved in enough of the detail to understand it. So
the programmers are making these very sophisticated business tradeoffs without
the vaguest idea of what's going on in the business.
So we start the
launch by bringing in management to give the team the business perspective and
bringing in the marketing or customer people. And we bring in customer people
if we can, who will tell them what they want and why. And then the programmers
have a perspective for making decisions. And what's interesting is during the launch
we open the launch with management and then we basically close the door and
have a meeting with the team. And I'll come back to that one on a later team
launch we had with Boeing, which was fascinating.
But in any event we
close the door, and the team goes off and puts together the plan. So we went
through the opening meeting with this team at Teradyne, and the management was
absolutely firm it had to have the product in nine months. I poked at that a
little bit. In the opening meeting the team is supposed to ask questions but
they practically never do. The team leader may ask a couple, but by and large
management says what they want and nobody says “boo.” So I asked a couple of
sort of quick questions. I didn't want to say too much because you don't want
to tell management that they're crazy and that sort of thing – at least not
until you have a plan.
The team doesn't know
enough to argue yet. So basically the team is in listening mode and the
management was really very firm. When the meeting broke I went out to the
washroom and then I joined them in the meeting room and they were in turmoil. They
were irate. You have heard of the storming phase? Well they were storming. They
were saying, "This is impossible. We can't possibly do it in nine months. This
is crazy. The last project took two years and it was a disaster and this is
more complicated than that."
And so I asked them I
said, "Whose date is the nine months?" They said, "It's their
date, management’s date." I said, "Okay. So what do you want to
do?" And they said, "We ought to go back and tell them it's
crazy." I said, "If you do what will happen?" They said,
"Well they'll beat us up and we'll, you know." they kind of mumbled
around and they finally concluded, "Yeah we'll end up that we will try if
you insist but it's a very tight date we don't think that we can make it."
I said, "Okay if
you do that now who owns the nine months?" They said, "Oh we
do." And I said, "Do you want to own the nine months?" And they
said, "No." I said, "Okay. So here's what you can do". I
said, "You've got to make a plan and do your utmost to make a plan that
will end in nine months. If you can't do that then you'll know why and you'll
have various alternatives you can say with some evidence that you can do the
following. But at least you'll be able to understand in some detail exactly
what it'll take to do the job and you'll have a foundation for debating the
subject with your management." So they bought it. So we went and made a
plan.
And the launch process
goes through starting there. We later added stuff to the launch, but we didn't
do all of this at the Teradyne because we kept learning as we went ahead. But
the process fundamentally is, we come out of the management meeting, you go
into a meeting on basically goals and roles. What is it we're trying to do and
how are we going to divvy up the work among team members? Now this wasn't
divvying up the project work; it's divvying up the management of the job.
And so the TSP
actually has role managers, and so we have a planning manager and a design
manager and a quality manager and a process manger and a test manager et
cetera. And these are people who have specific responsibilities for aspects of
the job. The test manager, for instance, doesn’t run all the testing. But the
test manager is the person, if there is any issue with testing, you give it to
the test manager, and he or she will take care of it. Make sure the test tools
are ready in time, that they're thinking about testing when they're doing
requirements work, et cetera. Same on design: what methods are we going to use?
Make sure everyone is using the same design approach and deciding what is the
notation we are going to use and what are our standards? The quality manager is
looking at the quality data with the process manager and deciding how to do
inspections and this sort of thing. So these are the jobs these role managers
do. And what's interesting is that on most projects today, no one does them. They're
all left for the project leader to handle. And the project leaders are handling
all these mechanics. They're trying to track the plan and this other stuff and
they don’t do it very well. They don’t have time.
And so fundamentally
most of the stuff that these role managers do doesn’t get done on a typical
project. And so the basic rule we use on the TSP is that team leader's job is
to make sure every definable task is given to somebody on the team. The team
leader's job is to do all the indefinable tasks. He or she handles the
motivation, the tracking, protecting the team from management. Staffing is a constant problem -- the minute
you launch a team, management wants to borrow somebody. And you've seen that
before.
Booch: Oh yes.
Humphrey: And so the team leader's job is
to protect the team to make sure issues get handled, to work on risks, and I
mean a whole bunch of stuff and motivation and measurement, evaluation, process
discipline, reporting. Team leaders are very busy people and they are the ones
who actually make the difference between whether the project succeeds or not --
the team leader and the coach. What's interesting -- you think about like a
ball team that is in the cellar, who do you get rid of to fix the team? You
don’t get rid of the players; you replace the coach or the manager. And we
don't normally have coaches in the software community. Coaches are extraordinarily
important.
This is one of the
issues I keep running into -- what's the difference between a team leader and a
coach? Why can't the team leaders be coaches? Well, we're probably going to
have to come to more manager-coaches just because coaches get laid off and team
leaders don't. We're running into that now in the in the economic squeeze. Unfortunately,
management doesn’t recognize the enormous value of coaches.
Booch: This reminds me of the story
that you told me earlier in your wrestling career -- how under the one Olympics
quality coach you guys performed very differently than with the subsequent
coach.
Humphrey: Exactly. Exactly.
And we don't see that in the software… We don't see it with development teams
at all. And I've not so far talked about my experience with my early teams in
engineering. When I was managing development teams I was really acting more
like a coach. If you remember, I was asking people what they were doing and
focusing on why you're doing that and this sort of thing. I wasn’t really
saying do this, do that, do that. We had a lot of things that we had to do, and
we had to put together a plan, but I was amazed at how motivating that was. People
love it and it worked like a dream. And it worked extremely well here in
Teradyne.
So to continue with
the Teradyne team: First they went through goals and roles and set that up, and
then we get into a strategy session where we figure out what's the process
we're going to use. What's the strategy -- well we do the strategy first. What's
the strategy for developing this product? And people aren't sure what the
strategy is, but there is an enormous range of strategies you can have as you
well know. And that is, you can build the whole thing in one big bang, you can
decide to build versions of it. You can use all kinds of cyclic processes, you
can prototype stuff. There are lots of different ways to build a system. And
you may not make every part following the same strategy; you may have several
strategies depending on what are the tough problems You
want to identify the real nuts that will have complex technical issues and get
them on the table early. And maybe you want to prototype them. All kinds of stuff. And so they work out the strategy first.
You have quite a bit of discussion on that. What is it? How are you going to do
this and why? And you think about the risks and what are the problems and
exposures you've got.
So we think through
alternatives -- you may want to just build it the way you did the last one. And
then once they lay out a strategy then we say, "Okay well now what's the
process you're going to use?" And notice we have all these unit processes
that they've got. They know how to do configuration management work, how you
write modulus and test them and all that sort of stuff. But then you have to
weave that together into an overall process that you can use for actually
making a plan. And so we have the team do that before they make a plan. And the
purpose of this is to make sure you don't confuse people, you don't confuse how
to do the job with how long it'll take. Because when you start right away to
make a plan and you're thinking about how you're going to finish it in time
while you're planning. It's amazing how often people came up and say we don't
have time to do that.
And the contention
that we follow with the TSP is that you want to do it the right way, and the
issue is to figure out what the right way is and lay out your plan based on
that. And what you'll discover is if you’re really doing it the right way it is
the fastest, and cheapest and highest quality way to do it. And so we have the
team start by figuring out what's the right way to do it, and only then do they
estimate how long it'll take and the resources. And that turns out to be a very
powerful way to do it because now the team has something to defend that they believe
it.
And so the team has a
big debate about the process flow and how do you want to do that? Do you really
want to do this? Do we want to inspect every module? And the guys, you know,
once they've got data and quality information they realize: yes, we're going to
inspect every module and yes, we're going to do this and we're going to do
that, et cetera. And you get the teams to buy that they're going to have
personal reviews and that sort of thing. So they went through all of that and I
talked about goals earlier, the goals they put together; management goals and
team goals.
And in the second meeting,
they actually set goals for the yields (% of defects found) they want to find
in their reviews and what they're going to accomplish in terms of various
things. There are a whole series of goals that they've already made when they're
going through a lot of this planning, and so when they finish that they finally
have got a process defined, they've got a list of all the products they've got
the build. They put all that together and then they make a team plan. And so in
the next meeting, the whole team works through how big are the parts, how long
will they take to do each step, and they lay that out. They're not talking
about who's going to do any of the steps yet. They're just laying out the whole
job, and how much time it'll take, and what the effort is, and they also lay
out how much time the individuals are going to have.