Monday, April 28, 2008

Thought of the day: Why Projects Fail

I got a lot of good feedback to my inquiry about what makes projects fail - both here and on the internet-briefing blog. (Am also very embarrassed that I haven't followed up with the poll. It's coming - the answers were much more subtle than I expected).

On a different thread, I ran into another interesting analysis:

"There’s a lot of discussion about failed software projects. I’ll make a bold assertion here:

"Most software projects don’t in fact fail because they’re over time and over budget, but because they implement poor solutions to poorly understood problems."

— Jeff Patton, writing in his new book "Agile Development Outside-In"

Good point: Exceeding time and budget are symptoms of failure, not the causes.

This leads us to a lean practice at work: "the 5 whys". When something is going wrong, you ask "why?" But the reason itself has a cause, and you'll probably have to dig down 5 or 6 layers before you find the root cause and can eliminate it:
  1. The project is late and over budget. Why?
  2. Because we had to re-implement the XX and YY modules. Why?
  3. Because 'Business' said the solution did not meet their requirements. Why?
  4. Because the work flow was convoluted and complicated. Why?
  5. Because the developers did not understand the requirements. Why?
  6. Because the requirements specified screens and widgets, but did not communicate the purpose of the functionality.
Combine this with a "Stop the line" mentality -- eliminate problems before allowing production to continue -- and you a long way towards establishing the foundation for successful projects.

Friday, April 25, 2008

Digesting Change

Two different customers have recently raised the issue of managing (or rather digesting) change in organizations that are resistant to change. "Our company is over 100 years old and people get suspicious whenever you propose too many new ideas."

So how do we get people moving in the right direction without disrupting work and without provoking resistance? Brett Bernstein recently wrote on the scrumdevelopment mailing list.
"One challenge that I have continually come across when introducing new principals/practices is the flow rate. Too much flow and people start to choke. Too little, and they don't get to that "a-ha!" moment....

"I think your challenge is in part to figure out the right flow rate for your organization. It gets tricky since different flow rates might be needed for different levels (team vs. management) and different individuals...."

So how much change can the organization handle? And which changes should you do?

I have found the Scrum retrospective to be a powerful tool for building and motivating a team. Even if you haven't started doing Scrum yet, you can do a retrospective (you don't even have to say the word "Scrum").

If the retrospective is moderated correctly, you and your team
  1. build a common understanding of the events leading the current situation
  2. recognize the team's accomplishments
  3. produce two prioritized lists of potential improvements: a list of own improvements which the team can implement without further coordination or approval. The second list requires coordination with other entitites, such as management, customer, or other teams, perhaps to get budget, approval, cooperation or whatever.
So start with a retrospective. You and your team agree to implement the top 2 or 3 items in the own list and to negotiate agreements with the concerned parties to implement the top 2 or 3 items in the other list. The deadline is the next retrospective meeting 4 weeks in the future.

What does this process achieve?
  1. You have recognized what the team is doing well and established a basis of mutual respect.
  2. You are empowering your team and encouraging them to work as team. They have identified problems and proposed solutions.
  3. The team will accept the changes. They proposed them and prioritized them, so it is only logical that they stand behind them.
  4. You are keeping the scope achievable. By limiting the number of improvements to tackled at once, you increase the chances of success, reduce the chances of overwhelming the team and keep most of your energy focussed on value producing tasks.
  5. You, their manager, are earning their trust and loyalty. By doing what they proposed, you are making yourself useful to them. (Of course, if you don't let them do what they proposed, you risk losing their trust!)
The process is self regulating, because every month you repeat the process. The issues that demand attention will rise to the top and stay at the top until they no longer need attention. If resistance to change is becoming an issue, that too will become a topic. In any case, you can deal with the problems as they arise and directly at the source: the place where they originated is the place where they can be solved.

Wednesday, April 23, 2008

Help! I'm Trapped in a Waterfall!

Back in February, I talked about Project Management being about the "Art of the Possible." Ken Schwaber invented this phrase and it really summarizes what project management, especially crisis project management is all about.

So what do you do, when the constellation of customer, schedules, partners and other third parties, and management makes a change of the fundamental working arrangements difficult or impossible? What do you do, when you are trapped in a waterfall?

I was confronted with this problem last fall when I took over a flagging project. It had serious organizational problems. It was a waterfall. My company was responsible for steps 2 and 3 of a 5 step process, but no one knew what really had to be done by when. People would start working on step n+1 before step n had been accepted by the customer. Even after formal acceptance, the customer would make changes to the graphics designs, so that nothing was ever really finished.

What to do? The problem was that there were four companies involved in the project: the customer, a usability specialist who did the functional spec, my company (graphic design and HTML publishing) and a software development company. My company, being the middle, had only limited influence on the overall organization of the project, so we were pretty much stuck with the waterfall, at least for the moment.

I decided to introduce Scrum (or what I like to call, "the Heart of Scrum") while disrupting work and procedures as little as possible.

We started with a retrospective. This got everyone on board. There is no better way to earn a team's loyalty than to start out by asking them what we should do and then doing it!

We identified a lot of potential improvements and got to work on implementing them. Turns out a high percentage of the issues were dealt with just by introducing Sprints, Self-Organization, Product and Sprint Backlogs, and a Definition of Done.

On big point was having a proper management of the deliverables: What do we have to produce for the customer.

At the begininng, I didn't talk about Sprints, just about "Synchonization Points" (a term I encountered a few months later reading Tom & Mary Poppendieck's excellent Lean Software Development).

We used Target Process to manage the requirements. TP is very good for agile Project Management but a bit challenged in the waterfall.

Were the requirements User Stories? Well TP called them that. But actually our basic unit was the Screen (one HTML Page). One published HTML Page. Each page was a user story. Pages were grouped into Features.

Since we were a step in the waterfall, we introduced a formal definition both for "ready" and "done". "Ready" for graphic design was achieved when the customer had uploaded the wireframe into the corresponding graphic design story. Ready for publishing was achieved when the customer had uploaded the corresponding document into the HTML Story.

This was a real plus for TP: it's easy to find and link information. (Another Lean Topic)

Our definition of 'done' included customer acceptance. That is, publishing wasn't finished until the customer said, 'yes, this is how I want it'

The advantages of this approach compared to the previous way of doing things were substantial. We stopped publishing designs that we still in progress. We got a clear understanding every sprint of what the customer wanted. We got a measure of velocity.

In preparing for the first sprint, we got our act together ('our house in order' as we say in German). By the third sprint we had substantial productivity improvements. Within three sprints, we had apparently dramatically improved our productivity and quality.

But there have been some problems. Most notably, the requirement that the customer approve each step has created a significant bottleneck: the customer. Furthermore, the approval process has not guaranteed that things that are declared 'done' are in fact done. They come back from the team downstream with issues.

A waterfall framework fundamentally means that requirements are pushed in to the pipeline and onto the following steps. Matching capacity across the teams is tricky and the risk of local sub-optimization is great.

[ Update: May 28: An even bigger problem with the waterfall is the cycle time. A Value Stream Analysis shows that the work to wait ratio for any given feature is probably between 1:6 and 1:10 - if the feature needs 20 working days (4 weeks) to realize, the calendar time could easily be 25 to 40 weeks, due to the wait times implicit in a waterfall model. ]

What did we accomplish? Was it a step in the right direction? Undoubtedly. The clarity afforded by scrum is priceless, even if you're working in a waterfall. Have we done away with the problems of the waterfall? No. But the problems are becoming clearer, and we can continue to ask the "why?" questions and "how do we make this better?" questions. Sooner or later, the customer will be ready for change...

Tuesday, April 22, 2008

Rational Scrum?

A query just came up on the scrumdevelopment list:
My work is a RUP based shop. My team has been doing agile development and lately our PMO office has asked if I was interested in publishing a white paper for the company on the two. I have a decent amount of info and experience but was wondering if anyone has written something similiar or could point me toward something that has already been published?
Googling for "rational scrum" produces some interesting results. Jeff Sutherland explains why there is no rational software process. IBM explains how to embrace Scrum in RUP. And Zacharias Beckman wrote a field report about getting a project under control using a fusion of RUP and Scrum.

Beckman writes, "Rational itself is an excellent methodology and scales very well.... However, for all that I like Rational, it does have some holes… and these are plugged most wonderfully by Scrum. In fact, Rational and Scrum benefit each other so well I’ve started referring to the combination as 'Rational Scrum.'”

He goes on to list 10 positive characteristics of his combined methodology. As I interpret his text, the main contributions of Rational are its focus on "risk [which] keeps the team focused on the most difficult, potentially risky aspects of the project first," and "quality [which] becomes integral and continuous."

2 Points out of 10. I'm overwhelmed. (I'm also fair, someone coming from a Rational background might argue that some points of Scrum, such as iterations, are also found in Rational).

I don't agree that risk and quality are insufficiently addressed with Scrum. Managing risk is the job of the Product Owner and the works of Mike Cohn and the Poppendiecks describe excellent strategies for managing risk. And quality is always defined through the definition of done (and possibly additional requirements as well), so it is assured and inspected at every sprint.

And Beckmans article confirmed my own modest contribution to the literature on the difference between RUP and Scrum: "Ownership and accountability become the mainstay as everyone develops a long-term interest in the project"

People deliver projects, and Scrum is first and foremost about people.

So how should you move forward? Adopt Scrum. Understand (and internalize) the principles and values behind Scrum. Herein lies the major difference between the two. But don't throw the baby out with bath. So start with a Retrospective and be sure to identify what you are doing well. Keep those practices which work for you. And then experience how, using Scrum, you take you company to the next level.

Monday, April 21, 2008

Handling a Good Sprint Gone Bad

I was recently confronted with a sprint dilemma: At the start the sprint, the team committed to delivering a certain amount of functionality within the defined time period at the defined level of quality. Sometimes bad things happen. In this case, there was a heavy support load -- getting the previous sprint's functionality formally accepted. This was preventing the team from working on the goals for the current sprint. The Scrum-Master recognizes this in a timely fashion. What happens next?

Certainly the Scrum-Master should give early warning to the Product Owner. But what alternatives can he or should he offer? Theoretically one could:
  1. Increase the staff
  2. "Do what it takes" to finish the sprint
  3. Prolong the sprint
  4. Finish the sprint at the date planned, completing less work than committed.
The right answer is almost always 4) Finish the Sprint on the date planned. But why?

Options one and three both break the sprint contract and raise the cost of the Sprint. Increasing the staff raises the issue of training and integrating the new staff. Brooks taught us 30 years ago that adding people late in a late project makes the project later. So it's not really an option.

"Doing what it takes" usually means compromising on quality to achieve quantity, often accompanied by overtime. It also breaks the contract, but in a much subtler fashion. The increase in "productivity" is accompanied by an increase in defects, which are much more expensive to identify and correct later in the release, deployment or operational phases than if they were caught in development. Overtime is not only not sustainable, it also compounds the effects of working under pressure. So it might be an alternative in some cases, but the side effects are probably not what you want.

Prolonging the Sprint is tempting. The quality is maintained, but there are other serious disadvantages.
  1. It is an administrative nightmare - all the sprint meetings which occured like clockwork have to be rescheduled, with the accompanying potential for scheduling conflicts
  2. It falsifies the velocity - it looks like the team is making better progress (as measured in points per sprint) then it really is
  3. It deprives the Product Owner of the chance to inspect progress and adjust priorities until the functionality is completed
  4. It discourages an examination of the issues behind the delay
This last issue is actually true of all the options (except for finishing the sprint as planned). If the project is not going as planned, it is important to recognize the fact and ask the "why?" questions to determine the causes.In our case, there are some serious problems getting to "done" (and I am starting to have real reservations about using scrum within a waterfall. It's better waterfall, but it's still a waterfall!)

Only by asking way can we determine and address the real issues dragging down the team's productivity.

So, take your lumps, get a true picture of the velocity and start asking the right questions to remove impediments, increase staff availability, solve the problems or fix whatever problems the answers to the those questions reveal.

BTW - there are few cases where prolonging the sprint might make sense. Well, I've prolonged two sprints. Once was legitamate, once was debatable. But that's a topic for another post....

Friday, April 18, 2008

/ch/open Event: Scrum, Success and Quality

Yesterday I had the honor of talking to some 15 or 20 members of the Swiss Open System User Group. I like talking to experts because you always learn something from them, and yesterday was no exception.

As part of the introduction, we talked about why projects fail, which brought us to the question, "what is a successful project?" We talked about market acceptance, achieving the technical goals of the project, and customer satisfaction.

But, as I learned from the participants, there is also personal success in the project. "The project is a success when I enjoyed working on it". This aspect is often severely challenged in the crunch phase before project release (particularly when management is standing around looking for someone to motivate). A motivated developer is at least twice a good as that same person when s/he is demotivated.

Afterwards we talked about numerous topics, including the quality in software. How much quality does a project need? How do you assure quality? What is the role of an automatic test suite?

I like the analogy of a test suite being like scaffolding on a construction site. The scaffolding helps you build the building. It provides safety and security. It actually goes further, because it provides an 'as-built' documentation, which can be used to assure that future versions of the software conform to the behavior of the current version.

So, like when renovating a building, step 1) build the scaffolding, step 2) do the renovation, step 3) remove the scaffolding. You don't build the scaffolding after you've finished the building. You need it to build the building properly. (This explains partly why it is difficult to get management to authorize developing tests suites after the software is running and largely completed -- the building has been built!) It also underlines the importance of building the test suite as you build the product.

Hans Merki goes a step further.

He believes that the test suite is more the foundation of reliable software. For truly reliable applications, e.g. real time applications which a) have to work and b) in which recreating a error case after the fact is simply not possible, there is no substitute for designing quality into the product. In this case, self tests and logging and an ability to replay events can be critical to the success of the product.


The "devil's square" (also "devil's quadrat") attributed in various places to someone named Sneed (but I haven't been able to find the original source, or even his/her first name -- can anyone give me a pointer to the source?) shows the relation between Scope, Quality, Time and Cost. Scrum teams also create a formal of definition of done, which is negotiated with and agreed to by the product owner. This is the first approximation of the quality standard. But as Hans points out, the quality standard is also a requirements issue which impacts the fundamental architecture. So the quality standard should be consciously adopted in the process of creating the product.

BTW - The slides to my presentation, "Project Management with Scrum - what, why and how" are now available online.

Wednesday, April 16, 2008

Market Survey: Agile in Germany

The latest issue of OBJEKTspectrum is dedicated to Agile SW development. In one article, they presented the results of a non-representative study on how widespread agile techniques are in Germany.

Not surprisingly, Scrum is the most widespread agile framework, 21% of the respondants are using it, 17% have had first experiences with Scrum 12% are planing to introduce Scrum. Second place goes to XP at 14%, 35% and 7%, respectively. Other agile methods are largely unknown and not widely in use.

Service companies ("probably" IT companies) at 27% and Financial Service companies at 25% provided the lion's share of the respondents, but Automotive and High Tech at 13%, Telecoms at 9% and Pharmaceuticals at 7% were all well represented.

Where these companies happy to use agile? 97% were satisfied: 30% "very statisfed" and 50% normally "satisfied" und 17% marginally satisfed. Only 3% said they were dissatisfied.

Which techniques were being used? Top on the list were: Short Release Cycles, a Product Owner with Product Backlog, incremental delivery and the daily scrum. Surprisingly, only 26% of the participants were doing retrospectives. (I admit, this is the easiest to forget, but is the secret of real improvement in productivity, quality, you name it!)

What are the effects of Agile?

  1. Improved Flexibility -- 89%
  2. Improved Job Satisfaction -- 88%
  3. Better Learning on the Job -- 87%
  4. Higher Productivity -- 78%
  5. Clearer Project Status -- 74%
  6. Higher Quality -- 74%
  7. Higher Velocity -- 72%

52% of the respondents thought Agile was suitable for large projects,
26% did not (and presumably the rest had no opinion). Only 14% of the
respondants though Agile produced "Chaos" in development.

What kinds of projects are developed using agile?

  1. Web Development -- 72%
  2. Individual Projects -- 71%
  3. Product Development - 70% (including embedded systems)

Agile is popular even for larger projects. More than 7 People: 52%, including 11 to 20 People: 16% and 20 to more than 50 people: 14%

Their conclusion:

"Agile Software Development is entering the mainstream. This is confirmed not just by the brand recognition, but also by the fact that many participants have already taken their first steps with Agile and that large organizations and many different branches are adopting it.... Agile is being successfully deployed for industrial projects for many different industries.

"The value of agile development can be identified, but it is not yet so great as we had expected. There is more unrealized potential for optimization. Further, agile is being used mostly in small and medium sized projects, even though agile would be applicatable in large contexts."

They concluded with a recommendation, which I share wholeheartedly: do the retrospectives! This continuous improvement is what makes Agile work.

The article is available online in German. The print edition comes out on April 25, at which time the PDF should disappear.


Thursday, April 10, 2008

May Scrum Breakfast in Zürich: Lean Software Development

Scrum is a successful project management framework, as the talks by Marcello Leonardi, Jiri Lundak and Patrick Weiss have clearly shown. But why? And what is the connection between Scrum and success as a company? The next two Scrum Breakfasts are dedicated to these questions.

On May 7, I will present Lean Software Development: the optimal Route from Concept to Cash.

The path from Concept to Cash can be long and risky. Delays, wrong decisions, budget problems, customer acceptance, bigger and faster competition. The list of risks seam endless but, it is in fact possible to develop successful systems and products.

Toyota developed what is now known as Lean Production and Lean Product Development. Properly applied, these assure optimal Fitness for Use and optimal efficiency in production. This combination of product quality, market fitness and low cost is largely responsible for Toyotas rise to the top of the auto industry.

This talk will close the circle between corporate agility and competitiveness at the strategic level and Scrum and Agile Software Developement at the operational level. You will find out how to use Scrum to apply the concepts developed by Toyota to improve your competitiveness and agility.

Date: 7 May 2008 (always the first Wednesday of the Month)
Time: Doors open at 8.00am, conclusion 10.00am.
The talk starts at 8.35 (so that you can come by train at 8.30 and still be on time).
Location: namics ag, Konradstrasse 12, CH-8005 Zürich

The talk will be held in German.

Attendance is free and our sponsor namics provides the coffee, gipfeli (croissants) and orange juice. To register, just send me a private message or better register on xing.

The Scrum Breakfast in Zürich is a monthly exchange of information around Scrum. The breakfast offers discussion, information and hands-on experience to CIO's, executive and operational project managers. The Scrum Breakfast takes place the first Wednesday of each month.

The program starts with a short presentation about on an in interesting topic around Scrum. Then follows a moderated discussion among the participants to encourage an exchange of know-how and experiences.

Sunday, April 6, 2008

Thought of the day: The Lean Definition of Done.

Under Scrum, one of the first questions the Team has to answer is: When is a task done? A related question is when is the project done? This is the question the product owner must answer.

My lean definition of done: A project is done when the product or software created starts to bring value to the company for whom it was created. So the objective of a project is minimize the time required to get to done.

You might say "Wait a minute! That's not done, work goes on, even after a software is released." True, any useful software will need to be improved or enhanced. The question is, what is a reasonable delay between request from business and the availability of the desired function?

But the first release, call it 1.0 or XXX-Beta, needs to get out there. No release, no ROI. Someone needs to focus on achieving doneness. Under Scrum, this is the Product Owner's responsibility.

The longer it takes, the more SW development costs need to be amortized, the greater the chances or someone else entering the market or taking a dominant market share. So the cost of delay may well be greater than the cost of the development.

How long is your turn around time? If it's long, say over 6 months, what would it mean to reduce it from 3 years to 1 year? Or from a year to 6 months?

Friday, April 4, 2008

From Scream to Scrum

On Wednesday, Patrick Weiss, Director of eBusiness Solutions for Publiconnect, came to the Scrum Breakfast in Zurich to talk about his experiences as Customer / Product-Owner of two large projects which were managed using Scrum. As his talk was in German, I will summarize here.

He didn't start out to do Scrum. A change was necessary due to serious problems in the progress of his projects. Unhappiness was widespread and trust was down to zero. Without going in to detail, the problems were ascribed to "communications issues", which were corrected by changing the Project Leader and introducing Scrum.

Scrum was a suggestion of the contractor (well, my suggestion actually), and so it met with initial skeptisicm. Can it really be "agile" if we can only change our mind every three weeks? Will creativity and design suffer? It felt very limiting to only be able to react to changes at the beginning of a sprint.

The advantages soon became clear. Problems stopped accumulating, The coordination effort declined substantially. Within the months, trust between customer and supplier was reestablished.

Challenges

"Scrum changes a lot!" So a lot of practices are affected and people need time to adapt to the new framework.
  • Budgeting (particularly working with Management) presents new challenges because it is not clear exactly what will be provided (I can tell you exactly what it will cost, but not what it will do!).
  • Because of the uncertainty in the deliverables, rollout planing with sales is difficult, because they want to know what the product will do.
  • Teams that extend over corporate boundaries are a challenge, particulary if the rolls compete with one another (2 architects, 2 designers, 2 project leaders, etc.).
  • Not everybody likes transparency.
  • The team becomes more important than the individual.
  • Interdisiciplinary thinking is not something everybody wants to do.
  • "Chicken Management" - how to deal with management and other stakeholders.
  • "Ken Schwaber vs. Commen Sense"
Highlights

The management and coordination effort for the customer declined substantially. The playing rules defined expectations, provided predictability and gave everyone a sense of security. A constant, relatively high level yet sustainable level of pressure kept everybody focussed. Teamwork encouraged thinking about the whole (before everyone was a specialist and integration was a problem). More trial and error, less reading of tea leaves. Problems don't accumulate. Easier to react quickly to changes in Market, Customer Requirements, and Management Directives.

Focus on Results ("Output") and rapid feedback move the project forward, rapidly and in the right direction. Frequent releases give marketing lots to talk to the customers about.

Summary

The results of Scrum were rather different than the fears about Scrum. Motivated people work better together. Collective wisdom of the team is better than the individual wisdom of an expert. Stable teams can handle changes (e.g. staff changes) better than collections of individuals. The flexibility in design and concept was better. And "transparency" (bringing all issues out into the open as quickly as possible) encourages trust and promotes rapid solutions, even of difficult problems.

(end of summary)

You can download his presentation. Thank you Patrick for a very interesting and thoughtful retrospective on realizing large projects from the customer's point of view.

Wednesday, April 2, 2008

Agile/Target Process Course

Dates Changed

Due to a scheduling conflict with an important (and yet to be announced) Software Developers Conference, the course "Agile Project Management with Target Process" has been rescheduled to May 21 and 22.

Marketing Director Andrey Mihailenko and CTO Michael Dubakov of Target Process have confirmed their participation. (In honor of Messrs Milahilenko and Dubakov the course will be held in English, but explanations are available in German or French if needed).

The course is an Introduction to Scrum Project Management using Target Process. So you will get an understanding of the how and why of Scrum combined with practical, hands on experience using a proven tool for managing agile projects.

Early Bird Registration Deadline

The Deadline for Early Bird Registration expires on April 15. CHF 1296.-- including VAT. After that CHF 1496.-- Deadline for Registration is May 14, 2008.

You can download the flyer for detailed information or sign up online. I look forward to seeing you there!

Scrum Master wanted

Well, it took less than 24 hours from my post before the first company contacted me, looking for Scrummasters. How long will it take before someone is looking for a job?

Cheers,

Peter

Tuesday, April 1, 2008

What makes projects fail?

I need your help: I'm working on a talk about why projects fail and how to prevent failure. One of my favorite exercises from my Scrum training was listing all the ways a project could fail. I'd like to come up with a list top causes. So please write a quick comment with your candidates (or better still, tell the story of a "favorite" failure).

Once I have enough candidates, I'll set up a poll, so we can vote on the biggest causes. All results will of course be published in this space.

Thanks in advance for your help.

P.S. I'll add my own candidates after I have some responses. I don't want to prejudice the results.

Looking for an Agile Job or Contract?

One of the most frequent questions I get when talking to people about Scrum: do you know anyone who is looking for a job? Where can I find a freelance Scrummaster? Do you know where I can get a job doing Scrum or XP?

The only way I can know is if people tell me. (Seems kind of obvious, doesn't it). So, if you are looking for a Scrummaster or Agile developer, please drop me a line. Freelance, or full-time, up to you!

Or, if you are a Scrummaster or Agile developer who is looking for work in Switzerland (or even considering looking for work), again please drop me a line.

When a suitable opportunity presents itself, I will be happy to play matchmaker. Of course, all information remains confidential until I have your agreement to reveal your name to the other party.