Blog

Next steps: Moving forward

Tue, 22 Nov 2011 20:48:00 GMT

Do you need help optimizing your software development lifecycle?

By assessing your software development practices against the four SQA quality levels, you should be able to determine your maturity level — and figure out roughly where you fall on the improvement continuum.

Now: Is that good enough?

If your team is already delivering software on schedule, only slight improvements may be necessary.

However, if you need deeper insight into your software development process — if you’d like to optimize it the way you’d ideally optimize your delivery of software — SQA can help.

Our specialists can work collaboratively with you to assess your maturity, identify your key drivers, and then create a tailored roadmap of change.

Got questions? Contact us today for more information! Email: info@sqassociates.com

A few near-universal best practices

Tue, 22 Nov 2011 20:46:00 GMT

One very interesting discovery from the two SQA Benchmark studies was this: there are certain key activities that can significantly improve software project delivery no matter what an organization’s maturity level may be.

In fact, we found that the most successful software development projects— those that succeed 80% of the time or more — typically utilize some or all of a shared set of core principles.

These principles include:

Centralized control of project resources. This is accomplished via formalized methods (independent of an implemented tool) to manage project resources at the Program level during inception of the project and at the Project level (from Requirements on through Release).
Formalized Project Tracking, Risk Management, and Quality Planning practices. In combination with project schedules, risk and quality plans are created to address not only quality control activities such as testing, but also activities such as code reviews, project reviews, release readiness indicators and risk factors. These activities are embedded in the project schedule.
Development of Test Strategies and / or Test Plans in the early phases of the Software Development Life Cycle. The team has a clear vision for the full spectrum of testing, obtained by developing test strategies and plans in the requirements and / or design phases that apply to all phases of the test lifecycle (unit, system, integration, acceptance, performance and regression).
Management of defects in the early phases of the Software Development Life Cycle. The idea is to hammer out defects in the requirements phase, when doing so costs the least in time and money. A significant number of participants that reported project success rates at 50% and below did not start this activity until the construction/code phase.
Dedicated Quality Assurance, Process and Governance Resources (not to be confused with Test Execution). Here, a software development process framework is in place with an identified set of activities and results. There is a centralized group, or at a minimum, dedicated resources (full time or contingent workforce) to provide:

Leadership in the definition and continuous improvement of the software development framework (activities, results, stage gates, workflow)
Governance and oversight of the software development activities once it is in place (including training / rollout)
Involvement at project initiation to ensure an appropriate stage approval process
Management of the workflow throughout the software development lifecycle
Facilitation and leadership of quality assurance activities such as walkthroughs, code reviews, and defect trending and analysis
Metrics and reporting of the pipeline activities

Measurement of System Maturity. The goal should be a defined and repeatable measurement system, well-designed for the needs of project decision makers. Metrics portfolios are expanded beyond defect and project per se, to other domains such as change management and resource utilization. There is an understanding of the cause and effect relationship between quality indicators, and they are utilized as predictors and criteria to evaluate release candidates.
Demonstrated ROI in Performance Testing. Getting high project ROI typically requires:

Performance testing
Performance enhancement that occurs in the earliest possible phases of the development lifecycle
Using performance measures as a criterion to assess release candidates

Be sure to read next week’s blog on next steps for furthering your lifecycle optimization.

Which level are you?

Tue, 22 Nov 2011 20:35:00 GMT

Level 4 Quality Management / Optimization

Level 4: Quality Management is the highest level of maturity in SQA’s assessment model.

It’s characterized by full deployment and use of tools, effective IT governance and oversight and advanced use of a skilled and scalable workforce.

Automation, just-in-time communication via dashboard reporting is also needed for intuitive project management at this phase. This means tools will need to be fully integrated.

Within dashboards, key performance indicators (KPIs), based on quantitative data, can help predict project-slip and mitigate risk. Any detected issues can then be addressed proactively, before they get a chance to escalate and threaten the project’s success.

The main characteristic of this level is that the processes are Governed. The following are additional characteristics of Level 4 organizations:

Inter-group communication, collaboration and coordination.
Work that’s accomplished according to a plan. Actual results are compared to planned results, guiding improvement in best practices.
Practices are executed consistently using defined processes (outlined as a Level 2: Quality Control Best Practice).
Ongoing process reviews, updates and training are rendered as required.
Well-defined roles/responsibilities.
Management formally commits to projects, and suitable resources are fully allocated and governed over time.
Reward and recognition are awarded for quantitative improvements.

As confirmed by multiple studies, our experience has been that some of the biggest rewards from high maturity stem from the fact that software quality is now much more predictable — regardless of software “size.”

Check back next week as we explore best practices in greater detail.

Which level are you?

Tue, 22 Nov 2011 20:33:00 GMT

Level 3: Quality Assurance

What differentiates Level 3: Quality Assurance from the first two levels is that business partners now play a key role in developing the software solution. It’s no longer strictly IT’s ballgame.

For this reason, while this level still uses all the best practices at the earlier levels, it also adds activities that now require business / end user involvement.

These activities include collecting user stories and creating early prototypes, to identify missed requirements and resolve critical defects early in the lifecycle when finding and fixing them is a relatively easy thing to do.

Organizations at this level go beyond just defining the activities to be performed in each phase of software development (and the target results). There’s also a new focus on improving efficiency (e.g., via automation).

To quantify the total cost of deploying software, a Level 3 organization also uses various new evaluative metrics. These turn a spotlight onto previously unseen roadblocks and suggest necessary rework – pinpointing what must be done and prioritizing its importance.

At this level, it’s also important that critical dependencies are identified (i.e. entry / exit / handoff criteria) to keep the software development lifecycle as integrated and process-driven as possible, which, in turn, minimizes costs and increases efficiencies. There is knowledge of external processes that support, govern and enable continuous improvement of all software development activities.

In order to identify, classify and score / rate all processes that affect the software development lifecycle, organizations at Level 3 typically have a defined process model. In addition to the model itself, they will have analyzed each process and will have a solid understanding of the relative importance and maturity of each one. Gaining insight on how mature each process is, and the impact it has on the delivery of software, will ensure that continuous improvement efforts are focused on the weakest links. Optimization can only be achieved when the whole is taken into account.

The following System Development Process Model is an example of a Level 3 System Development Process Model:

Example of Level 3: Quality Assurance System Development Process Model

Check back next week to learn about Level 4: Quality Management Optimization.

Which level are you?

Tue, 22 Nov 2011 20:28:00 GMT

Level 2: Quality Control

At Level 2: Quality Control, organizations are typically driven by a schedule, and are rarely focused on building systems that are efficient and maintainable.

For this reason, IT organizations at this level typically employ some kind of software development methodology to manage the project’s life cycle with greater control than you find at Level 1.

Various software development models may be used (i.e. Waterfall, Incremental, Agile, SCRUM, RUP, V-Model, and Hybrid). And regardless of which model is in place, the best practice activities performed will typically include the following¹:

Project Initiation / Business Case / ROI
Development of Business Requirements
System Requirements / System conceptualization
System Design
Architectural design / Detailed design
Unit Development and Testing
Software Integration and Testing (Functional)
System Integration and Testing (Performance and Regression)
Requirements Validation and Testing (User Acceptance)
Change Control / Site Testing and Acceptance
Implementation / Training (IT and Business) / Post Project Reviews
Maintenance

The software development methodology should also provide guidelines, templates and tools — customized for maturity — that support the key themes. Also helpful: mentors and metrics, both of which can help to ensure the defined framework will be implemented appropriately.

A company, organization or individual group will identify what model(s) work for them, what activities will or will not be included and in what sequence or overlap of events. A label (phase) will be added and specific activities that are accomplished as part of that phase will be identified. There are usually multiple choices or “paths” that can be taken based on project size, scope, risk, culture, etc. It is typical to have four or five choices.

In addition to having a well-defined framework with multiple paths, the organization will need to focus on the processes described by the SDLC Framework. Within it, requirements, development, testing, defect lifecycle management and compliance (phase gates / approvals) are the usual front-runners.

Some quality assurance practices, such as walkthroughs, may be expected and conducted. Project management is also well defined, although it may or may not be fully utilized. (And although project plans and schedules may be developed, too little time is usually spent on maintaining them and improving estimation practices.)

One of the keys to keeping the project on schedule is to employ workflow / pipeline / release management. This, in conjunction with change and control management, can really help streamline the software development cycle.

It also makes the experience of implementing new best practices much more tolerable for organizations used to Level 1 (which is, by definition, ad hoc).

Doesn’t sound like your organization? Read next week’s article to learn about Level 3: Quality Assurance.

¹List created from SQA Benchmark Study: The Value Proposition of Planning and lecture notes: Software Engineering Best Practices.

Which level are you?

Tue, 22 Nov 2011 20:18:00 GMT

PART III: LEVEL 1 HEROICS/ AD HOC BEST PRACTICES

Level 1 Heroics / Ad hoc Best Practices:

Since Level 1 is typically ad hoc in approach, activities performed are usually driven by individual, heroic efforts — not a governed, managed process that’s optimized over time for increasing excellence.

There will often be an implicit (rather than formal) methodology that is culturally accepted and maintained in the local context. This approach, though limited in scale, does work well in small IT environments and start-up operations where there is a direct connection with a business partner or end user.

Best practices pertinent to this level usually focus on how production is protected. And there are well-implemented change control and release management activities, as well as insight into technical specifics like application, network and data security.

In most organizations at this level, related domains such as performance testing, configuration management, release engineering and production readiness have also matured. This is intended to help ensure software is truly ready for the production environment on release.

Organizations at this level commonly also rely on software engineers with a broad range of skills, since they are typically asked to play many different roles. Thus, for Level 1, there is a primary reliance on people, their skills and experience.

Check back next week to read about Level 2: Quality Control.

How Best Practices Can Drive Improvement

Tue, 22 Nov 2011 20:15:00 GMT

PART II: BEST PRACTICES OVERVIEW

Over the last 10 years, SQA has worked with many organizations interested in improving the delivery of their software. Part of that work has involved creating best practices designed to enhance software development processes such as requirements, testing, release and maintenance.

In addition to our own insights, we’ve leveraged ideas from many external sources, such as popular frameworks and theoretical white papers. We’ve also confirmed our best practices work well by conducting two benchmark studies that yielded consistent results.

Of course, really making our best practices work effectively for you will mean thinking carefully about the complete business context in which they’re going to be applied.

Many variables will need to be weighed and considered, such as:

Your current level of maturity
Your willingness and capacity to reach a higher level of maturity, despite the time and resources (e.g. cost) it will take to mature
Your unique culture and special organizational drivers or themes

You see, no two organizations are going to have the same context. Fortunately, our best practices are flexible enough to be tailored or adjusted for every context. Any given SQA best practice can thus evolve, inside your organization, to become an even better practice.

This is rather similar to the way certain well-regarded development methodologies, such as Agile, sometimes must still be modified slightly. They may work well “out of the box” in one organization, yet when implemented in another organization will need some adjustment.

A custom fit, in other words, is almost always best. The challenge faced by most organizations, then, is to sort through the range of available best practices to find a combination likely to yield optimum results.

Ideally, the proper mix of best practices — adjusted for context — will then minimize software development process problems and unforeseen complications, drive down costs and drive up quality and customer and employee satisfaction.

How mature is your software development process?

Tue, 22 Nov 2011 20:16:00 GMT

PART I: MATURITY LEVELS

One useful way to assess your software development process is to consider how mature it is — that is, which of four general levels it falls into.

SQA has devised the following system based on our experiences at past customer engagements as compared with best practices:

Level 1: Heroics. This is unpredictable and essentially uncontrolled.
Level 2: Quality Control. Task-driven, with repeatable activities.
Level 3: Quality Assurance. Collaborative, integrated and implemented with the complete lifecycle in mind.
Level 4: Quality Management. Process-governed, measured and focused on continual optimization.

Each level includes all improvements from the lower levels.

Want a more specific look at these levels? Consider this chart:

As you can see, maturity improves as the various phases of the software development cycle become more integrated, and instead of being task-driven, the cycle is instead process-driven. Software is developed in an orchestrated, consistent way — not via ad-hoc, individual efforts.

Organizations at Level 4 (Quality Management), the column furthest to the right, typically create the best software. And that’s true whether you evaluate it in terms of schedule, cost, quality or customer and employee satisfaction.

Unfortunately, few IT organizations today are beyond Level 2 (Quality Control).

And actually, even getting that far is more than many have managed.

To achieve Level 2, previously mastered tasks have to be repeatable. Common examples include freezing requirements, quality planning, document control, early lifecycle reviews and defect management. And templates, checklists, phase gates will also all need to be in place.

Now imagine your process is being audited, with those elements all considered as part of the audit.

Would you pass such an audit?

The Improvement Continuum

You can also think of maturity as an answer to this question: How predictable is your project delivery?

Concepts of continuous improvement (e.g., Kaizen, Lean, Process Mapping, etc.) as applied to software development activities can really help — reducing costs, firming up the production schedule, enhancing quality levels and increasing customer and employee satisfaction.

And toward making the software development lifecycle both faster and more predictable, best practices can play a critical role. They’re discussed in our next blog entry.

Most of today’s software earns a failing grade. Why?

Tue, 22 Nov 2011 20:04:00 GMT

PART II: THE ANSWERS

Success in software development is no longer a simple matter of meeting project delivery targets and maintaining the operational status quo. Today, IT is asked to play a much more collaborative role with the host organization — creating enormous value by delivering software that can create a competitive advantage in a tough marketplace.

Yet, IT budgets are also usually either flat or falling. This means IT is, quite simply, being asked to do more with less.

How are IT groups accomplishing this? One common but problematic approach: point solution strategies that focus on only one aspect of software development. If there is a perceived “bad” process in the development lifecycle, the idea might be to “fix” or even outsource it, leaving the rest of the lifecycle alone.

We’d say that 90% of our assessment/evaluation work falls under this heading — it solely concerns a specific process such as testing or requirements.

But the truth is that things are rarely this simple, and really improving the software will mean implementing positive change in a way that is both deeper and broader than that.

Sure, you can improve a single process by uncovering and eliminating inefficiency. But the improvement you get will go only as far as the boundaries of that process – not the software development process as a whole.

Real optimization that leads to measurably superior software, faster development cycles and lower development costs, will require a more holistic approach that incorporates the complete software development lifecycle. The goal should be to improve what goes in, and what comes out, of each phase, not just one.

Big-picture process governance is also needed to ensure that everyone involved is equally invested and accountable for the success—or failure—of project delivery. From early lifecycle planning to ongoing execution, you’ll need to identify, define and utilize the best available tools, metrics and processes. The alternative is what we see all too often: fragmented planning and execution processes that work well in a limited sense, but don’t ultimately lead to better software.

Once planning and execution have been understood and tackled, the focus can then turn to excellence – i.e., ongoing optimization of the software development lifecycle, to achieve better and better results over time.

One great way to pursue this lies in leveraging established best practices — the methods, processes, activities, incentives or rewards that have repeatedly been proven to work well. So it will be important to determine just what those best practices are, in your particular context, always bearing in mind that a “best practice” for one organization may not really be the best for another.

The key is to maintain a holistic perspective – don’t miss the forest for the trees. Establish the key organizational drivers for your project, and then evaluate the entire software development lifecycle as well as the processes you need to govern it, support it, maintain it and enable it.

And don’t forget to take into account the cultural considerations unique to your organization, to help ensure that the organization as a whole is working on the weakest links (which are also the most critical for optimization). When possible, link business, development and operations team members in new ways, to collaborate as effectively as possible, instead of working in separate little worlds of their own.

If you succeed, you’ll have a top-down approach that also provides an end-to-end view of the software development lifecycle.

Most of Today’s Software Earns a Failing Grade. Why?

Tue, 22 Nov 2011 19:59:00 GMT

Part I: THE QUESTIONS

Defining successful delivery of software is easy. The act of creating it is not.

Successful software, in an abstract sense, is simply any software that satisfies the client’s business requirements. More specifically, you might define successful delivery of software like this: software that’s released on time, under budget, feature-complete and as stable and bug-free as possible.

Unfortunately, despite all the methodologies and tools commonly used to achieve these goals today, most of today’s software projects simply don’t measure up. According to several recent studies, only 32% of projects are considered successful as measured by these yardsticks. Worse, the trend is getting worse, not better, because that 32% is also a 3% drop since 1997.

What does that say about the project manager, business analysts, developers, and testers? Would you have confidence in an IT partner that only succeeded 32% of the time? Last time we checked, 32% was a failing grade.

Common-sense approaches intended to yield a better result – “drive quality from the start of the software development lifecycle, and minimize the number of builds and total project cost” — are certainly a good place to start.

But as a practical matter, moving from that abstract concept to a specific implementation is much easier said than done.

Let’s say you take the above advice and try to drive quality from the start – i.e., focus on the requirements phase. How exactly do you make that happen, and get it right? In our quarter-century of experience, as much as 40% of a project team’s time, even today, is wasted on requirements definitions that don’t lead anywhere useful.

What we see instead bears out industry studies – that well over half of the defects found in up-and-running production software ultimately stem from poor requirements definitions.

So what’s the solution? Small-team, rapid-development approaches such as Agile and SCRUM are gaining popularity, but are they really leading to better software delivery as we defined it at the beginning of this blog entry?

And if quality assurance best practices can help, why aren’t they seeing wider support and implementation?

Check back with us next week for Part II, in which we’ll move from these common problems to some of the best available solutions!