Commercial Off-the-Shelf (COTS) software promises substantial savings and reduced risk for large IT programs, but often falls short in meeting the unique and complex requirements of the Department of Defense (DoD). Success depends on DoD’s ability to assess how solution providers can completely address emerging, complex requirements.

Key challenge

Many COTS-based programs in the federal government have failed to live up to the promises of accelerated schedules, cost savings or reduced risk.

The unique, complex requirements of the Department of Defense introduce additional challenges for COTS-based programs, particularly for Contract Writing Systems.

DoD, as measured by scale, mission, scope, and nature of activity, is unique among organizations in the U.S. and globally. And while measures have been taken to try to adapt business processes to better leverage out-of-the- box COTS functionality, these often fall short in meeting DoD’s unique and complex requirements.

Four essential criteria for evaluating COTS software for complex requirements

FUNCTIONALITY

What can it do already, out of the box”?
A typical evaluation of COTS software includes a significant investment in defining the functional and technical requirements of the mission-critical function. These requirements typically are folded into matrices (e.g., self-certifications) where vendors evaluate their capabilities against the identified requirements, enabling organizations such as DoD to easily assess the viability of a solution based on the percentage of requirements it meets with no modification.

FLEXIBILITY

How easily can it be changed?
The percentage of requirements met by out-of-the-box functionality is only one dimension of viability. A solution may satisfy 90% of the requirements out-of-the-box, but how it addresses the remaining 10% is equally, if not more, important. That 10% typically is a primary contributor to delays and budget overruns. An important element in a COTS software evaluation is flexibility—how easily can the software accommodate initially unmet or future requirements.

AUDITABILITY

Can it prove it’s doing what it’s supposed to?
Each of the services and agencies within DoD have dedicated significant time and resources to achieving auditability by mandated deadlines. The introduction of a new system represents a significant risk to DoD’s audit readiness efforts. Therefore, DoD must ensure not only that potential COTS solutions have passed the rigors of an audit, but also that the design, development and integration of the solution support the goal of auditability.

SUSTAINABILITY

Can it be maintained at a reasonable cost?
The first three COTS evaluation criteria are highly interrelated, and have a significant impact on the sustainability of the solution. For example, a solution may have high functionality out-of-the-box, but if it requires significant customizations adversely impacting audit capabilities, the complexity to maintain and upgrade the system increases, as does the cost. As ongoing maintenance or upgrade costs increase, the affordability of implementing new, value-added features and functions decreases, limiting user productivity, and thus increasing costs.

The Success story

DoD will benefit most from a COTS solution that has substantial out-of-the-box capability, but more importantly, one that is highly configurable, with the ability to incorporate unique, unmet requirements into the core product maintained by the software vendor. To assess this, DoD should consider an in- depth fly-off of potential COTS solutions, where both out-of-the-box functionality and complex requirements are demonstrated.