If you have participated in any type of business project over the past 10 years, you have likely been introduced to or are leveraging the Agile process.
Agile began its life in the software development world during the late 1980s and early 1990s. At the time, software development projects were seeing a tremendously high failure rate. Many attributed those failures to inefficiencies with the traditional “waterfall” method, which consisted of gathering all the requirements up front, disappearing for six months to develop the solution, then reappearing with working software.
In the vast majority of cases, after months of research, deliberation, and development time, the final product did not fulfill customer requests, nor did it provide the outcome they needed. In essence, companies were investing large amounts of time and money up-front, only to be disappointed with the end result.
After a lot of research and a few ideas from some highly intelligent people, it was discovered that the some of the main causes of failures were largely due to three things:
- Poor ability to estimate the time to complete a task
- Constantly changing requirements
- Miscommunication between engineers and stakeholders
A Brief Overview of Agile
In contrast to the waterfall method, Agile is all about providing “quick wins”. It is about providing many smaller chunks of value in a much shorter amount of time, or sprints. Agile reduces the complexity of estimating and focuses on having real conversations with the business as new functionality is being implemented. This process is repeated over and over again until the entire project is completed to the business’s satisfaction. As a benefit, the business starts to realize value almost immediately, is more in tune with the overall progress, and can easily adapt to changing requirements.
As agile project management started seeing more success in the software development world, businesses started to realize that problems addressed by this new management style were not specific to software development; they were general problems that were part of almost any business project. Thus sparked the spread of “agile” everything.
Why Data Quality Needs to Be Agile
Data Quality Management (DQM) is great example of how an agile approach can make a big difference. Data quality solutions provided by companies like SAP, SAS, Microsoft, and Oracle are largely offered as on-premises solutions that require a significant up-front investment not only in software costs, but in changes to network infrastructure as well. They can easily take weeks or months for installation, user training, and deployment to production before the business realizes any tangible benefit. These solutions are similar to the waterfall method; the company is required to make a large, up-front investment while waiting a significant amount of time before improvements can be seen. The initial software costs can quickly reach $100,000 or more. In addition, the business is often required to purchase additional server resources, while employing the staff to maintain it.
Existing DQM solutions also suffer from the problem of having a steep learning curve. If you have ever tried to work with a tool like Microsoft Data Quality Services, you likely got lost in new terminology and concepts like Knowledge Bases and Data Domains. My guess is you didn’t make it very far because you realized the investment of time was going to be more than you could spare.
For an agile data quality process, you need a tool that meets these minimum requirements:
- Get up and running fast!
- Connect to your data easily
- Utilize your team’s existing skills
- Reduce (or remove) the up-front costs