“Rec early, rec often”

In recent years two major shifts in thinks have occurred in the software industry. The first is a focus on iterative development and the second is on shifting activities “left” in the delivery process.

Iterative development – agile

In traditional project management, and still popular in well understood sectors such as civil engineering (eg build me a bridge) a large waterfall style project plan is defined, and followed to completion. A number of factors are important here, one is an emphasis on decisions being made early and remaining in place (eg the design, location), while another is the high cost of a change in mind (planning permission, materials, skills, logistics etc).

A typical software focused project is unlike this in a number of ways (many stakeholders, comparably low cost of design change, unknown factors of complexity, high focus on user experience). This has led to the agile movement, which seeks to break up the delivery into many “small wins”, each aiming to provide measurable benefits while minimising the risk of excessive complexity and hence lack of progress. The trade off is while you may not end up where you thought you were going, it’s actually where you wanted to be - a process of discovery along the way. This process aims to prevent lack projects delivering late and with a system no-one wants anymore.

Shifting left – devops

Another factor driving modern software processes is the acknowledgement that the early you address a problem, the cheaper it is to fix. This chart from NIST via deepsource shows the relative costs:

This impact has driven an increasing number of factors to be included in the early stages of an iteration / project. Firstly much of the testing effort was moved to be integral to the development process (via automated test frameworks and continuous integration tools).

Next was the inclusion of operational processes and considerations bringing the portmanteau of development and operations to form devops. This has now been extended to include security, for example devsecops.

Rec early, rec often

Data reconciliation (ie simply comparing data in terms of expected vs actual) forms a key part of any stateful system delivery. For product migrations, functional changes, feed verification, schema changes, backups, cache propagation - the common theme is testing that assumptions and invariants in the data are in fact true. As per code bugs, the earlier you know about issues the cheaper and faster it will be to address them.

While the common focus of reconciliation is in operational flows, shifting the usage left and making early-and-often checks part of delivery, can provide similar bfeenfits to those seen in the software community. CloudSinc provides a lightweight framework which can accelerate efforts to improve quality, lower costs and reduce delivery risk.