With the increasing adoption of agile, there’s been a lot of talk about moving from the project-centric delivery to product-centric delivery model. Initially software/IT borrowed project management practices mainly from manufacturing / construction industries, and such projects generally used sequential, waterfall processes. The technology revolution and the pace of change in IT has made such approaches difficult to sustain and the software industry has been quick to adopt agile methods like scrum which are adapted to product-centric delivery instead of projects.
The last few years has seen a massive change in the data landscape. With the rise of big data, there’s been rapid innovation in the tools, skills and roles working on data systems. Data architectures have evolved beyond monolithic, centralized databases and unwieldy analytic applications to distributed, scalable architectures with simpler collaborative and interactive analytic tools. In this post, I look at the defining features of modern data architectures.
Modern data architectures generally feature the following (though not all of these may be present in the same system):
Over the past few years, Kafka has become the most exciting new addition in the big data distributed architecture. Originally developed at LinkedIn, its founders Jay Kreps, Jun Rao and Neha Narkhede have launched a company Confluent to develop its open-core business model. The software at its core, Apache Kafka reinvents the database log to provide a highly scalable and fault tolerant, high performance distributed system, which serves as the data pipeline backbone for stream data processing.
Eric Ries, a young American entrepreneur and ex-software engineer published “The Lean Startup” in 2011. The book is the collective wisdom of the valley and smart folks like Ries and Steve Blank about the purpose of a startup and the methods that help with it most efficiently. It’s because of the explosion of interest in the lean-startup methods that today concepts like “fail-fast”, “MVP”, “pivot” are commonplace. Coupled with lean-concepts and the experimentation afforded by agile methods, the lean-startup techniques improves the chances of success of a startup.
Scrum is the most popular agile framework out there. It uses 1 to 4 week iterations called sprints to deliver products. Developed by Jeff Sutherland and Ken Schwaber, it has been used since the nineties, i.e. even before the 2001 agile manifesto was written. The word scrum is taken from rugby.
Being a framework, scrum needs to be tailored to your situation. A scrum team is supposed to be cross-functional and self-organizing in order to minimize dependencies and wait times while improving collaboration.
