I’m really excited about this episode where we will do a deep dive into DataOps and discuss how organizations can get started implementing it today. To share his expert thoughts we have brought on the mic a pioneer in this space, Chris Berg,
Chris is the CEO at DataKitchen a DataOps platform that simplifies complex data toolchains and environments. Chris has more than 30 years of research, software engineering, data analytics, and executive management experience. At various points in his career, he has been a COO, CTO, VP, and Director of engineering. Chris is a recognized expert on DataOps. He is the co-author of the ‘DataOps Cookbook” and the “DataOps Manifesto,” and a speaker on DataOps at many industry conferences.
Top 3 Value Bombs:
- DataOps is not just DevOps for data
- Architect for change, don’t design architectures with a prod focus only. These systems are complicated, how you deploy is just as important as the tools used if not more so.
- CDO’s should be on the offense just as much as they are on defense. The DataOps Cookbook mentions the average tenure of a CDO or CAO is about 2.5 years.
There are four key software components of a DataOps Platform:
- Data pipeline orchestration
- Testing and production quality
- Deployment automation
- Data science model deployment/sandbox management.
It’s important to enable the locally distributed teams as they are typically closer to the business and are the “tip of the innovation spear.
CDO’s should be on the offense just as much as they are on defense. The DataOps Cookbook mentions the average tenure of a CDO or CAO is about 2.5 years. One of the top reasons the tenure is to show is that CDOs and CAOs often fall short of expectations because they fail to add sufficient value in an acceptable time frame.
Where does DataKitchen fit in a modern data architecture? DataKitchen overlays on top of a lot of your existing tools and processes. It does not replace the transformation and data tools. It will allow you to quickly spin up new environments and holistic test pipeline/code changes. You will start to be able to answer the following questions following DataOps principles and using DataKitchen:
- How do I go from a dev system to a production system?
- How do I change pieces of my architecture?
- How do I monitor and observe the system in every environment to make sure that it’s working correctly?
When creating data architecture diagrams, show how the various components will be maintained and potentially replaced. These systems are complicated, how you deploy is just as important as the tools used if not more so.
Where do you see data architectures heading over the next 2-5 years from now?
- Low code tools will continue to evolve and enable non-developers to interact with the data
- Data Storage will shift to more code-centric systems “high code”