Data-DrivenOrganizationalTransformation:GovernanceandPlanning

# Data - Driven Organizational Transformation: Governance and Planning ## 1. Organizational Planning in Data - Driven Enterprises As data platforms expand, incorporating complex subsystem teams becomes necessary. Large - scale systems, like intricate data management platforms, often have complex subsystems such as data onboarding systems using machine learning. These subsystems demand experts with in - depth knowledge and experience for development and operation. Having these teams allows stream - aligned teams to delegate tasks they lack the expertise for. There are no strict resource boundaries between different types of teams. For instance, a domain team building data products may need a data modeling service from another team. In such cases, teams can either wait or rotate members, a practice known as inner sourcing, which is based on open - source development models and can speed up delivery and enhance knowledge sharing across the organization. ### 1.1 Planning Cadences - **Sprints and Sprint Reviews**: Sprints are part of the Scrum framework, lasting a few weeks. Teams focus on achieving goals during this period. At the end of each sprint, a review session is held where teams assess results and revise plans for future sprints. Teams can seek support from others during this process, and the interaction is usually one - to - one. - **Program Increment (PI) Planning**: After a series of sprints, a face - to - face event called PI planning takes place. All teams gather to share goals and ambitions for the next sprint series. This event can last all day, with teams demonstrating work, giving, and receiving feedback. It's also a rewarding social activity. The delivery of a new architecture is complex, involving planning, execution, coordination, and organizational change. Architects play a leadership role in guiding and facilitating teams, being pragmatic, setting a vision, and inspiring others. ### 1.2 Transitioning to a Decentralized Model Shifting from a centralized to a decentralized model is a significant change for data - driven enterprises. It's advisable to start with foundational activities. First, delineate domain (stream - aligned) teams based on business capabilities and align them with the organization. Central infrastructure and data platform teams should collaborate to build the architecture. Pay close attention to metadata management from the start and implement automation and self - service capabilities. Enterprise architects are crucial in the early phases, overseeing, communicating, guiding, and connecting different parts of the organization. The success of these initial phases determines the progress of the program. Once the foundation is in place, the program can expand by incorporating more domains. Consider demand to decide which patterns to provide first, use available technologies, strive for standardization while remaining flexible, and keep services under infrastructure and platform team control to avoid cost explosion. ## 2. Data Governance and Security Data governance and data security are closely related. Data governance determines what data represents and its uses, while data security ensures that only authorized parties can access the data according to its intended uses. Applying security uniformly across the architecture requires standardization and metadata management, including assigning data owners, setting classifications, and maintaining contracts, with all information stored centrally. ### 2.1 The Need for Data Governance In small companies or start - ups, data governance is often implicit in daily activities as most applications and data are managed by a few individuals, and data volumes and varieties are limited. However, as companies grow, issues such as increased travel time between departments, scattered knowledge, longer decision - making processes, unclear responsibilities, and data quality problems emerge. New regulations like GDPR and CCPA also demand strict control of data usage, distribution, and clear documentation of data origin, storage, usage, and management. Most data governance programs are ineffective. They often consist of complex policies focused on control rather than data use. Implementing these policies can restrict data access until it's "governed." Complex tooling for data governance is difficult to configure and integrate, and implementing workflows and uploading