Why agile methodologies are better suited to BI and Data Science when requirements change and fast feedback matters.
Tag: According to rule 1-10-100: $1 is spent checking new data, $10 cleaning it in the system, and $100 fixing the damage caused by using poor-quality data. Data on its own is often useless: to create value, knowledge must be extracted from it.
The Waterfall method, or the sequential approach to project management, involves moving through a series of phases step by step:
Some Waterfall models include variations of these phases. Is Waterfall suitable for data science?
Waterfall model effective, when the technology is well understood, the product market is stable, and requirements are unchanged.
Data science projects rarely fit at least one these criteria, which is why the waterfall model rarely fits them.
However, the structured Waterfall approach works well suitable for certain stages of a data science project, such as planning or validation.
It has the following advantages: - a clearly defined and understandable project structure; - easier communication and the ability to spot schedule delays using Gantt charts; - accurate timelines and details for each step thanks to careful planning and upfront documentation; - no project overload caused by scope creep thanks to early requirements fixation; - fewer conflicts and the ability to evaluate objectively
project progress thanks to a carefully developed plan.
The main drawback of the Waterfall methodology is its inability to align plans with the realities of constantly changing business needs. A plan may be good, but when it meets reality, it falls apart. In data science projects, applying Waterfall leads to a number of problems: - False sense of clear business requirements.Business can rarely define all its needs before a project begins. - False comfort from understanding the problem domain.Even when business requirements are clear, data specialists still need to explore the problem domain.
Only then will they determine whether the problem is solvable and, if so, how to solve it. - Delayed value realization.Extensive upfront planning delays the start of implementation, and the entire project is delivered at the end.
Therefore, stakeholders cannot receive value until the project is completed. - Poor feedback.Without access to working parts of the product, users cannot provide effective feedback. - Uncertain timelines. In data science, it is impossible to specify the exact time required for some tasks, which makes project timelines imprecise. - Rising cost of mistakes.Performing verification and validation after the development phase makes poor feature design and defects more expensive and harder to fix. - Higher overhead.Extensive documentation takes a lot of time and adds no direct value.
In traditional BI projects, each layer of the infrastructure is built separately. This requires significant investment in complex internal systems for the IT environment and data warehouses. Systems that interact with users appear in the later stages of the project. Testing is often the final stage. This carefully planned, step-by-step approach leads to many problems: - Unknown needs. End users often do not know what they want.
If you ask them to define every filter and chart in advance, the resulting plan will not meet real needs. - Difficulties of change. Even if end users understand their current needs, they may not know which metrics will matter next year. - Testing errors. The cost of defects rises as the project progresses. The same query error can recur many times.
If testing is postponed until the end of the project, you may run into errors that will be difficult to fix. - Delayed value. According to the 80/20 rule 80% of the value comes from 20% of the decisions. There is no point in delaying value delivery until the entire project is complete. - Delayed feedback. You do not know how people will use your dashboard until they use it.
Traditional approaches are based on product representations such as data flow diagrams or wireframes. They are useful for feedback, but less substantive than an incremental version of a ready-to-use BI system.
Agile business analytics is the application of agile practices to building BI products. Agile - is a mindset that emphasizes: - open collaboration among team members and with stakeholders; - rapid iteration and the frequent creation of small incremental value; - continuous controland improvement, which increase team velocity over time; - self-management project by the product team; - the ability flexible adjustment plans as needed.
Agile BI breaks a project into several small, useful stages that gradually move it toward completion.
This has the following advantages: - High engagement.
A common BI project mistake is insufficient implementation.
When end users are involved in the project from the start, they take on more responsibility.
They understand the value of the system and are more likely to use it. Flexibility.
Did you plan for COVID, the effects of sudden inflation, or any other major disruption that could happen in your business or in the world?
But with short, flexible plans outlined in roadmap, you will be able to adapt quickly to the situation. - Fast value delivery. Vertical slices end-to-end value delivers benefits to users before the project is completed.
For example, you can first deliver a table with a convenient dashboard based on it, and only then a full data warehouse. High motivation. In BI projects, the entire team can see the results of their work.
Seeing a working dashboard with partial functionality is more motivating than a complete shell buried somewhere on a shared drive.
What Is Kanban The Kanban methodology is a sequence of processes that makes work visible and aims to reduce lead time for delivering value. Its goal is to minimize work in progress and align supply with demand. The Kanban methodology focuses on two principles: - visualize the workflow; - reduce work in progress. Workflow visualization At the core of the methodology is the Kanban board, a visual representation of the workflow.
In its simplest form, it consists of three columns: "To Do", "In Progress", and "Done". Project tasks move between columns as they are ready. Kanban boards often include additional columns. For example, software development teams may split the "In Progress" column into "Development" and "Testing". The "Testing" section can be further divided into "Verification" and "Validation".
Minimizing work in progress Work in progress is an investment whose value has yet to be realized. To reduce work in progress, Kanban teams set WIP limits. These define the maximum number of tasks that can be worked on at once. To fill a column, a task must be moved from it to the next one. Kanban implementation steps for a BI team 1. Set up a Kanban board.The board columns are the work stages.
Business analysis teams can use the columns "Pending", "Analysis", "Development", "Testing", and "Deployment". 2. Define WIP limits. Set a task limit for each column, the maximum number of items that can be in that stage at any time. 3. Define work items. Assess how much work needs to be done and break it into small work items. Each item is a task in the first column of the Kanban board.
4. Set priorities for work items.Consider value, effort, and dependencies. 5. Track work progress. Team members focus on a small number of work items each time and take on new tasks when possible. 6. Measure progress. The team checks the board at least once a day. Analyze its flow to identify and implement ways to improve throughput and reduce cycle time.
When Kanban works for business analysis Thanks to its light and flexible structure, Kanban is useful for any business analysis team.
It is suitable when: - you need to process an incoming stream of requests for work items from stakeholders; - it is difficult to estimate how much time is needed to complete specific tasks; - you need a minimalist approach to project management; - the focus is on minimizing the time to complete a specific request; - team continuity is unstable and team members may be involved in different initiatives.
Scrum is the most popular agile approach. It is used not only by software development teams, but also by other companies: John Deere to create new equipment, National Public Radio for new programs, Saab for fighter jets, and CH Robinson for workforce management.
Compared with Kanban, this method is more comprehensive because it defines a number of entities. Three pillars - transparency: make current work visible; - verification: pay attention to deviations; - adaptation: adapt your processes to minimize deviation and make the most of useful opportunities. Five values - commitment; - focus; - openness; - respect; - courage. Three roles in the team Scrum recommends forming cross-functional teams of up to ten people.
There are three roles: - product owner: sets the product vision; - Scrum master: facilitates the Scrum process as a servant leader; - developer: ensures the release of new product versions. In a Scrum team focused on data analytics, the development team may include data scientists, data engineers, data analysts, systems analysts, and software engineers.
One person can combine multiple roles. Five events Agile Scrum defines five events, the first of which is the container for the others: - Sprint. Scrum divides a large project into a series of mini-projects with a fixed duration of no more than one month. Each mini-project cycle is called a sprint. - Planning. A sprint begins with structuring the work. First, the product owner explains the main backlog items - features.
Then the development team forecasts what it can deliver by the end of the sprint and creates a plan. - Daily Scrum stand-up. During the sprint, the team coordinates closely and develops daily plans in stand-up meetings. - Sprint review.At the end of the sprint, the team demonstrates increments - completed work - to stakeholders and requests feedback during the review.
These increments must be potentially releasable and meet a predefined definition of done. - Sprint retrospective. To close a sprint, during the retrospective the team analyzes and plans how it can improve results in the next sprint. Three artifacts - product backlog- an ordered set of ideas that can be implemented to create a product; - backlog - contains the sprint goal, the selected items needed to achieve it, and the implementation plan; - increment- a set of items delivered in a sprint.
A sprint begins with planning, when developers move the highest-priority items from the product backlog into the sprint backlog. The latter reflects their commitment by the end of the sprint. During sprint execution, the team meets at least once a day to develop short-term plans. Developers work closely together to achieve the sprint goal.
After it is completed, the team asks for feedback on its increment during the review and assesses how it can work better during the retrospective.
Scrum has several advantages over Kanban. The team focuses on the value of each sprint. For example, during the first sprint, a set of filters, visualizations, or a dashboard module may be created. The next sprint improves this set of results, and the third adds features. The cohesion of each feature set demonstrates progress and moves the entire project closer to completion.
What Is Data-Driven Scrum Data Driven Scrum (DDS) is a new agile methodology designed for data-driven teams to improve communication and collaboration. It is a modified version of Scrum with similar meetings, roles, and artifacts. DDS changes some of Scrum's limiting aspects, making it more flexible for the research-oriented nature of data-driven projects. The main change is the DDS iteration concept.
An iteration is a cycle of experimentation and adaptation. The goal of each cycle is to form an idea or experiment, implement it, observe the results, and analyze those observations to create the next idea. The essence of an iteration is moving from the initial idea to its implementation and the analysis of results. The end of an iteration is the end of the empirical process, not a predetermined number of hours. Scrum vs DDS
| Parameter | Scrum | DDS |
|---|---|---|
| Logical unit | Sprint | Iteration |
| Time limit | Each sprint has the same duration, set in advance | Iteration duration and start/end timing depend on when the work is completed |
| Overlap limit | Only one sprint can be run at a time | Iterations may overlap depending on work completion timing |
| Meetings | Meetings align with a predetermined sprint cadence | Meetings are held on a regular calendar schedule, but do not necessarily start or end iterations |
How a BI team can use DDS The Scrum BI team constantly faces a dilemma when preparing sprints: too many commitments will prevent the sprint goal from being achieved, while too few will lead to idle time. This is not a problem for DDS teams: they do not need to create detailed estimates for every work item to understand what fits into the upcoming sprint.
The Scrum BI team plans a predetermined cadence for each sprint according to the calendar, while the DDS BI team starts an iteration at the first opportunity. For example, when a data engineer begins setting up the next dashboard in "Iteration 2," the BI developers finish work on the current dashboard in "Iteration 1." Iterations are capability-based. They vary in length so that a logical piece of work can be completed within a single iteration.
DDS BI teams conduct observation and analysis collaboratively, embedding them into the main workflow. When DDS fits business analysis DDS has a clearer structure than Kanban but is more flexible than Scrum. The method works well for teams that want to: - visualize the workflow on a card board; - focus on the incremental set of outcomes defined in the iteration; - stay flexible without the constraints of sprint time boxes.
Moving to agile business analytics makes it possible to turn data into meaningful decisions faster. Agile methodologies help deliver early results before the project is finished, shorten feedback loops, and reduce the cost of mistakes. The right methodology improves team efficiency, decision-making speed, and provides a competitive advantage.
