Чем Data Lake отличается от Data Warehouse?

Data Warehouse собирает структурированные данные предприятия и делает их пригодными для потребления сразу: структура определяется заранее, обработка идёт перед загрузкой. Data Lake собирает максимум данных как есть - структурированных и неструктурированных - чтобы обработать их в будущем, когда станет понятно, зачем они нужны. На практике граница между типами хранилищ условна и зависит от качества реализации и дальнейшего использования.

Что выбрать под нашу задачу?

Отталкивайтесь от цели. Если задача - обмен данными и аналитика для целей настоящего дня, нужен Data Warehouse: структура известна, данные готовы к потреблению. Если данные явно ценны, но формат их будущего использования неизвестен, их сохраняют в озере, а обработкой занимается аналитик уже после загрузки.

Зачем выносить историю из рабочих систем?

Простой пример: система управления заказами хранит историю статусов всех заказов. Для аналитики эти данные важны, а саму систему они только засоряют. Вынос такого объёма в общее хранилище разгружает систему и сохраняет данные для отчётов, которые понадобятся через месяц в другом формате.

Что будет, если сервер откажет?

В архитектуре нет единой точки отказа. Компоненты хранилища не зависят друг от друга и могут работать на разных серверах в разных локациях, хранилища дублируются на нескольких серверах, и при отказе одного поток данных перенаправляется на другой. Серверы могут быть облачными или on-premise.

Как хранилище связывается с остальными системами?

Через ESB-подход к обмену данными: создаётся общее хранилище предприятия, к которому быстро подключаются аналитика и отчёты. Такой обмен снижает нагрузку на системы, ослабляет связанность их изменений и позволяет агрегировать и трансформировать данные под задачу потребителя.

Data Lake and DWH: Flexible Data Storage for Big Data

Using an ESB approach to data exchange, we create a shared enterprise repository for quickly connecting analytics and reports. Discuss the project. You will also receive a comparison table of integration types. Data Warehouse. A Data Warehouse is a set of structured data for the entire enterprise. It consists of storage for different objects (orders, prices, products, etc.). The data structure is defined in advance because the business knows which systems exist and how they are used for current needs.

The repositories are independent of each other and can run on different servers in different locations. So a failure of one will not affect the others. Data Warehouse makes it easy to connect any analytics system. Data can be easily filtered by open fields and the needed objects can be found. Data Warehouse improves data exchange between IT systems:

Reduces load on systems;

Reduces coupling between system changes;

Allows data to be aggregated and transformed in any way. We describe integrations in more detail here.

Comparison of Data Warehouse and Data Lake

Criterion	Data Warehouse	Data Lake
Implementation goal	Data exchange and analytics for current business needs	Storage and processing of data for future use
Data type	Structured data that is understandable to any consumer	Structured and unstructured data
Processing order	Data processing before loading into storage	An analyst processes the data after loading. The difference between DL and DWH by implementation goal. In practice, the boundary between storage types is conditional. It all depends on the quality of implementation and future use.

Data Lake

Data Lake What is a Data Lake? Let's look at a simple order management system (OMS) example. For example, an OMS stores the history of status data for all orders. This data is important for analytics, but it only clutters the system itself. It is better to move such a volume of historical data to a shared repository so as not to overload the system. It is impossible to account for all future ways this data may be used. Today, you need a report on how long it takes for an order to move from one status to another.

In a month, order data may need to be used differently and in another format. To store data for future use, a Data Lake is created - the enterprise data lake. Data enters such storage as is. When it becomes clear what the data should be used for and how, analysts can begin processing it.

Difference between DL and DWH by implementation purpose

Data Lake	Data Warehouse
Collect as much data as possible for future processing.	Collect the data and make it ready for immediate consumption.

We build fault-tolerant data repositories

With our approach, the architecture has no single point of failure.
The storage components are independent of one another and can run on different servers in different locations.
We create duplicate storage instances across multiple servers. If one server fails, the data stream will be redirected to another server 4.

You can use both cloud servers and on-premises. We select proven international open source products. We use open source solutions, so our clients reduce licensing costs without the risk of restrictions under the laws of different countries. A visual studio for creating connectors. A simple and flexible low-code platform that is part of Salesforce, with annual revenue of more than $30 million. We use the Community Edition.

03

Version control and access rights setup (roles). An open-source DevOps lifecycle web tool. More than 30 million registered users. Database (storage). A powerful open-source object-relational database system that has been under active development for more than 35 years. Store, analyze, and search logs. Enterprise security, observability, and search solutions built on the Elasticsearch platform used by thousands of companies.

04

Dashboards with flow status information A visualization and analytics system that lets you work out of the box with a wide range of data sources. 1 2 2 /

A typical workflow

We have structured the implementation process so that you get the maximum benefit. You can go through the entire journey with us. Or you can order any of the steps separately and hand the rest over to your team.

1. Designing a loosely coupled architecture

We will analyze the current IT architecture, AS-IS; work through exchanges for key entities; design the TO-BE architecture; prepare a roadmap for transitioning to the new architecture; prepare recommendations on tools; and prepare documentation. You will receive

A transition plan tailored to your business specifics

2. Migrate your most critical flows

BPMN flow diagrams; deployment and configuration of the required components (ETL, storage, logging, monitoring); connector setup; log collection and integration monitoring setup; documentation and training. You will get

Solving 80% of data exchange problems between systems

3. Migrating the remaining flows to populate the data warehouse

BPMN flow diagrams; connector setup; log collection and integration monitoring setup; documentation and training. You will get

A single enterprise-wide exchange mechanism and complete data for analytics

YouTube

We talk about integrations on our YouTube channel

View All

1C integrations through ESB

Buses vs. message brokers

2023-01-31

Greenplum as the storage analytics core

When query volume and concurrency outgrow a traditional DBMS, the warehouse is built on an MPP architecture: Greenplum distributes compute across cluster nodes, so heavy analytics do not bottleneck on a single server and scale by adding nodes. What we do in such an environment: deploy the cluster, configure loading from 1C, ERP, CRM, IoT, external API, and ClickHouse, optimize partitioning, indexes, and ETL pipelines, and build end-to-end data marts for BI and predictive calculations.

Kafka and Hadoop are connected as sources of streaming and raw data. The choice between Greenplum, ClickHouse, and a classic DWH is made based on workload profile, not fashion: they have different strengths for complex joins, wide scans, and peak multithreading.