The data ingestion / collection enables to connect different data sources through batch or real time modes. The AWS data lake architecture is based on several components:ġ. The diagram below presents a first solution for the requirements. What is the best AWS architecture for these requirements? produce ML models, QuickSight dashboard or external API with the output data.ingest data manually or from data pipelines.Your company needs a data repository with the following expectations: Let’s evaluate how these data architecture patterns can be applied using a concrete example. This decentralized architecture brings more autonomy and flexibility.Įach data architecture model has benefits and shortcomings, there is no good and bad approach, it depends on the context and the use cases. Interoperability and standardization of communications.Domain-oriented data owners and pipelines.The data mesh architecture is based on few principles: The domains are all connected through an interoperability layer that applies the same syntax and data standards. Unlike a “monolithic” central data lake which handle in one place consumption, storage and transformation, a data mesh architecture supports distributed, domain-specific data consumers and views “data-as-a-product,” with each domain handling their own data pipelines. The lakehouse allows to manage ACID transactions through the metadata layer while keeping the data stored in the low cost data lake storage.Īnother alternative that fits well in a complex and distributed environments is the data mesh architecture. The lakehouse provides a metadata layer on top of the data lake (or object) storage that defines which objects are part of a table version. query performance implies to select wisely the data format (such as Parquet or ORC)Ī first alternative is the lakehouse architecture which bring the best of 2 worlds: data lake (low cost object store) and datawarehouse (transactions and data management).managing ACID transactions or rollback requires to write a specific ETL/ELT logic.data management is challenging with data lakes because they store data as a “bunch of files” of different format.The data lake is very powerful solution for big data but there are some limitations: setup the right storage (and the corresponding lifecycle management).schema on read versus the traditional schema on write.all data stored in a single place with a low-cost model.The data lake is scalable and provides the following functionalities:

The data lake is the central repository that can store both structured data (such as tabular or relational data), semi-structured (key/value or document) and unstructured data (such as pictures or audio).

The data lake is the key foundation of data analytics. This article is a result of a chat discussion with Willian ‘Bill’ Rocha, Kevin Peng, Rich Dudley, Patrick Orwat and Welly Tambunan.

Amazon data lake personal backup