The main focus is to optimize resources while enabling the platform to enable advanced data analytics like machine learning and data warehousing workloads. Why implement a data lakehouse?ĭata lakehouses have been defined to solve some of the existing challenges in legacy data warehouse implementations, as well as the combination of Data Lake and data warehouse implementations. The Data Lakehouse experimentation, presentation / serving layer brings great benefits to developers and data consumers by allowing them to create actionable insights while importing the information from a single platform.ĭata Lakehouses enable data warehouse professionals, report writers, data analysts and data scientists to create solutions that enrich the platform and avoiding data duplication during the process.ĭata lakehouses enable an ecosystem where data professionals can all contribute. Whether you are creating Power BI reports or Machine Learning solutions, the data comes from the same platform and offers different types of resources depending on your workload. Data Lakehouse Experimentation, Presentation / Serving Layer The best example is to think about DataFrames APIs or SQL APIs in Apache Spark.ĭepending on the APIs and type of workload, you might want to assign different types of computing power (CPUs/GPUs). These libraries and APIs are highly optimized for consuming your data assets in your Data Lake layer. The APIs layer creates a level of abstraction that allows developers and consumers to take advantage of different libraries and languages. In the Data Lakehouse metadata APIs layer, a set of APIs is used to access information. You still need to look at following data warehousing best practices, applying complex transformations and performing data cleansing. This layer does not replace critical thinking for data warehouse workloads. Durability: guarantees that once a transaction has been committed, it will remain committed even if the system fails.Isolation: multiple transactions can be executed at the same time without affecting each other.Consistency: ensures that when a transaction starts and when it finishes, it will be in a consistent state.Atomicity: guarantees that each transaction is treated as a single “unit”, which either succeeds or fails completely.ACID transactions to guarantee data consistency.Partitioning and bucketing – allow you to distribute data using different predicates (like filters) to optimize reads.The metadata layer includes features like: Examples are creating tables, slowly changing dimensions, using upserts, defining access and features that help match and improve the performance of RDBMS. This layer allows you to have data warehouse capabilities that are available in relational database management systems (RDBMS. The Data Lakehouse Metadata layer allows you to govern and optimise your data assets. This layer is decoupled from computing, allowing the compute power to scale independently from storage. With the rise of cloud technologies like Azure Data Lakes and AWS S3, storage has not only become fast but also cheap, accessible and limitless. The Storage or Data Lake layer in Data Lakehouses is used to store structured, semi-structured and unstructured datasets using open-source file formats (example: ORC or Parquet files).

Experimentation, Presentation / Serving layer.To begin, data lakehouses combine the best of data warehousing, Data Lakes and advanced data analytics to create a single platform. Users can build analytical solutions while minimising costs and overcoming existing challenges.ĭata lakehouse architecture is defined in multiple layers: Data Lake and Data Warehouse Challenges.Data Lakehouse Experimentation, Presentation / Serving Layer.Data Lakehouse Data Lake / Storage Layer.

Lakehouse databricks paper