Data Engineering on Microsoft Azure: Designing Data Storage Structures

IN THIS COURSE

• 1.  Course Overview
  1m 30s
  This course will introduce you to data lakes and the files types that are most appropriate for your use case. See how to design table storage that supports efficient queries, how to prune unneeded data, and how to organize data effectively. FREE ACCESS
• 2.  Designing a Data Lake
  7m 12s
  Study how to design a data lake. Consider the reasons to use a data lake and compare its purpose with that of a data warehouse. Examine use cases for data lakes, such as for descriptive, diagnostic, predictive, and prescriptive analysis. Review the characteristics of a data lake, and some challenges. FREE ACCESS
• 3.  Big Data File Type Planning
  7m 17s
  Identify and evaluate criteria to select a file format for big data applications, such as text, or binary, data types, schema, and OLTP or OLAP. Review big data storage considerations, such as splitability, compression support, bath or streaming, organizational standards, and data catalog needs. FREE ACCESS
• 4.  Big Data File Formats
  8m 46s
  Explore the defining characteristics of the supported file formats in Azure Data Lake. Examine benefits and issues with the comma separated value (CSV) format, extensible markup language (XML), Apache Avro, Apache Parquet, and optimized row columnar (ORC). Review protocol buffers. FREE ACCESS
• 5.  Designing Table Storage for Querying
  8m 31s
  In this video, you will study how to design efficient table storage for queries. Discover the benefits of denormalized data, point queries, and the long tail pattern. Consider alternate approaches to domain models. FREE ACCESS
• 6.  Dynamic Data Pruning
  6m 25s
  Examine the dynamic data pruning feature in Databricks at the file and partition level. Review nested filters, static partition pruning, pruning challenges, and dynamic partition pruning considerations. FREE ACCESS
• 7.  Designing a Folder Structure
  7m 3s
  Watch how to design a folder structure. Discover why data structure is important for a data lake, so that it does not become a data swamp. Review governance practices for metadata management. See why nested elements should be avoided. FREE ACCESS
• 8.  Data Lake Zones
  5m 51s
  Explore how to define the zones within a data lake to organize data distribution. Review the roles of data separation, governance, service level agreements, and security. Consider the requirements of the raw zone, the structured zone, the curated zone, the serving zone, and the exploratory zone. FREE ACCESS
• 9.  Storage Archiving Tier
  6m 47s
  Examine the data access tiers in Azure Blob storage, and how data can be moved between them for efficient and cost-effective storage. Look at blob types, access tiers, blob lifecycle management, the archive tier, and immutable blobs. FREE ACCESS
• 10.  Data Archiving, Rehydrating, and Life Cycle Management
  7m 52s
  Walk through the steps to archive data in an Azure Blob storage container, rehydrate blob data, and automate access tiers using life cycle management. See how to move data between tiers for cost effective data management. FREE ACCESS
• 11.  Course Summary
  54s
  This course introduced you to data lakes and the files types that are most appropriate for your use case. You learned how to design table storage that supports efficient queries, how to prune unneeded data, and how to organize data effectively. FREE ACCESS