Data Engineering on Microsoft Azure: Data Lake Storage

IN THIS COURSE

• 1.  Course Overview
  2m
  In this video, you’ll learn more about your instructor and this course. You’ll learn the features of the Azure Data Lake Storage Gen2, and learn when to use this storage type. You’ll also learn the features and methods for securing data for the Azure Data Lake Storage Gen2 service and Securing Data at REST. Then you’ll learn methods for processing Big data using the Azure Data Lake Storage Gen2 service and monitoring Blob storage. FREE ACCESS
• 2.  Azure Data Lake Storage Gen2
  7m 52s
  In this video, you’ll learn more about Azure's Data Lake Storage Gen 2. Data Lake Gen 2 is the Azure solution for managing massively scalable data to support big data analytics. It supports many data ingestion methods. It also allows you to use any number of popular analytics tools on the stored data. It’s an ideal choice for big data analytics needs. Data Lake Storage Gen 2 is an extension of Azure Blob Storage. FREE ACCESS
• 3.  Creating a Data Lake Storage Gen2 Storage Account
  4m 55s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to create an Azure storage account with a Data Lake Storage Gen2 hierarchical namespace. A hierarchical namespace allows you to store your objects in a collection of directories and subdirectories in the same manner that directories of files work in Windows. First, you’ll go to Azure and head to resource groups. FREE ACCESS
• 4.  Managing Azure Data Lake Storage Gen2
  10m 11s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to create a container, a directory, and a subdirectory in data lake. You’ll learn how to set the access control list on a directory to manage access permissions. You'll also learn how to upload a text file as a blob. You’ll do all of this through .NET code. FREE ACCESS
• 5.  Extracting Data from Azure Data Lake Storage Gen2
  17m 48s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to perform an ETL or extract, transform, and load using an Azure Databricks notebook. You’ll see the pipeline will extract data from Azure Data Lake Storage, transform it, and then load it into Azure Synapse Analytics. You’ll need several resources for this, which you’ll see are already set up. In Azure, you’ll go to your Resource group. FREE ACCESS
• 6.  Transforming Data Using Apache Spark
  12m 54s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to transform data using Spark in a data factory pipeline. First, you’ll create the resources in Azure that you’ll need. In Azure, you’ll hit Create a resource. You’ll need a data factory, so you’ll type in data factory, click Data Factory, and then hit Create. FREE ACCESS
• 7.  Transforming Data Using Data Factory
  11m 6s
  In this video, you’ll watch a demo. In this demo, you’ll learn to use data factory to create a data flow between two tables. These two tables are in an Azure SQL database. Onscreen, you’ll see you’re in the SQL database in Azure. You’ll head to the Query editor. You’ll create some tables, and the first table is called Inventory. FREE ACCESS
• 8.  Transforming Azure Synapse Pipelines
  17m 56s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to create a pipeline in Azure Synapse Analytics. You’ll look at some of the details, such as monitoring running pipelines and scheduling pipelines. In Azure, the first thing you’ll do is create a Synapse Analytics instance. You’ll click Create a resource. Then, you’ll type in synapse analytics and choose Azure Synapse Analytics. FREE ACCESS
• 9.  Streaming Data to Azure Databricks
  10m 13s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to create an event hub in Azure, create a Databricks instance in Azure, and use Databricks notebooks to send and receive messages via that event hub. You’ll see how Databricks can be used to ingest Event Hub messages. FREE ACCESS
• 10.  Transforming Data with Databricks
  11m 11s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to connect to an Azure Databricks resource using Data Factory, and then run a Databricks notebook using a Data Factory pipeline. You’ll start in Azure, but you’ll need a few resources. Those are already set up here. You have a data lake, which is a Storage account with a data lake option. You also have Data Factory and a Databricks instance. FREE ACCESS
• 11.  Course Summary
  1m 11s
  In this video, you’ll summarize what you’ve learned in the course. In this course, you’ve learned the features of Azure Data Lake Storage Gen2. You also learned how to secure data for the Azure Data Lake Storage Gen2, secure data at rest processing big data, and monitor the Azure Blob Storage service. You explored using and managing Azure Data Lake Storage Gen2 and extracting and accessing data from Azure Data Lake Storage Gen2. FREE ACCESS