Data Engineering on Microsoft Azure: Data Flow Transformations

IN THIS COURSE

• 1.  Course Overview
  1m 46s
  In this video, you’ll learn more about your instructor and this course. In this course, you’ll explore the types of Azure Data Flow transformations available. You’ll learn how to transform data, split, handle duplicate data, and flatten data using Azure Data Mapping Data Flows. Next, you'll explore the types of expression functions available in Azure Data Flow, and learn how to perform error handling for data rows that would truncate data. FREE ACCESS
• 2.  Azure Data Flow Transformations
  7m 40s
  In this video, you’ll learn more about the transformation blocks available in Azure Data Flow. The first ones you’ll learn about are schema modifiers or transformations to the layout of the data. The aggregate transformation allows you to group data and perform aggregate functions. The aggregate can be stored as a new column or it can be stored in an existing column. FREE ACCESS
• 3.  Using Azure Data Mapping Data Flows
  9m 6s
  In this video, you’ll watch a demo. In this demo, you’ll see how to clean incoming data by removing unwanted data using a filter in a data factory data flow pipeline. You’ll use Azure and go to your Resource group. Then you’ll use Blob Storage. You’ll head to the Blob Storage called storage2426 and open it. From there, you’ll go to Containers, and see there are two Containers set up. FREE ACCESS
• 4.  Performing an Azure Data Flow Conditional Split
  10m 50s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to use a conditional split transformation in a Data Factory Data Flow. This transformation allows you to split the flow of logic based on a condition. In Azure, you’ll go to your Resource group. You’ll be using a SQL database, which is already set up. You’ll also be using a data factory, which is also already set up. FREE ACCESS
• 5.  Using Azure Data Flow Conditional Flatten
  8m 17s
  In this video, you’ll watch a demo. In this demo, you’ll learn how to flatten JSON data using Data Factory. You’ll open Azure and go to your Resource group. The only thing you need for this is a storage account and a data factory. You’ll use the storage account called tempstorage2426. Now, you’ll head to the Containers. Choose the Container set up called demo-data. FREE ACCESS
• 6.  Azure Data Flow Expression Functions
  5m 26s
  In this video, you’ll learn about Azure Data Flow. Azure Data Flow allows you to build a series of steps that transform your data. The Expression Builder is a tool within Azure Data Flow that can be used with many of those transformation steps to define logic. Here, you’ll look at some of the functions available in Azure Data Flow's Expression Builder. The Expression Builder has many math functions. FREE ACCESS
• 7.  Configuring Azure Data Flow Error Handling
  10m 3s
  In this video, you’ll watch a demo. You’ll learn how to handle a table schema error issue in a Data Factory Data Flow. You’ll start in Azure, and you’ll read a CSV file from a storage account called tempstorage2426, and this will be read by Data factory. You’ll output those records to a SQL database table in the sqldatabase2426 database. You’ll start off by looking at the Storage account and click Containers. FREE ACCESS
• 8.  Transforming Azure Data Flow Derived Columns
  10m 16s
  In this video, you’ll watch a demo. You’ll learn how to use derived columns in a Data Factory Data Flow to normalize columns. You’ll use Azure and the Storage account called tempstoarage2426 and a Data factory instance called datafactory2426. You’ll open up your Storage account and go to Containers. You’ll see a Container called demo-data. You’ll open it and see there’s a CSV file. You’ll open that and hit Edit. FREE ACCESS
• 9.  Transforming Data Using Azure Spark and Scala
  9m 56s
  In this video, you’ll watch a demo. You’ll learn how to instantiate an HDInsight instance in Azure and run a Scala transformation on a CSV file. You’ll head to Azure and you’ll see a cluster is already set up, the HDInsight cluster. It's called cluster2426, and it’s a Storage account. You’ll learn HDInsight clusters can take a long time to load up. They typically take 20 to 30 minutes. Here, you’ll just learn the steps. FREE ACCESS
• 10.  Handling Duplicate Data in Azure Data Flows
  8m 46s
  In this video, you’ll watch a demo. You’ll learn how to filter duplicates in your data by using Data Factory. In Azure, you’ll use two things. You’ll use a Data factory instance that you have in your Resource group datafactory2426. You’ll be using a Storage account with blobs called tempstorage2426. You’ll go to tempstorage2426, click "Containers", and see there’s a Container already created called demo-data. FREE ACCESS
• 11.  Course Summary
  1m 19s
  In this video, you’ll summarize what you’ve learned in the course. You’ve examined how to describe the type of available Azure Data Flow Transformations, transform data, split data, flattened data, the types of expression functions, configure error handling, normalizing data values, ingesting data using Azure Spark and Scala, and handling duplicate data. You explored available Azure data flow transformations, transforming data with Azure data mapping data flows, and Azure data flow. FREE ACCESS