Processing Data: Integrating Kafka with Python & Using Consumer Groups

Apache Kafka    |    Intermediate
  • 12 videos | 1h 24m 27s
  • Includes Assessment
  • Earns a Badge
Rating 4.7 of 31 users Rating 4.7 of 31 users (31)
Producers and consumers are applications that write events to and read events from Kafka. In this course, you will focus on integrating Python applications with a Kafka environment, implementing consumer groups, and tweaking Kafka configurations. Begin by connecting to Kafka from Python. You will produce to and consume messages from a Kafka topic using Python. Next, discover how to tweak Kafka broker configurations. You will place limits on the size of messages and disable deletion of topics. Then, publish messages to partitioned topics and explore the use of partitioning algorithms to determine the placement of messages on partitions. Explore consumer groups, which allow a set of consumers to process messages published to partitioned Kafka topics in parallel - without any duplication of effort. Finally, learn different ways to optimize Kafka's performance, using configurations for brokers and topics, as well as producer and consumer apps.

WHAT YOU WILL LEARN

  • Discover the key concepts covered in this course
    Create a kafka producer and consumer in python
    Process messages at the consumer using python
    Configure kafka brokers using properties files
    Enable and disable autogeneration of topics
    Publish fake data to a kafka topic
  • Write out data to a partitioned kafka topic and configure partition splits
    Recognize the use cases of consumer groups
    Create and use a consumer group
    Describe the relationship between consumer groups and partitions
    Recognize the different options available to improve the performance of a kafka environment
    Summarize the key concepts covered in this course

IN THIS COURSE

  • 1m 32s
  • 11m 49s
  • Locked
    3.  Processing Messages at the Consumer
    6m 16s
  • Locked
    4.  Tweaking Kafka Broker Configurations
    9m 48s
  • Locked
    5.  Defining Automatic Topic Creation in Kafka
    6m 54s
  • Locked
    6.  Generating Fake Data for Kafka Consumption
    10m 12s
  • Locked
    7.  Setting a Destination Partition in Kafka
    7m 22s
  • Locked
    8.  Kafka Consumer Groups
    5m 54s
  • Locked
    9.  Creating and Using Consumer Groups in Kafka
    10m 40s
  • Locked
    10.  Working with Consumer Groups and Partitions in Kafka
    6m 49s
  • Locked
    11.  Kafka Configuration
    4m 17s
  • Locked
    12.  Course Summary
    2m 55s

EARN A DIGITAL BADGE WHEN YOU COMPLETE THIS COURSE

Skillsoft is providing you the opportunity to earn a digital badge upon successful completion on some of our courses, which can be shared on any social network or business platform.

Digital badges are yours to keep, forever.

YOU MIGHT ALSO LIKE

Course Processing Data: Introducing Apache Spark
Rating 4.7 of 57 users Rating 4.7 of 57 users (57)
Course Faust: Getting Started with Stream Processing
Rating 4.2 of 5 users Rating 4.2 of 5 users (5)
Course Working with Apache Cassandra: Getting Started
Rating 4.5 of 18 users Rating 4.5 of 18 users (18)

PEOPLE WHO VIEWED THIS ALSO VIEWED THESE

Course Stakeholder Communication: Software Projects & Stakeholder Communication
Rating 4.5 of 407 users Rating 4.5 of 407 users (407)
Course Processing Data: Using Kafka with Cassandra & Confluent
Rating 4.8 of 17 users Rating 4.8 of 17 users (17)
Course Performance Testing & JMeter
Rating 4.5 of 178 users Rating 4.5 of 178 users (178)