Data Infrastructures with Apache Kafka Competency (Intermediate Level)

Topics covered

• configure Kafka brokers using properties files
• contrast batch, streaming, and micro-batch processing
• create a Kafka producer and consumer in Python
• create and use a consumer group
• create and use topics with partitions and replicas
• create multiple topics and perform cleanup operations for topics
• describe how Apache Hadoop and Spark work
• enable and disable autogeneration of topics
• install and set up Apache Kafka
• outline a use case for and the services offered by Apache Kafka
• process messages at the consumer using Python
• produce and consume basic messages with a Kafka topic
• publish fake data to a Kafka topic
• recall the architecture and features of Apache Spark
• recall why brokers, partitions, and replicas are used in Kafka
• recognize the different options available to improve the performance of a Kafka environment
• recognize the use cases of Spark in general and specifically, its structured streaming engine
• write out data to a partitioned Kafka topic and configure partition splits