Realtime Data Streaming | End To End Data Engineering Project
CodeWithYu87:482023Ciência e Tecnologia
In this video, you will be building a real-time data streaming pipeline, covering each phase from data ingestion to processing and finally storage. We'll utilize a powerful stack of tools and technologies, including Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra—all neatly containerized using Docker. MORE FREE COURSES: https://datamasterylab.com 📚 What You'll Learn: 👉 Setting up a data pipeline with Apache Airflow 👉 Streaming data with Kafka and Kafka Connect 👉 Using Zookeeper for distributed synchronization 👉 Data processing with Apache Spark 👉 Data storage solutions with Cassandra and PostgreSQL 👉 Containerizing your data engineering environment with Docker ✨ Timestamps: ✨ 0:00 Introduction 0:53 System architecture 3:47 Getting data from API with Airflow 17:10 Docker Compose for the architecture 26:09 Streaming data into Kafka 44:29 Apache Spark and Cassandra setup 49:33 Streaming data into cassandra 1:27:05 Outro 👦🏻 My Linkedin: http
Chegaste aqui por um link? Lança tu também.
🎲 Lançar os DadosReproduzido pelo leitor oficial incorporado. Ver no YouTube.