Smart City End to End Realtime Data Engineering Project | Get Hired as an AWS Data Engineer
CodeWithYu147:072024Science & Tech
In this video, you will be building a Smart City End to End Realtime data streaming pipeline covering each phase from data ingestion to processing and finally storage. We'll utilize tools like IOT devices, Apache Zookeeper, Apache Kafka, Apache Spark, Docker, Python, AWS Cloud, AWS Glue, AWS Athena, AWS IAM, AWS Redshift and finally PowerBI to visualize data on Redshift. MORE DATA ENGINEERING VIDEOS AVAILABLE on datamasterylab.com Like this video? - Buy me a coffee: https://www.buymeacoffee.com/yusuf.ganiyu - Become a member: https://www.youtube.com/@CodeWithYu/join Timestamps: 0:00 Introduction 1:29 System Architecture 7:22 Project Setup 9:00 Docker containers setup and coding 26:17 IOT services producer 38:19 Vehicle information Generator 48:10 GPS Information Generator 50:13 Traffic information Generator 53:13 Weather information Generator 58:35 Emergency Incident Generator 1:03:39 Producing IOT Data to Kafka 1:14:43 AWS S3 setup with policies 1:16:38 AWS IAM Roles and Credential
Landed here from a link? Roll your own.
🎲 Roll the DicePlayed through the official embedded player. Watch on YouTube.