Master the fundamentals of data engineering by building an end-to-end data pipeline from scratch. Gain hands-on experience with industry-standard tools and best practices.
Join Slack • #course-data-engineering Channel • Telegram Announcements • Course Playlist • FAQ
- Start Date: January 13, 2025
- Register Here: Sign up
- Access Cohort Materials: 2025 Cohort Folder
All course materials are freely available for independent study. Follow these steps:
- Watch the course videos.
- Join the Slack community.
- Refer to the FAQ document for guidance.
The course consists of structured modules, hands-on workshops, and a final project to reinforce your learning.
To get the most out of this course, you should have:
- Basic coding experience
- Familiarity with SQL
- Experience with Python (helpful but not required)
No prior data engineering experience is necessary.
- Introduction to GCP
- Docker and Docker Compose
- Running PostgreSQL with Docker
- Infrastructure setup with Terraform
- Homework
- Data Lakes and Workflow Orchestration
- Workflow orchestration with Kestra
- Homework
- API reading and pipeline scalability
- Data normalization and incremental loading
- Homework
- Introduction to BigQuery
- Partitioning, clustering, and best practices
- Machine learning in BigQuery
- dbt (data build tool) with PostgreSQL & BigQuery
- Testing, documentation, and deployment
- Data visualization with Metabase
- Introduction to Apache Spark
- DataFrames and SQL
- Internals of GroupBy and Joins
- Introduction to Kafka
- Kafka Streams and KSQL
- Schema management with Avro
- Apply all concepts learned in a real-world scenario
- Peer review and feedback process
Join the #course-data-engineering channel on DataTalks.Club Slack for discussions, troubleshooting, and networking.
To keep discussions organized:
- Follow our guidelines when posting questions.
- Review the community guidelines.
Past instructors:
A special thanks to our course sponsors for making this initiative possible!
Interested in supporting our community? Reach out to [email protected].
DataTalks.Club is a global online community of data enthusiasts. It's a place to discuss data, learn, share knowledge, ask and answer questions, and support each other.
Website • Join Slack Community • Newsletter • Upcoming Events • Google Calendar • YouTube • GitHub • LinkedIn • Twitter
All the activity at DataTalks.Club mainly happens on Slack. We post updates there and discuss different aspects of data, career questions, and more.
At DataTalksClub, we organize online events, community activities, and free courses. You can learn more about what we do at DataTalksClub Community Navigation.
