Introduction to Data Engineering (5 op)

Arviointiasteikko

H-5

Sisällön jaksotus

Course Overview
This course provides an introduction to data engineering, combining theoretical concepts with practical applications. The course is divided into two main parts, each with a distinct focus:

- Part I: Theories and Practice
• Instructor-Led Sessions: Covering general topics in data engineering, taught and supervised by the instructor.

- Part II: Optional AWS Academy Self-Paced Course
• Self-Paced Learning: Students have the option to independently complete the AWS Academy Data Engineering course, gaining in-depth knowledge and earning a certification. This can replace the requirement to complete standard homework assignments.

Course Structure
Part I: Theories and Practice (Instructor Supervision & AWS Academy)
• Week 36: Course Overview and Introduction to AWS Academy Data Engineering
• Week 37: The Data Engineering Ecosystem & AWS Practice
• Week 38: ETL Processes & AWS Practice + Exercise Demo (I)
• Week 39: Introduction to Apache Airflow & AWS Integration
• Week 40: Data Engineering Life Cycle: Data Wrangling & ETL + AWS Practice
• Week 41: Data Wrangling and ETL in Apache Airflow + AWS Practice
• Week 42: Autumn Break
• Week 43: Data Governance and Compliance in Data Engineering + AWS Practice
• Week 44: Exercise Demo + AWS Practice
• Week 45: Continued AWS Course Study
• Weeks 46 & 47: Group Work on Final Projects (in-class) + AWS Practice
• Week 48: Final Project Presentations

Part II: Optional AWS Academy Data Engineering [91081]
- Self-Paced Modules: Students can choose to complete the full AWS Academy Data Engineering course, covering the following modules.
- Module Timeline:
• Week 36: Module 1 - Welcome to AWS Academy Data Engineering
• Week 37: Module 2 - Data-Driven Organizations
• Week 38: Module 3 - The Elements of Data
• Week 39: Module 4 - Design Principles and Patterns for Data Pipelines
• Week 40: Module 5 - Securing and Scaling the Data Pipeline
• Week 41: Module 6 - Ingesting and Preparing Data
• Week 42: Module 7 - Ingesting by Batch or by Stream
• Week 43: Module 8 - Storing and Organizing Data
• Week 44: Module 9 - Processing Big Data
• Week 45: Module 10 - Processing Data for ML
• Week 46: Module 11 - Analyzing and Visualizing Data
• Week 47: Module 12 - Automating the Pipeline

Student Responsibilities
1. Class Participation and Assignments:
• Active participation in all classes, including the completion of in-class assignments, which must be submitted during class hours.
2. Homework Assignments:
• Option A: Complete eight individual homework exercises, partially demonstrated during contact sessions.
• Option B: Complete the full AWS Academy Data Engineering course as a substitute for the homework assignments. To do this, students must follow the weekly schedule and upload their AWS Academy course certificate to the Itslearning platform.
3. Final Project:
• A group project (3-4 students) to be completed over Weeks 46 & 47, culminating in a presentation in Week 48.
________________________________________
Additional Notes
• Flexibility: The option to replace homework with the AWS Academy course allows students to tailor their learning experience to their interests and career goals.
• Integration of AWS: The inclusion of AWS Academy in both the core and optional parts of the course provides a strong foundation in cloud-based data engineering, which is highly relevant in today's industry.
• Project Work: The group project encourages collaboration and the practical application of the skills learned throughout the course.

Tavoitteet

After completing the course the student is able to:
Understand and describe the data engineering process life cycle

Sisältö

What is Data Engineering
Data Storage and Retrieval
Data Engineering Lifecycle
Extract, Transform and Load (ETL) process
Introduction to Big Data Frameworks

Oppimateriaalit

- The learning materials including slides and exercises will be prepared by the lecturer from various sources such as online courses and articles, books, videos, etc. The material will be introduced during the lectures and will be available via the learning environment (ITS).

- AWS Academy Data Engineering [91081] Course Materials

- Recommended books:
1. Data Engineering with Python: Work with massive datasets to design data models and automate data pipelines using Python, Crickard III, Paul, Packt Publishing, 2020.
Slides provided by teacher can be found via Itslearning.

2. Fundamentals of Data Engineering, Plan and Build Robust Data Systems
By Joe Reis and Matt Housley, Publisher: O’Reily, First edition, 2022.

Opetusmenetelmät

- Participating in lectures (theory and practice)
- Learning through hands-on programming (classwork assignments)
- Completing homework assignments or AWS Academy Course
- Interacting with the teacher and classmates
- Enhancing knowledge through teamwork projects
- Following the flipped-classroom model (pre-session self-study of theoretical concepts followed by in-class practical application)

Tenttien ajankohdat ja uusintamahdollisuudet

No exam, and retake not possible after evaluation grade is published.

There is a final teamwork project where students must demonstrate their work during a presentation event in week 48.

Kansainvälisyys

- The course includes approximately 12 theory and practice sessions, where students engage with practical tasks.
- Additionally, there are 4 online Q&A sessions to provide extra support.
- Homework exercises will be assigned, with some parts demonstrated during contact sessions.
- Integration of Cloud-based data engineering through the AWS Academy course.
- A teamwork project will be introduced in the second month, requiring students to apply their teamwork skills and the knowledge gained from the course to implement their final project.
- A flipped-classroom model may be used for some lectures, where students study the theoretical content at home and focus on practical implementation and discussions during class.

Toteutuksen valinnaiset suoritustavat

The practice works and exercises are mainly performed using VS Code, Jupyter Notebook, Apache Airflow, and AWS services.

Opiskelijan ajankäyttö ja kuormitus

- Contact teaching:
• We have 12 theory and practice sessions, each lasting 3 hours, conducted weekly: 12 x 3 = 36
• Additionally, there are 4 online Q&A sessions, each lasting 1 hour.
• Total contact teaching hours per course: 40 hours.

- Homework and teamwork assignment:
• Personal assignments (homework) and independent studies: 75 hours
• Teamwork assignment: 20 hours

Total: approximately 135 hours (5 x 27h)

Lisätiedot

Qualifications:
Before taking an "Introduction to Data Engineering with Python" course, students typically need a foundational understanding of several key areas. Here are the mandatory and recommended prerequisite courses and topics.

1. Mandatory Prerequisites: 
1.1. Programming:
1.1.1. Introduction to Programming: Knowledge of programming fundamentals,
including concepts like variables, loops, conditionals, and functions. 
1.1.2. Python Programming: Familiarity with Python, including basic syntax, data
types, control structures, and function and modules
1.1.3. Error Handling 
1.1.4. Object-oriented programming (OOP) 
1.1.5. Data Manipulation: Skills in using Pandas library including DataFrames and
Series, reading, writing, filtering, and transforming data
1.2. Databases: Knowledge of how databases work, including concepts like tables, keys, normalization, and indexing.

2. Recommended Topics:
2.1. Algorithms and Data Structures: Basic understanding of algorithms and data
structures such as arrays, lists, trees, and graphs, which are crucial for data
processing
2.2. Having the fundamental knowledge of cloud services or passing the Cloud Services
Course in TUAS (Lecturer: Ali Khan)
2.3. Version Control Systems: Basic understanding of tools like Git for version control.
2.4. Basic Algebra and Calculus: Fundamental math skills to handle data transformations
and calculations.
2.5. Statistics: Understanding of basic statistical concepts like mean, median, standard
deviation, and probability distributions.+