DSCI551 - Foundations of Data Management - Spring 23 - USC

Welcome to the comprehensive repository for DSCI 510 - Foundations of Data Management at USC during the Spring 2023 semester, taught by Professor Wensheng Wu. This centralized hub contains all coursework materials, including assignments and project solutions, organized into folders representing distinct modules covered in the course.

Within this repository, assignments and projects delve into fundamental concepts like data storage, file systems, relational vs. semi-structured data, SQL, and more. The course explores big data analytics, addressing NoSQL databases, cloud storage, distributed file systems, and MapReduce. Advanced topics like data cleaning, transformation, warehousing, and integration are also covered. It's essential to use this repository responsibly, refraining from engaging in plagiarism-related activities.

Tip

Before exploring the materials, take a moment to review the license and disclaimer for responsible utilization. The repository covers various topics, providing valuable insights and hands-on experience in Data Management.

Course Details:

• Course Name: DSCI 551 - Foundations of Data Management
• Instructor: Prof. Wensheng Wu
• Semester: Spring 2023
• About the Course:
  • Function and design of modern storage systems, including cloud; data management techniques; data modeling; network attached storage, clusters, and data centers; relational databases; the map-reduce paradigm.

Feel free to explore the assignments, projects, and solutions provided as learning aids. Whether you're a beginner or an experienced practitioner, this repository aims to be your companion in mastering the intersection of foundational data management fundamentals within Data Science & Engineering. Happy learning!

Caution

Please note that this repository serves as a reference guide and should be utilized as a tool for learning and comprehension. It's paramount to refrain from engaging in any activities associated with plagiarism. Embrace the wealth of knowledge herein to enhance your understanding and augment your skill set in Data Engineering.

Table of contents

Assignment	Topic Covered
Homework 1	Analyse Air Quality Index (AQI) data with Firebase Realtime DB
Homework 2	Emulate HDFS command line using Python
Homework 3	Understanding the usage of SQL and XML parsing on HDFS snapshot as XML document
Homework 4	Indexing and Querying on B+ Trees
Homework 5	Analysing AQI Index data of multiple countries using Hadoop MapReduce & Spark
---	---
Lab 1	Playing around with file system using the bash command line on AWS EC2
Lab 2	Setup HDFS on AWS EC2 and execute basic file system commands
Lab 3	Ingest and Query AQI data on MongoDB using Mongo Atlas
Lab 4	Analyse Books data on AWS DynamoDB
---	---
Exam Solutions	Consists of PDF documents with snapshots of 2 midterms and final exam quizzes along with solutions
---	---
Project Documents	Consists of all documentation related to the development of the final project
Project Repository	Firebase Realtime Database Emulator using FastAPI and MongoDB

Authors

1. Kayvan Shah | MS in Applied Data Science | University of Southern California

LICENSE

This repository is licensed under the BSD 5-Clause License. See the LICENSE file for details.

Disclaimer

_{The content and code provided in this repository are for educational and demonstrative purposes only. The project may contain experimental features, and the code might not be optimized for production environments. The authors and contributors are not liable for any misuse, damages, or risks associated with using this code. Users are advised to review, test, and modify the code to suit their specific use cases and requirements. By using any part of this project, you agree to these terms.}