A DATA TRANSFORMATION-VISUALIZATION PROJECT

Hi, this is my final project for my DE course, where I'll be convert raw data into a more accessible format for extracting insights.

Table of Contents

• Project Goal
• Task
• Technologies Used
• How I Did It
• Data Report
• Improvements

Project Goal

• "In this project, we'll use dbt (Data Build Tool) and SQL on Google BigQuery for data transformation. Our goal is to convert raw data into a more accessible format for extracting insights. dbt, an open-source tool, will help us effectively transform data in our warehouses."
• "We'll use SQL for data management and Google BigQuery, a fully-managed, serverless data warehouse, for super-fast SQL queries."

Technologies Used

• dbt
• Google BigQuery
• Visual Studio Code for query statements
• Looker Studio for visualization

Task

• Collect raw data about a jewelry business.
• Data Ingestion.
• Transform data to dimensional model in Bigquery.
• Visualize the data.

How I Did It

• Research the raw data.
• Create a virtual machine on Google Cloud compute engine, where I'll be storing raw data on Mongodb.
• Crawl missing data from the business website (product_names, location, ect.)
• Load data into GCS and Bigquery using Cloud Function.
• Transform data using dbt.

Or you can view the model here to see more description.

• Load transformed data from Bigquery into Looker Studio for visualization.

Data Report

Or click here to view the report.

Improvements

My code is still not fully optimal yet, any feedback would be very helpful! Thank you!