Chapter 12: Data Preparation

Data preparation is a crucial process in transforming the raw data into a format that is suitable for analysis or modelling. In our system, we have ready with some data preparation job for our customer to perform the transformation before analysis.

Job Type Applicable Metric Time Granularity Entity Level Description
[New Pipeline] Footfall Counting Data Patching
  • [PFC01_1] Footfall Count IN_All People
  • [PFC02_1] Footfall Count Out_All People
  • Hourly, then reaggregated to Daily
  • Site
Data cleaning for missing values or inconsistencies. Value updated in hourly value and later reaggregated to daily value.
[Old Pipeline] WiFi Counting Data Patching (Hourly)
  • [PWA01] Outside Traffic
  • Hourly
  • Site
Data cleaning for missing values or inconsistencies in WiFi Counting value. Hourly and Daily unique MAC will be different, no reaggregation required.
[Old Pipeline] WiFi Counting Data Patching (Daily)
  • [PWA01] Outside Traffic
  • Daily
  • Site
Data cleaning for missing values or inconsistencies in WiFi Counting value.
[Old Pipeline] Area In / Out Data Reaggregation
  • [PFC01_1] Footfall Count IN_All People AND [PFC02_1] Footfall Count Out_All People
  • Hourly and Daily
  • Area
Area footfall counting data reaggregation, to rerun the aggregation for hourly and daily, to correct late data issue.
[New Pipeline] In Out Data Correction for New Time Zone
  • [PFC01_1] Footfall Count IN_All People AND [PFC02_1] Footfall Count Out_All People
  • Hourly, then reaggregated to Daily
  • Site
  • Area
Corrective steps for wrong time zone set.
[New Pipeline] In Out Data Correction for New Operating Hour
  • [PFC01_1] Footfall Count IN_All People AND [PFC02_1] Footfall Count Out_All People
  • Hourly, then reaggregated to Daily
  • Site
  • Area
Corrective steps for wrong operating hour set.

 

Updated on March 19, 2024