Accuracy 360

In Out Counting (Pro2)

Working Principle:

Tracking each human detected by AI algorithm
Constantly checking if the human is within the tracking zone
If the human crosses either the In-line or Out-line, a counting event is registered.
However, the event is ignored if the human makes a U-turn.
When the human leaves the tracking zone, the event is collected and sent.

Counting Line/Zone Definition

Data Integrity Report (Accuracy Validation)

Step 1: Click on "Settings"

The v9 page starts off with showing the "Reporting page", which then has to be navigated to the "Settings" page.

Step 2: Click on "Device"

Upon clicking on "Settings", head over to the left column and look for the "Device" button, and click on it.

Step 3: Use the "Search" Filter to Find the Specific Device to View the Report

Upon completing the steps, there is a search bar which contains filters that can be used to filter specific devices of the user's choice.

Step 4: Viewing the Device Info

Upon Completing the steps mentioned above, there will be a "Edit" for the filtered device, click on the "Edit" button.

Step 5: Viewing the Data Integrity Tab

Item	Description
1. Stage	Display the stage of the device.
2. Last Tuned Date	Display the last tuned date
3. Last Certified Date	Display the last certified date
4. Accuracy Certificate	Display the button to navigate to the accuracy certificate.
5. Accuracy Disclaimer	Display the accuracy disclaimer
6. Offline Video Schedule Count	Display the count of offline video
7. Available Offline Video Count	Display the count of the available offline video Note: Click the hyperlink to schedule Online Sim video

Step 6: Schedule Video (Click Hyperlink from previous Item 7)

Step 7: Verifying Video

Once the recording has been successfully scheduled, the Audit button will become available for the recorded video(s). Clicking the Audit button will navigate you to the Accuracy Audit page.

On this page, you can perform the audit by using the provided keyboard shortcuts for efficient counting. The system will display both System Count and Manual Count, allowing easier comparison and assessing the accuracy.

Accuracy Certificate Report

This is a guide on checking for certificate reports after a device has been certified, along with the certificate reports of sub-channels (if any) will all be shown in this menu. It is crucial to ensure that v9 has been logged into to view the v9 portal.

Step 1: Click on "Settings"

The v9 page starts off with showing the "Reporting page", which then has to be navigated to the "Settings" page.

Step 2: Click on "Device"

Upon clicking on "Settings", head over to the left column and look for the "Device" button, and click on it.

Step 3: Use the "Search" Filter to Find the Specific Certified Device to View the Report

Upon completing the steps, there is a search bar which contains filters that can be used to filter specific devices of the user's choice.

Step 4: Viewing the Accuracy Report(s) Included during Certification

Upon Completing the steps mentioned above, there will be a "View Accuracy Report" for the filtered device, click on the "View Accuracy Report" button to view the amount of channels of the device and their report links.

Step 5: Select a Report to View the Certificate Included During Certification

Step 6: Viewing the Details Included in The Certificate Report of the Certified Devices or Channels

After clicking on "View Report", the page will be redirected to the Certificate report that has been included, full details, and video proof(s), will be shown in the report.

Staff Exclusion

The Staff Exclusion Validation Workspace is designed to help users verify and improve the accuracy of staff exclusion within the people counting system.

In many retail environments, employees may tend to move in and out of the store during their shifts. Without proper identification and exclusion, staff movements may be incorrectly counted as shopper traffic, which can significantly distort analytics such as:

Actual customer footfall in
Store conversion rates
Customer traffic trends

This section focuses on how users can validate, benchmark, and confirm that staff members are correctly excluded from shopper traffic counts using annotated images and comparison tools, whilst feeding the model accurate data to improve future predictions.

The system will automatically generate machine-inferred predictions that estimate whether a person entering the store is a staff member or a visitor based on an AI model. However, validation can be done:

If there are doubts about the accuracy of the machine-inferred results.
To help improve our future predictions for a specific site by correctly identifying staff and visitors for further model training.
To understand the different movement trends of customers and visitors in a store.

The process of the Validation Workspace enables users to compare our current machine predictions with ground truth annotations collected from scheduled image campaigns, and users can review differences and determine whether the model is accurately distinguishing staff from visitors via graphs, summaries, and path analysis tools, to name a few.

By analysing this data, users can validate the effectiveness of staff exclusion and improve the reliability of shopper traffic metrics for their store environment.

Overview

Users can use the Overview Section to obtain an overall high-level summary of the number of staff being predicted and excluded when entering the store. This page also provides a visual comparison between machine-inferred predictions and ground truth annotations, allowing users to quickly evaluate model performance and identify discrepancies.

The data is presented through a series of charts and summary panels that highlight:

Hourly distribution of staff and visitors
Total visitor and staff predictions in a day
Differences between machine-inferred results and annotated data

The validation graph compares the number of annotated images (Ground Truth) against the machine-inferred predictions.

This comparison allows users to identify potential differences between:

Predicted staff counts
Actual staff counts identified during manual annotation

These differences can be analysed on an hourly basis, helping users determine whether the model consistently misclassifies individuals during certain periods of the day.

Users can also view a daily summary of machine-inferred predictions of the number and trend of the shoppers and staff by selecting a date range, as demonstrated in the picture below. This summary shows the overall trend of predicted staff and visitors.

The pie charts provide a quick visual representation of the distribution between staff and visitors based on both machine-inferred predictions and ground truth annotations.

By comparing these distributions, users can quickly determine whether the system's classification performance aligns with the manually validated data, where a discrepancy of 5% is an acceptable difference to take into consideration.

If significant differences exist, further annotation campaigns may be required to retrain the model.

Scheduler

Users can schedule a campaign via the Scheduler Section, where images of each person entering the store at the scheduled hour are captured. To do so, press the "New Schedule" Button.

In order to schedule a campaign, users would need to:

When creating a schedule, users need to

Device Serial Number - identifies the specific people counting device.
Channel Number - provides the specific channel(s) needed to be recorded.
Recording Dates - the day(s) on which the recordings should be performed on.
Maximum Number of Images - the limit for the total number of images captured during the campaign.

To obtain useful training data, it is recommended to schedule campaigns during periods with a mixture of staff and customer activity, such as:

Predicted staff counts
Actual staff counts identified during manual annotation

Once successfully recorded, a list of rows are produced, with a Harvest Status "Pending". To harvest a campaign, or to turn a recording into a list of ready-to-use annotations, select the rows that would need to be harvested and press "Start Harvest" to harvest a single schedule, or "Bulk Harvest" for multiple campaigns.

Once harvesting is done, there should be 2 statuses in the "Harvest Status" Column:

Ready: The row is ready for training creation.
Failed: The harvesting failed, and is not possible for training creation.

There will be row(s) available for annotation to be done, which can be selected on and begin creating an annotation training dataset by pressing "Create Training"

Training

In the Training Section, users can perform manual annotations on captured images from scheduled campaigns.

These annotations form the ground truth dataset, which is used to validate and improve the machine learning model responsible for identifying staff members.

Each campaign contains a collection of images captured during the scheduled recording window. The number of captured images should generally correspond with the number of detected entries into the store during that time period.

There are several use cases for the training workspace, namely for creating a ground truth to be compared with in the Overview section, or to provide additional training to improve the accuracy of the model.

This retraining of the model would be helpful in several operational scenarios:

If a store layout changes (e.g., relocating entrances, installing barriers, or modifying checkout areas), movement patterns may change and require revalidation.
If previously distinct staff uniforms change or staff no longer wear identifiable clothing, the system may require retraining to maintain classification accuracy.

To begin annotating a campaign, users can click the "Annotate" button.

The goal of the annotation process is to classify each captured individual as one of the following:

Staff Member
Visitor
Disqualified Image

These labels allow the system to classify and then train the model via visual and behavioural features that are associated with staff members and visitors.

How to Perform Annotations

Users can annotate on a campaign by identifying and annotating the images captured as a staff member or a visitor.

If the image shows a staff member, select all of the staff images and press the "1" button.

If the image shows a visitor, no action is required, as visitors are considered the default category. However, if you wish to change the status from "staff" or "disqualified", select all relevant images and press the "2" button.

Disqualified Images

Some captured images may not be suitable for model training. These should be marked as Disqualified, by selecting all disqualified images and selecting "q"

Examples include:

Extremely blurry images
Images where the person is extremely obstructed
Images where the person cannot be clearly identified
Images where the image is falsely captured as a human
Images with extreme lighting conditions

Below are some examples of images that are likely to be disqualified:

Using the Map View

For additional context, users may click on a selected image and open the Map View.

The Map View displays:

The movement path of the detected individual
Up to three images captured during different moments of the same trajectory

By observing these movement patterns, users can make more accurate annotations.

Annotation Status

Once annotation is completed, the user can click on "Save Annotation Progress", and an option to change the stage to 3 different categories can be made.

These statuses are:

In Progress: The campaign annotation is partially completed or requires further review.
Completed: All images in the campaign have been successfully annotated and are ready for training.
Not Useful: The campaign data is unsuitable for model training (e.g., poor image quality or incorrect scheduling).

Risk Map

The Risk Map Section allows users to analyse movement behaviour and spatial patterns derived from the annotated data.

Using this data, the system generates visualisations that help users compare how staff and visitors move within the store environment.

Two main visualisation modes are available:

Heat Map: Highlight areas with high movement or dwell density.
Linked Paths: Allows for viewing of each individual path taken by a person.

Users can toggle between staff-only data and visitor-only data to observe behavioural differences.

Analytical Tabs

Several other tabs also allow for the comparison of data between staff and visitors, which can be accessed via the 4 tabs:

Path Behavioural: Data being derived from overall movement patterns to understand how individuals navigate and interact with a space.
Spatial Distribution: Where movement is primarily located within the space by capturing the average position and the furthest points reached along both the X and Y directions.
Motion: Data derived from how fast and consistently a person moves, including their overall speed, peak speed, speed variability, and total time spent being tracked.
Visual: Determine the type of person entering the store based on visual cues.

Staff Validation

The goal of the validation process is to ensure that the model that was previously trained is accurate and produced optimum results. Similar to the staff annotation page, there are 2 different classes of validation, which are:

Staff Member
Visitor

However, these classes are already organised according to the model's inferred results, and it is the annotator's job to do a 'cross check' to see if they are indeed in the right categories, hence the term validation.

How to Perform Validations

The validation page presents the classification results for the two classes, together with the corresponding input images and prediction confidence scores. A prediction score greater than 50% indicates that the input image is classified as the "staff" class, whereas a prediction score of less than 50% or below indicates classification into the "visitor" class.

The prediction score also provides an indication of the model's confidence in its classification. Scores that deviate further from the 50% threshold represent greater confidence in the predicted class, while scores closer to 50% indicate greater uncertainty in the classification.

Users can validate images within a campaign by reviewing the captured images and assigning the appropriate class labels. If an image depicts a staff member, the user should select all relevant staff images and press the "1" key. If an image depicts a visitor, the user should select all relevant visitor images and press the "2" key.

Some captured images may be unsuitable for use in model training. These images should be marked as Disqualified by selecting the relevant images and pressing the "q" key. Examples of images that should be disqualified are provided in the "Disqualified Images" subsection of Section 2.3.1. Disqualified images are excluded from the model training and are therefore not considered when evaluating the model's accuracy.

If an image has been assigned an incorrect class and its status needs to be changed, the user can select the relevant image(s) and assign the appropriate class by selecting the corresponding class. The validation page will be updated accordingly, as illustrated in the figure below.

This validation process allows users to verify whether the model's predictions are correct by comparing the predicted classifications against the manually assigned labels. The resulting validation data can then be used to assess whether the model is performing at an acceptable standard or whether further configuration, validation, or model training is required.

Demographic

Gender Classification

The Gender Validation Workspace is designed to help users verify and improve the accuracy of the Gender Classification within the people counting system.

In many retail environments, people are always moving in and out of the store during operating hours. Without proper identification, the gender distribution may be incorrectly counted, which can significantly distort analytics such as:

Gender Distribution
Demographic trends

This section focuses on how users can validate, benchmark, and confirm the overall gender distribution of an area using annotated images and comparison tools, whilst feeding the model accurate data to improve future predictions.

The system will automatically generate machine-inferred predictions that estimate whether a person entering the store is a Female or Male based on an AI model. However, validation can be done:

If there are doubts about the accuracy of the machine-inferred results.
To help improve our future predictions for a specific site and can be replicated and reused for multiple sites of the same company by correctly identifying a person's gender, used for further model training.

The process of the Validation Workspace enables users to compare our current machine predictions with ground truth annotations collected from scheduled image campaigns, and users can review differences and determine whether the model is accurately predicting the right gender of a person. The workspace mainly visualises the data through the overview graph that compares both machine inferred and ground truth annotation.

By analysing this data, users can validate the effectiveness of the AI model and improve the reliability of gender data for their store environment.