# From a Shelf Photo to Product Matches: Designing a Retail Recognition Pipeline

Author: AlgoFly Team
Published: 2025-03-12T06:55:54Z
Canonical: https://algofly.ai/blog/retail-shelf-product-recognition

A shelf photo contains more than a set of objects. It contains repeated packages, almost\-identical variants, glare, price labels, and products partly hidden by their neighbours. Finding a rectangle around a bottle is useful, but a retail application often needs to know which bottle it is.


AlgoFly's shelf\-image product\-recognition video introduces an application for this task. Its public description discusses processing shelf images for product recognition and retail operations; it does not publish a model architecture or an evaluation dataset. The pipeline below explains a practical design for that kind of application and clearly separates detection from catalogue identification. It should not be read as an undisclosed benchmark or an exact reconstruction of the demo's backend.

![AlgoFly retail recognition demo with bounding boxes on shelf products and a product-match detail card.](/content/videos/xhSzQYSQ4qw.jpg)

The original demo image shows detected products, unique\-product counts, confidence controls, and a candidate product match. Displayed scores are individual demo outputs, not benchmark results.

## Detection tells you where; recognition tells you which item

Object detection proposes visible product regions. Recognition maps a crop to a catalogue item or leaves it unresolved. Combining these stages lets a detector learn common package shapes while a retrieval system handles a larger catalogue of individual products.


A detector can correctly locate a product even when the catalogue match is wrong. Conversely, an excellent product matcher cannot recover a package that was never detected or was cropped with most of its label missing. Evaluate the stages separately before interpreting a single end\-to\-end score.


For a first prototype, save the detected crop alongside its source image, bounding box, detector version, and score. That makes it possible to review whether a mistaken identification started with the crop or with the catalogue.

## Prepare reference images that resemble the real task

Clean pack shots are useful reference data, but a shelf camera rarely sees the same view. Reference images should cover meaningful packaging changes, product variants, side views where relevant, and the lighting conditions the application will encounter.


Keep the product identity separate from the image identity. Several reference images may describe the same SKU, and some catalogue entries may share nearly identical branding. Package volume, flavour, formulation, and bundle count can be the distinguishing information.


Retain metadata such as brand, product family, barcode if available, and package size. Visual similarity can narrow the candidate list; explicit metadata and readable text can help avoid merging visually similar variants.

## Use embeddings to produce candidates, not unquestionable answers

An embedding model turns a crop into a vector. A nearest\-neighbour index compares that vector with catalogue reference vectors and returns likely matches. The same preprocessing and embedding version must be used on both sides of the comparison.


If cosine similarity is the chosen metric, normalize vectors consistently. Faiss documents how normalized inner\-product search relates to cosine similarity. A returned score remains a similarity measurement; it is not automatically a calibrated probability that a package is the correct SKU.


Here is an illustrative exact\-search baseline for already\-computed vectors. It does not train an embedding model or prescribe the model used in the video:


```
import faiss
import numpy as np

references = np.asarray(reference_vectors, dtype="float32").copy()
queries = np.asarray(crop_vectors, dtype="float32").copy()
faiss.normalize_L2(references)
faiss.normalize_L2(queries)

index = faiss.IndexFlatIP(references.shape[1])
index.add(references)
scores, rows = index.search(queries, 5)
# Map returned rows to catalogue records; review ambiguous matches.
```
## An unresolved result is better than a confident wrong SKU

A production catalogue is incomplete. New packaging appears, shelves contain damaged products, and a camera may capture a promotion that was never indexed. A pipeline that always chooses the nearest item turns every unknown into an incorrect known item.


Introduce a review state based on validation data. Consider the top candidate's score, the gap to competing variants, crop quality, and whether the item belongs to a known product family. Set thresholds using held\-out shelf images, including products deliberately excluded from the catalogue.


A useful review interface shows the crop, a few candidate references, and an explicit “none of these” action. Corrections should preserve the original prediction and model version so later analysis can distinguish a model improvement from a catalogue edit.

## Counts and price labels require their own associations

Counting visible facings is different from counting inventory. A photo can show three front\-facing packs while several more sit behind them. Label the output as visible facings or detections unless another measurement supports a stock estimate.


Reading a shelf price is another task. OCR can identify text, but associating that text with the correct product needs geometry and layout context. A price label below one facing may span several products; a promotion may apply only to a subset.


Keep product detection, catalogue matching, OCR regions, and price\-to\-product associations as distinct records. This makes a user correction local: fixing one price association should not require pretending the entire image was recognized incorrectly.

## Test the cases that actually confuse the application

Build a test split by store visit, location, or collection session rather than randomly splitting near\-duplicate crops. Otherwise, almost\-identical images can leak across training and evaluation and hide how the system behaves on a new shelf.


Track detection recall, recognition accuracy on valid crops, unknown\-item rejection, and correction effort. Break results down by small packages, glare, occlusion, and look\-alike variants. The aggregate number can conceal the categories that make a retail workflow frustrating.


Latency has stages too: image upload, detection, crop embedding, catalogue retrieval, and rendering. Batch crop embeddings when practical, cache catalogue vectors by model version, and avoid recomputing unchanged reference data for every photograph.


The goal is an inspectable workflow that moves from a shelf image to useful, reviewable product information. The video provides the application context; a careful pipeline keeps every intermediate decision understandable enough to improve.

## Video and technical references

* [Watch the original AlgoFly video](https://www.youtube.com/watch?v=xhSzQYSQ4qw)
* [Faiss: distance metrics and cosine similarity](https://github.com/facebookresearch/faiss/wiki/MetricType-and-distances)
* [AlgoFly: image embeddings for similarity search](https://www.youtube.com/watch?v=sGQexJjjdms)
* [AlgoFly: retail product detection with YOLO\-NAS](https://www.youtube.com/watch?v=rlRPx8Sr914)