Visual Inventory Intelligence — Multimodal AI Project

Applied AI · Multimodal AI · Prompt Engineering · Workflow Design

View Site

Project overview

Visual Inventory Intelligence is a multimodal AI concept designed to convert photographs of physical environments into structured, reviewable inventory data.

The project develops practical exercises completed during generative AI and prompt-engineering coursework into a more complete operational workflow. Those exercises included scene inventory extraction, object classification, spatial reasoning, image comparison, visual change detection and constrained object-presence detection.

The resulting concept allows a user to photograph a storeroom, equipment installation, workspace or stock area. Multimodal AI analyses the image, identifies visible objects and organises the observations into a consistent inventory schema.

A later photograph can be compared with the original inspection to highlight items that may have been added, removed, moved or changed.

Human verification remains central to the process. AI-generated observations are treated as provisional until they have been reviewed and approved.

The business problem

Businesses frequently use photographs to document equipment, stock, storage areas and changes between inspections.

Although photographs preserve a visual record, the information inside them is difficult to search, compare or analyse systematically. Employees must manually examine each image, identify relevant objects, describe their location and transfer their findings into a spreadsheet or inventory system.

This creates several operational problems:

• Manual recording takes time.

• Different employees may describe the same object differently.

• Important items can be overlooked.

• Changes between inspections may go unnoticed.

• Photographic evidence remains disconnected from inventory records.

• Unstructured image archives are difficult to search.

• Manual reports can use inconsistent formats and terminology.

The challenge was therefore to explore how multimodal AI could accelerate visual documentation without removing the need for human judgement.

Project objective

The project was framed around the following question:

How might multimodal AI transform photographs into consistent operational records while preserving traceability, uncertainty and human control?

The objective was not to create an autonomous stock-control or safety-inspection system.

Instead, the project investigates how AI can assist with:

• Initial object identification

• Consistent data entry

• Spatial documentation

• Visual change detection

• Review prioritisation

• Structured inventory exports

Coursework foundation

The concept is grounded in practical multimodal AI exercises completed during the associated learning programme.

Scene inventory extraction

A photograph was analysed to identify the visible objects and convert them into an itemised inventory.

The example equipment-cupboard image included:

• Unvented hot-water cylinder

• Expansion vessel

• Copper pipework

• Pressure gauge

• Isolation valves

• Circulation pump

• Electrical control unit

• Electrical wiring

• Towel stack

• Cupboard enclosure

Spatial reasoning

The same photograph was queried again to describe how the objects were positioned relative to one another.

Example relationships included:

• Expansion vessel above the hot-water cylinder

• Copper pipework surrounding the cylinder

• Pressure gauge mounted beside the expansion vessel

• Circulation pump positioned below and to the left

• Electrical control unit mounted on the adjacent wall

• Cupboard enclosure surrounding the installation

Visual change detection

Two photographs of the same environment were compared to identify visible differences.

The change taxonomy included:

• Added

• Removed

• Moved

• Orientation changed

• State changed

• Unchanged

• Uncertain

Constrained visual detection

A reusable detector prompt was developed to determine whether a specified object was present in an image.

The output was deliberately constrained to a predictable YES/NO format, with explicit instructions for handling uncertainty.

These exercises demonstrate the underlying AI capabilities. The full application described in this case study is a portfolio extension created from that learning.

Proposed solution

Visual Inventory Intelligence combines the individual exercises into a six-stage operational workflow:

Capture → Analyse → Structure → Compare → Verify → Export

1. Capture

A user photographs or uploads an image of a defined physical area.

The interface asks the user to identify the location, inspection type and relevant inventory categories before analysis begins.

2. Analyse

The multimodal model examines the photograph and identifies visible objects that meet the inspection criteria.

It can also describe approximate locations and relevant spatial relationships.

3. Structure

Detected objects are converted into a consistent inventory schema.

Recommended fields include:

• Item ID

• Item name

• Category

• Material

• Approximate location

• Relationship to nearby objects

• Source image

• Qualitative confidence

• Review status

• Reviewer correction

• Inspection date

4. Compare

When a later image is uploaded, the system compares the two inspection records and proposes visible changes.

Each change is linked to the relevant source images and inventory records.

5. Verify

Users review the proposed inventory and visual differences.

Low-confidence observations, conflicting labels and potentially important changes are placed into a verification queue.

6. Export

Approved records can be downloaded as CSV or transferred into an existing spreadsheet or inventory-management process.

Interface structure

The proposed experience contains seven connected views.

Operations dashboard

The dashboard provides an overview of:

• Recent inspections

• Locations awaiting review

• Total visible inventory

• Detected changes

• Low-confidence observations

• Completed exports

Its purpose is to help the user understand where attention is required rather than presenting AI output as automatically correct.

Mobile image capture

A mobile interface guides the user through photographing an environment.

The experience includes framing guidance, image-quality checks, location selection and inspection notes.

Users can take a new photograph or choose an existing image without completing inventory fields manually.

Image-analysis workspace

The uploaded photograph is displayed alongside the generated inventory.

Selecting an inventory row highlights the corresponding area of the image. Selecting an image annotation opens the related structured record.

This two-way relationship makes the AI output easier to inspect and correct.

Spatial relationship view

The spatial view documents how visible objects relate to one another.

Examples include:

• Above

• Below

• To the left of

• To the right of

• Attached to

• Mounted beside

• Surrounding

• Inside

• Enclosed by

Spatial descriptions are presented as observations requiring review, particularly when perspective or partial visibility creates ambiguity.

Change-comparison view

Two inspection images are presented side by side.

The system highlights possible changes and organises them into a structured report.

Example output:

The comparison interface distinguishes clear observations from uncertain or visually ambiguous changes.

Human-verification queue

The review queue prioritises records that need attention.

A reviewer can:

• Confirm the proposed item

• Edit the item name

• Change its category

• Correct its location

• Reject a false detection

• Mark an item as obscured

• Request another photograph

• Add a reviewer note

Approved records retain the original AI observation and the final human-reviewed value, creating a visible audit trail.

Export and reporting

The export view displays the validated inventory, schema status and unresolved review items.

Users can export:

• Full inventory

• Change report

• Uncertain observations

• Human corrections

• Inspection summary

• CSV-formatted data

Prompt-engineering approach

The project demonstrates prompt engineering as workflow design rather than a collection of isolated questions.

Persona Pattern

The model is instructed to behave as a visual inventory analyst working within a defined operational scope.

Template Pattern

A fixed schema controls the output structure and helps maintain consistency across different images.

Semantic Filter Pattern

The system can be instructed to include only objects relevant to the inspection, reducing irrelevant scene descriptions.

Recipe Pattern

The image-processing workflow follows a defined sequence:

1. Identify visible objects.

2. Classify relevant items.

3. Describe their approximate location.

4. Record spatial relationships.

5. Flag uncertain observations.

6. Return the required structure.

Cognitive Verifier Pattern

Potentially uncertain conclusions are separated into verification questions rather than being presented as confirmed facts.

Fact Check List Pattern

Items requiring confirmation are placed into a review list with links to the relevant source image.

Combined Prompt Patterns

These patterns operate together as a repeatable workflow, supporting more consistent results than a single open-ended image-analysis request.

Example inventory schema

Item_ID,Item_Name,Category,Location,Relationship,Confidence,Review_Status
001,Hot Water Cylinder,Plumbing Appliance,Right Side,Inside Cupboard,High,Confirmed
002,Expansion Vessel,Plumbing Component,Upper Left,Above Cylinder,High,Confirmed
003,Copper Pipework,Plumbing Pipe,Throughout,Surrounding Cylinder,High,Confirmed
004,Pressure Gauge,Plumbing Control,Upper Left,Mounted Beside Vessel,Medium,Review Required
005,Electrical Control Unit,Electrical Component,Lower Left,Mounted Beside Pipework,Medium,Review Required

Confidence values are qualitative model judgements. They are not presented as calibrated statistical probabilities.

Human-in-the-loop design

Human verification is a core product requirement rather than an optional safeguard.

The system separates three types of information:

Observed

Information directly visible in the photograph.

Example: a cylindrical metal object is present on the right side of the cupboard.

Inferred

A likely classification based on appearance and context.

Example: the object appears to be an unvented hot-water cylinder.

Verified

An observation or classification confirmed by a human reviewer.

This distinction reduces the risk of inferred information being mistaken for an established operational fact.

Evaluation plan

The project will be tested against manually verified image inventories.

A suitable evaluation set would contain 10–20 photographs across several controlled environments. Each image would have a human-created reference inventory before being processed by the AI workflow.

Evaluation metrics

The evaluation will also document qualitative failure cases, including:

• Partially obscured objects

• Reflections

• Similar-looking components

• Poor lighting

• Perspective differences

• Cropped objects

• Changes in image position

• Objects visible in one photograph but hidden in another

Responsible AI considerations

Visual Inventory Intelligence is designed as a decision-support tool.

It should not independently make decisions concerning safety, legal compliance, equipment condition or stock valuation.

The proposed safeguards include:

• Human approval before records are confirmed

• Clear separation of observation and inference

• Source-image traceability

• Visible uncertainty labels

• Original AI output retained alongside corrections

• No automatic deletion of existing inventory records

• Review requirements for significant changes

• Defined limits on what the model is asked to identify

• Documented failure cases and evaluation results

Images may also contain people, personal information or sensitive workplace details. Deployment would therefore require appropriate consent, access controls, retention policies and image-handling procedures.

Skills demonstrated

The project brings together:

• Prompt engineering

• Multimodal AI

• Computer vision

• Object recognition

• Image comparison

• Visual change detection

• Spatial reasoning

• Structured data extraction

• CSV schema design

• Data validation

• AI workflow design

• Human-in-the-loop systems

• Responsible AI

• Evaluation planning

• Operational process improvement

• Interface and information design

Deliverables

The completed portfolio project will include:

1. Responsive project website

2. Mobile image-capture interface

3. Visual inventory dashboard

4. Image-analysis workspace

5. Spatial relationship interface

6. Before-and-after comparison view

7. Human-verification workflow

8. CSV inventory examples

9. Change-detection report

10. Reusable prompt library

11. Evaluation framework

12. Documented limitations and safeguards

Outcome

Visual Inventory Intelligence transforms a set of multimodal AI coursework exercises into a coherent operational concept.

The project demonstrates how photographs can become structured, searchable information through a repeatable AI-assisted workflow. More importantly, it shows how uncertainty, source traceability and human verification can be designed into that workflow from the beginning.

The concept is positioned as applied AI and workflow design rather than a production-ready computer-vision system.

It demonstrates the ability to:

• Identify a realistic operational problem

• Translate business requirements into structured AI instructions

• Design consistent data outputs

• Combine multiple multimodal capabilities

• Plan measurable evaluation

• Communicate limitations honestly

• Introduce appropriate human controls

• Turn course learning into a credible portfolio project

Recommended next steps

The next stage is to build the working demonstration around the equipment-cupboard scenario.

This will involve:

1. Creating a manually verified reference inventory.

2. Preparing two controlled inspection images.

3. Running the inventory-extraction workflow.

4. Running spatial-reasoning analysis.

5. Comparing the two images for visible changes.

6. Recording human corrections.

7. Calculating the evaluation metrics.

8. Designing the mobile and desktop interfaces.

9. Publishing the findings as a responsive portfolio experience.

All evaluation results will remain clearly separated from proposed functionality until the testing has been completed.