Commerce

Search for what you see, not what you type

Search for what you see, not what you type

SAP Commerce cloud is already a great solution when it comes to ecommerce functionality, integration capabilities, addons, api’s and its robustness. Complimented with additional emerging functionalities not only compliment the business success but can enhance the customer loyalty as well, we will see how with one example of integrating the tensorflow for image search

About Author: Sandeep Chakka

Sandeep Chekka is a certified professional in SAP Commerce Cloud, Marketing Cloud, and Cloud Platform Integration, he is having 6+ years of experience in web development, cloud platform integration and platform architecture design. He is very passionate about exploring AI and Machine Learning library’s to create a personalized customer experience in the eCommerce platform.

Visual search goes back to a decade when Google first introduced their “Google Glasses” but how far did the market adapted, probably null and the demand never picked up forcing google to eventually surpass the efforts and reinvesting to latest and emerging technologies.

Looking at the market trends, companies like Microsoft, Alphabet, Facebook, Nvidia, AMD, and Amazon are heavily investing on usage of data and built artificial intelligent systems. On the contrary companies like e-commerce platforms will only supplement the trend with connectors, addons and hooks in the technical world. One of such kind of extensions include, Alexa, Hi Google, Image Search and or Visual search.

The ecommerce solutions are struggling to innovate into the visual world digitally and tensorflow simplifies this with the built-in algorithms and patters to make the visual transformation and searches seamlessly. Several companies have innovated into different technologies, the example below explains in detail how an SAP Commerce can be integrated to data patters, image patters using Tensorflow for a visual image search

Before we get to know the integration, it is important to understand the underlying details on Tensorflow and the technologies like Neural networks which is the base foundation for Machine Learning

Neural Networks, a Base foundation for Tensor flow Visual Patters

  • Neural Network is a computational model that works in a similar way to the neurons in the human brain. Each neuron takes an input, performs some operations then passes the output to the following neuron.
  • Computers are able to perform computations on numbers and is unable to interpret images in the way that we do. We have to somehow convert the images to numbers for the computer to understand.
  • There are two common ways to do this in Image Processing:
  • Using Greyscale, Using RGB Values.
  • Colors could be represented as RGB values (a combination of red, green and blue ranging from 0 to 255). Computers could then extract the RGB value of each pixel and put the result in an array for interpretation.
  • When the computer interprets a new image, it will convert the image to an array by using the same technique, which then compares the patterns of numbers against the already-known objects. The computer then allots confidence scores for each class. The class with the highest confidence score is usually the predicted one.
  • One of the most popular techniques used in improving the accuracy of image classification is Convolutional Neural Networks (CNNs for short).
  • In this technique instead of feeding the entire image as an array of numbers, the image is broken up into a number of tiles, the machine then tries to predict what each tile is. Finally, the computer tries to predict what’s in the picture based on the prediction of all the tiles. This allows the computer to parallelize the operations and detect the object regardless of where it is located in the image.
  • For a better accuracy we need train model with images as many as possible.
  • We can analyze the logs for accuracy and to understand how the model is getting trained.

What and who is TensorFlow

  • TensorFlow is an open-source machine learning library for research and production. It offers APIs for beginners and experts to develop for desktop, mobile, web, and cloud version.
  • It provides many library’s one of them is Image classification.
  • TensorFlow Lite is lightweight solution for mobile and embedded devices.

Search with real time image capturing using TensorFlow/Keras
TensorFlow is a framework by Google for creating deep learning models. Deep Learning is one of several categories of machine learning (ML) models that use multi-layer neural networks. The TensorFlow library allows users to perform functions by creating a computational graph.
TensorFlow library’s offers a feature called image classification which is used to predict the image with the trained dataset and provide the most accurate search results. Below is the technical overview on how to integrate and train Tensorflow.
This can be integrated with any eCommerce site in three ways.

  1. Using JavaScript library(tensorflow.js)
  2. Importing the library’s into java code base.
  3. For Android and IOT setup

A general overview of how the user interaction works is captured using image search

In the picture above, the electronics image file can be substituted with any article image file, electronics taken as an example to show the ease of integration

Integration of TensorFlow with SAP Commerce 6.5 and above using JavaScript.

If you take SAP Architecture and Tensorflow integration in place, the architecture example can be portrayed as below

 

With all the components in place, what is the typical implementation looks like? that is a million-dollar question, below are some of the technical considerations for implementation

  • Python, image dataset and TensorFlow code base are the prerequisite.
  • TensorFlow was first trained with few electronic image dataset for each product of Hybris eCommerce application.
  • TensorFlow reads each image in 0’s and 1’s as a 2d image.
  • Product Images were stored creating a folder and the folder name should be the same as product name.
  • Each product should be trained with 1k images for better accuracy.
  • TensorFlow will provide two file once training gets completed, *graph.pb and *_labels.txt files.
  • The pb files is the trained model and the labels file contains the names of products which are trained.
  • This pb file has to be converted to lite file which is used which is used for integrating with Android application.
  • We have enabled camera in search bar and hitting TensorFlow with the captured image.
  • If it find the product in the trained dataset with 95% accuracy it will do a Solr search in Hybris with the product name and provide search results.
  • For now we have showing the matching product and relevant products but, we can even configure the search results page with exact match item.

Before I finalize, one of the best examples on coding ML programs without dealing directly with Tensors

Built on top of TensorFlow.js, the ml5.js library provides access to machine learning algorithms and models in the browser with a concise, approachable API.
Prerequisites:

  • Hybris 6.5 up and running
  • Python 3.6 and above installed
  • NPM
  • Pip

Part of TensorFlow library, ml5.js aims to make machine learning approachable for a broad audience of artists, creative coders, and students. The library provides access to machine learning algorithms and models in the browser, building on top of TensorFlow.js with no other external dependencies. Typical development process can be done by following the steps as described below:
Step 1: Create an HTML file and add the below HTML content and script to import library.
<!DOCTYPE html>

<html>

<head>

<title>Getting Started with ml5.js</title>

<script src=”https://unpkg.com/ml5@0.1.3/dist/ml5.min.js”></script>

</head>

<body>

<script>

// Your code will go here

</script>

</body>

</html>

 

Step 2: Add something more to classify an image using the pre-trained Mobile Net model.
<!DOCTYPE html>

<html>

<head>

<title>Getting Started with ml5.js</title>

<script src=”https://unpkg.com/ml5@0.1.3/dist/ml5.min.js”></script>

</head>

<body>

<script>

// Your code will go here

</script>

</body>

</html>

 

<!DOCTYPE html>

<html>

<head>

<head>

<title>Getting Started with ml5.js</title>

<script src=”https://unpkg.com/ml5@0.1.3/dist/ml5.min.js”></script>

</head>

<body>

<h1>Image classification using MobileNet</h1>

<p>The MobileNet model labeled this as

<span id=”result”>…</span> with a confidence of

<span id=”probability”>…</span></p>

<img src=”https://ml5js.org/docs/assets/img/bird.jpg”

crossorigin=”anonymous” id=”image” width=”400″>

<script>

// The image we want to classify

const image = document.getElementById(‘image’);

// The result tag in the HTML

const result = document.getElementById(‘result’);

// The probability tag in the HTML

const probability = document.getElementById(‘probability’);

// Initialize the Image Classifier method with MobileNet

const classifier = ml5.imageClassifier(‘MobileNet’, function() {

console.log(‘Model Loaded!’);

});

// Make a prediction with the selected image

// This will return an array with a default of 10 options with their probabilities

classifier.predict(image, function(err, results) {

result.innerText = results[0].className;

probability.innerText = results[0].probability.toFixed(4);

});

</script>

</body>

</html>

Step 3: Open it in a web browser and you should see something like this (note it may take a few seconds for the model to load!)

An example for Image prediction using the existing trained dataset:
The examples are meant to serve as an introduction to the library and machine learning concepts.
The url for the ml5.js library to copy into an index.html file is:
<script src=”https://unpkg.com/ml5@0.1.3/dist/ml5.min.js” type=”text/javascript”></script>

Image Classification

Image classification example using MobileNet. MobileNet is a machine learning model trained to recognize the content of certain images.
Code: // Initialize the Image Classifier method with MobileNet. A callback needs to be passed.
const classifier = ml5.imageClassifier(‘MobileNet’, modelReady);
// A variable to hold the image we want to classify
let img;
function setup() {
noCanvas();
// Load the image
img = createImg(‘images/bird.jpg’, imageReady);
img.size(400, 400);
}
// Change the status when the model loads.
function modelReady(){
document.getElementById(‘status’).html(‘Model Loaded’)
}
// When the image has been loaded,
// get a prediction for that image
function imageReady() {
classifier.predict(img, gotResult);
// You can also specify the amount of classes you want
// classifier.predict(img, 10, gotResult);
}
// A function to run when we get any errors and the results
function gotResult(err, results) {
if (err) {
console.error(err);
}
// The results are in an array ordered by probability.
select(‘#result’).html(results[0].className);
select(‘#probability’).html(nf(results[0].probability, 0, 2));
}
Bottomline,

One of the more exciting applications of machine learning is going to be the ability for ecommerce solutions to search for the things when you see it. Right now, to find a product on an e-commerce store, you must search for it using the text, Imagine what there is a possibility to search for a product with ease by capturing real time image and the ecommerce store gives you the exact product you are looking for! Isn’t it cool right! Yes, this can be possible as explained above using TensorFlow.
This is the future, eCommerce revolution was built on the back of digital advances and automation, In the eCommerce space, machine learning is being used to slowly learn what customers prefer and how they want to see information to get them to make a purchase. It tests and adapts, using new options and information to slowly refine the best way to reach your customers.

About The Author