Market Intelligence

What is Market Intelligence?

            Market intelligence is the information relevant to a company’s market – trends, competitor, and customer monitoring gathered and analyzed specifically for the purpose of accurate and confident decision-making in determining strategy in areas such as market opportunity, market penetration strategy, and market development.

Why do we need Market Intelligence?

For a company, it has a huge amount of text data of its users. It has a collection of data talking about their brands, what their users wrote about their brand, is it good or bad everything present across many platforms. But still, the company isn’t able to understand their valuable customers, how their products perform in the market. By just tracking the number of sales it can’t conclude what the users think of their customers, it needs to analyze the feedback coming from the customers, and having this large amount of unstructured data becomes less relevant to draw some analysis. To analyze the market they need to understand the strategies and technologies used to analyze business information and make informed decision-making It helps create a better understanding of the global market as well as provide an insight into how your company operates. Understanding the needs of customers allows them to build better products and services which are welcomed by customers.

AI-based Market Intelligence System

            With the help of AI, we can build a system where we are able to track the sentiments, what people are talking about, how they react to the products, etc. To build a market intelligence system we will have to incorporate multiple models that will give us different pieces of information. Now we will see the process that we will use to build a market intelligence application.

Data Collection/Extraction

Before we start working on the application we need to collect the data. To collect data we will use technologies like web-scraping, data extraction from social media like Facebook, Twitter, etc. Then by using a data versioning like DVC we will keep track of the data we are collecting.

Data Annotation

Now once we have collected the data, we will have to prepare multiple datasets that we will use to train our models in the Model development part. When we mentioned the above intelligence and talk about other analytics that can

be drawn from the data, it’s always important to have well-annotated data. A well-annotated data helps you not only build models for sentiment analysis, intent classification, etc but also can help you in content enrichment. A powerful AI model heavily depends upon the quality of the data you give to the train. And in a system like market intelligence, you have a set of unstructured data that has multiple dimensions to annotate for like for a particular social media comment you want to have the aspect of the comment, sentiment related to the aspect, what topics are present in that comment and what’s the intent behind it.

As we will require datasets like sentiment analysis, Intent Classification, and topics we will start a hierarchical data annotation process where each text data can be annotated in multiple dimensions and we will arrange our data in such a way that we can easily extract the individual labels as per our requirement which is to extract the required labels for specific models.

For example for a single social media comment we will prepare structural data as below:

{   “text”: “comment”.   “annotation”: { “sentiment”: (aspect_label, sentiment_label), “intent”: “intent_label”, “topics”: “topic”, “keywords”: [“keywords”] } }

Data pre-processing

After studying the requirements of the application, we will start to preprocess the textual data which involves things like removal of stopwords, lemmatization, tokenization etc.

Model Development

Now once we have our dataset preprocessed we can start working towards building the model. We will show you how we will use NLP techniques like sentiment analysis (Aspect Based Sentiment Analysis), Intent Classification, topic modeling, etc in the application.

Market analysis through Sentiments

When a company has a larger audience it’s always important to keep track of the sentiments of the audience over time. Is their sentiment changing over time or it gets better. So to understand the sentiment you need to build a Classification model that will tell you the sentiments of the customers in real-time. It not only tells you what the sentiment is like instead of just saying positive, negative, and neutral it will tell you where it is positive or negative. For instance, it will tell you on a particular aspect what should be the sentiment. For example, it will tell you the sentiment of a product in the pricing aspect or quality aspect. In this way, the company can understand in which aspect they are getting negative reviews and in which positive.

To build such a model we will need to train a model which will first extract the aspects associated with the text and then find out what that aspect is talking about. By using models like BERT and constructing auxiliary sentences we can able to extract the aspects and sentiments.

Topic Modeling

Like to understand what the sentiment is, it’s also important to understand what people are talking about. It gives an insight to the company to understand the topics which are frequently discussed among the customers and understand their behaviors towards those topics. We have both probabilistic and deep learning models to extract topics and keywords from a text and we can use our annotated data to train such models and extract the related topics and keywords associated with a text.

Intent Classification

Intent classification is a way to understand the intentions behind customer queries, emails, chat conversations, social media comments, and more, to automate processes, and get insights from customer interactions. For example, identifying purchasing intent is pivotal in transforming sales leads into fully-fledged customers. Addressing customer’s requirements as soon as possible always keeps you ahead of your competitors. Intent classification allows companies to be customer-centric and understand their needs and solve them, especially in the support sector. We can treat this as a multi-label classification problem and by using Transfer learning methods on SOTA models we can train on our annotated data which can help us to come up with good accuracy.

Deployment, Visualize and Analytics

In this step, we will use our trained models to infer the input data and give us the necessary information to draw some visualizations. Below we have mentioned some of the examples of it.

• Aspect-based sentiment analysis through a drill-down bar chart
• Line graph to show the trends of customers either on social media or sells growth
• Bar with negative stack plot to show key positive and negative keywords used often through n-gram models
• Pie-chart to show the percentage of intents from the Intent Classification model E. A polar chart to show aspect-sentiment
• Like this, we can have many visualizations and Analytics based on the data and the problem statement.