How to scale GenAI products integrating machine learning best practices on high-quality metrics

Crazy news guys, we’ve just launched a startup program for AI founders (the perks are crazy).

You can get $2000 worth of credits (Anthropic, Mistral, OpenAI, and Phospho) + a call with our amazing team to guide you in your product-market-fit journey.

You can apply here.

We now see generative AI (genAI) products developed and shipped rapidly with the help of new AI tools and ML capabilities, but scaling them effectively requires the right approach.

With quality metrics and ML practices we can systematically improve the generation quality and overall performance of our genAI products with real-time data driven insights.

In this article we’ll be going over these three core principles to implement ML in genAI product development:

1. Data collection
2. Data analysis (extracted from text)
3. Data visualisation and feedback loops

Why are these principles important?

To scale and iterate as effectively as we can, we need to make sure we constantly improve our AI products in close alignment with our users. This requires the above approach to data.

Quality metrics are also required to benchmark this data against standards to accurately evaluate and improve our genAI products.

But all of this starts first with data collection.

1) Data Collection: The Foundation of Quality Metrics

Before we analyse any data for insights or train our AI models, we have to first collect data. It’s important to note here the need for high quality, diverse data sets in order to capture the nuance that comes with genAI outputs and user queries. These factors directly influence the accuracy and reliability of the metrics used in evaluating our GenAI products.

Diversity in our datasets helps our AI model generalise better and ultimately improve overall accuracy and relevance in its outputs. But the need for diverse data means we’ll need a balance of both qualitative and quantitative data, and for this we’ll need to leverage multiple different data sources. If we leverage the right ones we can also navigate the common pains associated with obtaining this data:

1) User feedback

****What better data than direct feedback from your users? In terms of data relevance we can’t collect data more useful than this.

2) Synthetic data (augmented)

Augmenting data basically means we take what data we already have and create variants of it for testing and training. This ‘synthetic’ data is a practical way of maximising and diversifying your existing data to train your AI model. You can also sidestep any privacy concerns with this data source as it’s completely synthetic.

3) Public datasets

Integrating external datasets we can access freely provides more data and context to help build your AI model’s understanding. This way of collecting more data also side steps any privacy concerns because it’s public, the only thing to make sure is for the data’s relevance to your AI model. Otherwise we risk skewing the accuracy of its outputs.

Unlike traditional data gathering, we’re dealing with large volumes of unstructured data from user interactions. Collecting this with traditional methods won’t cut it, in order to gather and translate this text data into any actionable insights, we need AI analytics tools.

For a full deep dive on data optimisation for AI softwares in 2024, read our previous article here.

2) Analyse and Extract Insights from Text

We’ve all been seeing AI evolve rapidly in a very short time. But what we haven’t seen is analytics tools evolve accordingly to keep up pace.

Traditional analytics tools are not capable of handling the scale of data, or capture the nuance of user interactions that are most common with AI integrations and chat-like functionality to provide real-time insights.

This is why we need to start using text analytics tools like Phospho which are specifically made for AI products to properly understand user inputs and ultimately improve the generation quality of our GenAI products.

To get an idea of the importance of AI analytics when building AI products, read our previous article here.

AI text analytics uses these three main techniques to get actionable insights from data:

1) Natural Language Processing (NLP)

User inputs are unstructured, so we need NLP to parse and structure this text for AI to ‘understand’ it.

2) Sentiment Analysis

Should be self explanatory, this is for determining the emotional tone in text to generate the right responses. We can use this to identify and quickly respond to any ‘negative’ interactions.

3) Topic Modelling

This is identifying and clustering main themes and patterns but it needs a large volume of text data.

These techniques provide clarity on generation quality scores which are metrics we can measure. This is important because if we can quantify them, we can evaluate outputs and compare them against expected standards.

Generation Quality Scores: Example Metrics

I didn’t define generation quality scores when I mentioned them above, but they are metrics that evaluate factors like coherence, relevance, and accuracy of generated content. By looking at these we can gauge which areas of our products need the most improvement.

Let’s take a look at some common generation quality metrics and what they mean:

1) BLEU (Bilingual Evaluation Understudy)

Measures the quality of machine translated text by comparing it to human translations.

2) ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

Evaluates text summarisation quality by comparing to human summaries, making sure it captured all the key points a human would include.

3) BERTScore (Bidirectional Encoder Representations from Transformers Score)

Computes the semantic similarity between two different sentences by converting words into numerical representations (contextual embeddings) and calculates a similarity score on a scale from 0 to 1, where 1 indicates perfect similarity. This nuanced evaluation of quality goes a lot further than simple word matching.

Academic Benchmarks: Standardised Evaluation

We can also look into academic benchmarks which are a standardised set of metrics to consistently compare performance across AI models and LLMs. You’ll find these metrics being used by the biggest LLM providers such as OpenAI’s Chat GPT, Anthropic’s Claude, and Google’s Gemini to compare themselves.

You can find a comparison between the above LLMs using these benchmark metrics in our previous article here and here.

Three popular benchmarks include:

1) MMLU (Massive Multitask Language Understanding)

Comprehensive test across a wide range of subjects and industries. Considered the gold standard for evaluating an AI model’s general knowledge and reasoning capabilities.

2) HellaSwag

Evaluates common sense reasoning and natural language inference, important metric for understanding and responding to real world scenarios as it tests for true understanding rather than relying on superficial patterns.

3) HumanEval

This specifically tests for coding understanding and ability, it’s evaluated based on functional correctness.

By using the above metrics and benchmarks we can ensure a standardised evaluation of our GenAI’s outputs against established datasets and definitively track improvements over time.

Hypothetical Example: Content Creation Assistant

Let’s consider a hypothetical SaaS that acts as a content creation assistant. Here you might use ROUGE scores to evaluate the quality of article content produced based on its ability to summarise large amounts of relevant data.

When tracking this score continuously, if we notice it drops below a certain threshold in performance we can trigger a retraining process. We could then test our further trained AI model against the MMLU benchmark to ensure it maintains a broad knowledge base which we’d need to produce diverse content for future articles. If the performance on MMLU decreases, it might indicate we need to expand the training data and re-think our approach to data collection (see principle 1 above).

By combining different metrics and benchmarks that are most relevant to your use case, you can continuously refine your GenAI’s outputs based on quantitative performance data to practically meet user needs.

3) Dataviz and Feedback Loops For Continuous Improvement Through Visualisation

There’s no two ways about it, we can get to answers faster with data visualisation. When it comes to understanding our AI product’s performance, real-time data visualisations make patterns and anomalies far easier to spot.

Data visualisation is one part of a larger process that comes from implementing a feedback loop, which typically works like this

1. Monitor performance
2. Collect and analyse feedback
3. Identify areas for improvement
4. Make necessary changes
5. Repeat

This cycle of constant monitoring and evaluation to feed into development cycles is crucial if you want to adapt quickly to changing user needs and market conditions. Something that can determine whether you maintain a competitive position in the market or not.

However, we shouldn’t rely too much on automated metrics and analytics. This is why we use Human evaluations (HumanEval) which involves real people assessing different outputs of AI models and choosing which they prefer.

Why is this important? Because it captures nuances that automated metrics can miss. For example, for an AI writing assistant, accuracy would score higher than creativity with benchmark metrics, but humans in this scenario might prefer more creativity.

It’s these types of insights from real human preferences and judgements that are best for fine tuning our AI models to have closer alignment with desired outputs.

We can instead use more subjective metrics through vibe checking which essentially helps to ensure the AI model’s outputs are aligned with the users’ expectations.

Combining feedback loops with both objective and subjective metrics can provide a more nuanced, context specific evaluations to help fine tune models more accurately for specific use cases and user needs.

If you’re building an LLM app and want to see how you can visualise data with Phospho, read our previous article here.

Conclusion: Integrating Quality Metrics for Scalable and High-Performing GenAI Products

In competitive markets and fast moving ones such as AI products, teams need to build solutions that not only meet the current needs of the market, but also quickly adapt to any changes in the future when they arise.

By adopting the 3 principle process of data collection, text analysis, and data visualisation, teams are positioned well to remain competitive in markets that are subject to regular change.

ML algorithms and both traditional and subjective metrics will need to be used together in order to understand and respond to these changing user needs with speed, accuracy and nuance.

If you want to understand your users closely and future proof your positioning, sign up here and try out Phospho for free. You can integrate our API for real-time analytics, or simply import your own data like a CSV or Excel file to quickly try out our features!