How to use Phospho.ai to improve your AI software in 2024:

Crazy news guys, we’ve just launched a startup program for AI founders (the perks are crazy).

You can get $2000 worth of credits (Anthropic, Mistral, OpenAI, and Phospho) + a call with our amazing team to guide you in your product-market-fit journey.

You can apply here.

The hard truth is, we’re past the point where we can just simply integrate an LLM into an app and call it a day. When building an LLM app in 2024 we need to understand the higher demands and expectations from normalising AI in our everyday apps.

It’s for this reason we now need to constantly improve and fine tune its performance through concrete data driven insights in order to keep our users.

One way you can do this is through text analytics. Phospho an open source text analytics platform specifically designed to log, monitor, and evaluate user interactions in real-time to provide actionable insights towards continuously optimising your LLM app.

Why Text Analytics is Crucial for AI Software in 2024

How many AI integrated products have a conversational functionality? Most, if not all.

But what a lot of us overlook is these conversations are a rich source of data we should be collecting and leveraging.

Therefore, it’s imperative that teams building AI software perform analysis on the conversations users have with the AI model to extract the richest insights possible and iterate their products more effectively.

After all, what better feedback than verbatim queries and responses while users are conversing with your app?

Text analytics is what enables AI software companies to form an understanding of their users from large amounts of this unstructured conversational data.

These deep insights from your users’ interactions results in (but not limited to) a few things:

• More informed iteration cycles
• Faster product market fit
• ‘Stickier’ products with lower churn
• and increased competitiveness in the market

The ability to do this in real time and Phospho’s unique positioning to specifically help extract these insights from LLM apps, positions it as the best analytics tool for effective AI software improvement.

How To Use Phospho: An Example Walkthrough

We’ll go into Phospho’s features more as we go through the steps in this example walkthrough of a chatbot service.

If you were building an AI chatbot which was struggling with inconsistent user experiences, you could use Phospho’s text analytics to perform analysis and visualisations to detect any trends, patterns, or specific queries that trigger poor user satisfaction or inaccurate responses.

1 - Integrate or Import Data into Phospho.ai

You have two options here - you can either connect Phospho to your LLM app via our API to collect and analyse real-time user interactions data, or you can import data yourself directly into the platform.

For integration with our API for real-time logging of user interactions you can see our docs here.

For importing historic data like a CSV or Excel file you can read this.

2 - Cluster Messages to Understand User Intents

let’s imagine we uploaded a week’s worth of user queries with our chatbot.

You can use Phospho’s clustering to group interactions based on intent to identify patterns and common use cases.

For example you might find with this chatbot the biggest groups are related to specific issues related to technical support, or a repeated complaint about a certain product.

Phospho does with without any manual tagging and speeds up the rate at which you can identify the most important aspects of your chatbot’s interaction quality to optimise.

Tip: Phospho’s clustering is configured to group by intent as default, but you can change this and apply a custom instruction by going into advanced settings. You can see how in our docs here.

3 - Automatically Tag, Score, and Classify New Messages

Every message that’s logged to Phospho goes through an analytics pipeline.

In this pipeline, Phospho looks for pre-defined tags defined in your project settings. You can create them to detect topics, hallucinations, behaviours, intents, or anything you want because you define your tags in natural language.

Example tags could be:

• User requests to speak to a human
• User providing product feedback
• Chatbot unable to understand user query

Phospho will then automatically tag, annotate, assign a performance score, and classify user interactions with complete flexibility and specificity to help you make better product decisions.

Read our docs here to see how you can create your tags.

4 - Set Up Evaluations to Quantify Performance

Let’s say we detected from using the previous steps that our chatbot was underperforming when receiving and managing user queries about complaints.

Teams can set their own custom KPIs and metrics for what improvement looks, let’s use this example for our chatbot:

KPI 1 - improved classification of figurative or literal speech from users.

KPI 2 - number of responses before customer is satisfied and finishes the chat.

You can do this with no code setups on our platform as well.

5 - A/B Test Your LLM App for Continuous Improvement

Once your custom metrics are set you can automatically A/B test different versions or iterations of your chatbot to see whether performance has improved.

Which in this case is whether the chatbot has correctly understood when the user was using either figurative or literal speech to provide a better response, and require fewer messages to satisfy the user.

Being able to measure outcomes and performance differences between different iterations helps you continuously and qualitatively provide a better user experience with each new release of your AI model.

6 - Explore and Visualize Results on the Phospho Dashboard

You can create completely customised dashboards with Phospho to track and visualise what is most important to you.

Using our example so far, let’s now say we release this new version or iteration of our chatbot and want our dashboard to visualise user sentiment trends over the course of this launch.

You can configure the dashboard to include key metrics, clustered messages, or any performance insights you like and your team can then adjust the AI model based on these findings.

To see how you can create custom dashboards with Phospho read our previous article on it here, or alternatively you can read about the importance of data visualisation first here.

Tip: you can export reports and data from here to other platforms if you wanted to perform further analysis elsewhere.

Conclusion: Why Phospho.ai is a Game Changer for AI Software in 2024

With this comprehensive step by step guide, you now have a blueprint for continuous improvement of your AI software using Phospho.

We hope you can now see a glimpse of the potential behind text analytics tools like ours by leveraging text from user interactions through clustering, tagging, custom KPIs, and automated evaluations.

You can use Phospho for free by signing up here and start improving your AI software with actionable insights and more effective iteration.