Skip to main content

AI Assist

AI Assist (KAI) is an AI-powered assistant designed to help users interact more efficiently with our A/B testing platform. It provides quick, relevant answers to questions about our tool. KAI leverages the most relevant documentation and your A/B test data to provide responses that help you make informed decisions.

How KAI works​

KAI is integrated directly into Kameleoon's user interface and can respond to a variety of queries related to our Web and Feature Experimentation platform. The assistant's responses are generated using a large language model (LLM) provided by OpenAI. KAI is designed to use only the content we provide to generate responses, ensuring answers are relevant and accurate.

To access KAI, click the sparkle icon in the top-right corner > Ask a question.

Step-by-step process​

  • Understanding the query: When a user asks a question, KAI first analyzes the type of information needed.
  • Fetching relevant documents: KAI identifies whether the question pertains to our tool's documentation. If the question is about the usage of our tool, KAI fetches the most relevant documents from two main sources:
    • Developer documentation: Technical guides, API references, and coding examples for developers.
    • User documentation: Simplified explanations, use cases, and best practices for non-developers.
  • Handling user data queries: If the question pertains to user-specific data, such as A/B test results, KAI identifies the need for specific data retrieval. KAI makes a secure API call using JSON Web Token (JWT) authentication to fetch the data associated with the user's account. This call ensures that KAI only accesses data for which the user has permissions. The following data types are used to provide detailed responses related to A/B test results:
    • Experiment metadata: Contains general information about the experiment, such as its name, the website it was run on, its duration, objectives, and the segment it was targeting.
    • Goal metadata: Provides details on the different objectives included in the experiment, mainly their names and types.
    • Segment metadata: Holds information about the visitors who participated in the experiment, including the conditions that visitors had to meet to be targeted by the experiment.
    • Experiment data: This data provides a comprehensive summary of the experiment's results. It includes various metrics used for health checks, such as Sample Ration Mismatch (SRM), and offers insights into the performance of the experiment's variations.
  • Generating a response: Once the relevant documentation or data is retrieved, the information is sent to the LLM to formulate a response. The model strictly uses the provided content, ensuring that responses are relevant to your queries about our platform and your data.

Data privacy and security​

We take privacy and security very seriously. Here's how we ensure that your data is protected:

  • Documentation data: KAI only uses our company's publicly available documentation to answer questions. It does not access any external data sources beyond our controlled environment.
  • User data access: When a query involves user-specific data, KAI uses secure API call with JWT authentication. This means:
    • KAI can only access data tied to the authenticated user, preventing access to any other users' data.
    • Each data request is logged and monitored to ensure compliance with our security policies.
  • Data minimization: The assistant only accesses the minimum necessary data required to answer your question.

Limitations​

  • Scope of responses: KAI is limited to responding to questions related to our product: it cannot answer questions unrelated to our services, tools, or your specific experiment data.
  • Accuracy of information: While KAI strives to provide accurate and relevant responses, the quality of answers depends on the available documentation and data. We continuously update our documentation to improve response accuracy.

Common concerns​

Will KAI access my data without permission?​

No, KAI will only access data it has permission to view. It uses secure methods to ensure that data retrieval is compliant with our privacy policies.

Can KAI access data from other users?​

No. Due to the use of JWT authentication, KAI can only access data associated with your account. It cannot view or retrieve data from other users.

Is my experiment data safe when using KAI?​

Yes, your data is safe. Although experiment data is shared with OpenAI when requesting takeaways for a specific experiment, it consists of aggregated data with no personal identifiers linked to your visitors. If you still have concerns, please contact us to disable KAI.

Use cases​

You may want to use KAI for various reasons, whether you need it to run a basic analysis of your test's results, identify the key metrics of your experiment, explain unexpected results, or have it recommend potential next steps.

Here are some example prompts to get you started:

  • Summarize the key findings of this experiment.
  • Which of the attached goals should we prioritize in evaluating the experiment's effectiveness?
  • How do you explain the test's observed results?
  • What could be a follow-up experiment?

Looking for more? Feel free to read this article that offers over 50 prompts.

Contact us​

If you have further questions or concerns about KAI, please reach out to our support team at support@kameleoon.com. We are here to ensure your experience with KAI is secure, efficient, and valuable.