Introduction
AI Assist (KAI) is an AI-powered assistant designed to help users interact more efficiently with our A/B testing platform. It provides quick, relevant answers to your questions about our tool, whether you’re a developer, marketer, or another type of user. KAI leverages the most relevant documentation and your own A/B test data to provide responses that help you make informed decisions.
How KAI works
KAI is integrated directly into Kameleoon’s user interface and can respond to a variety of queries related to our Web and Feature Experimentation platform. The assistant’s responses are generated using a large language model (LLM) provided by OpenAI. KAI is designed to use only the content we provide to generate responses, ensuring that the answers are relevant and accurate.
Step-by-step process
- Understanding the query: When a user asks a question, KAI first analyzes the query to determine the type of information needed.
- Fetching relevant documents:
- KAI identifies whether the question pertains to our tool’s documentation.
- If the question is about the usage of our tool, KAI fetches the most relevant documents from our two main sources:
- Developer Documentation: Technical guides, API references, and coding examples for developers.
- User Documentation: Simplified explanations, use cases, and best practices for non-developers.
- KAI then passes the relevant documents along with the user’s query to the LLM, which generates a response based solely on that content.
- Handling user data queries:
- If the question pertains to user-specific data, such as A/B test results, KAI identifies the need for specific data retrieval.
- KAI makes a secure API call using JWT (JSON Web Token) authentication to fetch the data associated with the user’s account. This ensures that the AI only accesses data for which the user has permissions.
- The following types of data are used to provide detailed responses related to A/B test results:
- Experiment metadata: Contains general information about the experiment, such as its name, the website it was run on, the duration, the objectives aimed to be optimized, and the segment it was targeting.
- Goal metadata: Provides details on the different objectives included in the experiment, mainly their names and types.
- Segment metadata: Holds information about the visitors who participated in the experiment, including the conditions that visitors had to meet to be targeted by the experiment.
- Experiment data: This data provides a comprehensive summary of the experiment’s results. It includes various metrics used for health checks, such as Sample Ratio Mismatch (SRM), and offers insights into the performance of the experiment’s variations.
- The assistant then uses this data to provide answers to questions like “Which variation performs better?” or “What is the p-value of this test?”
- Generating a response: Once the relevant documentation or data is retrieved, the information is sent to the LLM to formulate a response. The model strictly uses only the provided content, ensuring that responses are relevant to your queries about our platform and your data.
Data privacy and security
We take privacy and security very seriously. Here’s how we ensure that your data is protected:
- Documentation data: KAI only uses our company’s publicly available documentation to answer questions. It does not access any external data sources beyond our controlled environment.
- User data access: When a query involves user-specific data, KAI uses secure API calls with JWT authentication. This means:
- KAI can only access data tied to the authenticated user, preventing access to any other users’ data.
- Each data request is logged and monitored to ensure compliance with our security policies.
- Data minimization: The assistant only accesses the minimum necessary data required to answer your question.
Limitations
- Scope of responses: KAI is limited to responding to questions related to our product. It cannot answer questions unrelated to our services, tools, or your specific experiment data.
- Accuracy of information: While KAI strives to provide accurate and relevant responses, the quality of answers depends on the available documentation and data. We continuously update our documentation to improve response accuracy.
Common concerns
- Will KAI access my data without permission? No, KAI will only access data that you have permission to view. It uses secure methods to ensure that data retrieval is compliant with our privacy policies.
- Can KAI access data from other users? No, due to the use of JWT authentication, KAI can only access data associated with your account. It cannot view or retrieve data from other users.
- Is my experiment data safe when using KAI? Although experiment data will be shared with OpenAI when requesting takeaways for a specific experiment, it consists of aggregated data with no personal identifiers linked to your visitors. If you still have concerns, please contact us to disable this KAI feature
Use Cases
You may want to use KAI for various use cases, whether you need it to run a basic analysis of your test’s results, to identify the key metrics of your experiment, to explain some unexpected results or even have it recommend potential next steps.
Here are some examples of prompts to get you started:
- Summarize the key findings of this experiment.
- Which of the attached goals should we prioritize in evaluating the effectiveness of this experiment ?
- How do you explain the observed results of this test ?
- What could be a follow up experiment ?
Need more? Feel free to read this article that offers over 50 prompts!
Contact us
If you have further questions or concerns about KAI, please reach out to our support team at support@kameleoon.com. We are here to help ensure that your experience with KAI is secure, efficient, and valuable.
