Search query parameter evaluation framework
AI Search performs A/B test evaluations that compare result relevancy for alternate sets of search query parameter values. These evaluations determine the models that AI Search uses for machine learning relevancy and for Q&A Genius Results.
The search query parameter evaluation framework enables A/B relevancy testing on live search traffic. AI Search considers results from this live testing along with its offline evaluation of aggregated search signal data when making the following determinations:
- Determining which relevancy model to publish for a search profile. For details on generation and publication of relevancy models, see Machine learning relevancy in AI Search.
- Determining which machine reading comprehension (MRC) model to use when validating potential Q&A Genius Result answers. For more information on the MRC model used for Q&A Genius Results, see Q&A Genius Results.
AI Search computes A/B testing evaluation results on a nightly basis.
Note: The search query parameter evaluation framework does not support user configured A/B testing. All A/B testing is automatically performed by AI Search as part of its evaluation of machine learning relevancy models.
The search query parameter evaluation framework is part of the Adaptive Telemetry suite of features.
Parent Topic:Administering AI Search
Search query parameter evaluation framework tables
The search query parameter evaluation framework for AI Search uses tables to store results and metrics from A/B testing of live search traffic. AI Search uses the stored data when determining which relevancy model and Q&A Genius Result answer validation model to publish for a search application.
A/B Testing Evaluation table
The A/B Testing Evaluation [evaluation] table stores details for A/B testing evaluations performed on live search traffic.
Note: The system automatically cleans this table, removing records for inactive evaluations that are more than two years old.
| Column | Description |
|---|---|
| Number | Autogenerated number to identify the evaluation. |
| Name | Name and description for the evaluation. |
| Source | Name of the table containing records to run the evaluation for. |
| Testing Scope | Scope of the evaluation.- Supported values: - ALL: Run the evaluation against all records in the table specified by Source. - SELECTED: Only run the evaluation against records specified in the Selected Artifacts field value. - Default value: ALL |
| Artifact Provider | List of artifact filtering classes that provide records for testing. |
| Selected Artifacts | Comma-separated list of record sys_id values. When Testing Scope is set to SELECTED, the evaluation only runs against records with sys_id values specified in this list. |
| Minimum Signals for Evaluation | Minimum number of search signals the evaluation should collect before ending.- Type: integer - Default value: 0 (no minimum) - Maximum value: 30000 For details on how this condition interacts with the Minimum Days for Evaluation condition, see End Criteria Operator. |
| Minimum Days for Evaluation | Minimum number of days the evaluation should be active before ending.- Type: integer - Default value: 0 (no minimum) - Maximum value: 180 For details on how this condition interacts with the Minimum Signals for Evaluation condition, see End Criteria Operator. |
| End Criteria Operator | Logical operator to use for evaluation end conditions when both Minimum Signals for Evaluation and Minimum Days for Evaluation conditions exist.- Supported values: - AND: The evaluation remains active until the Minimum Signals for Evaluation and Minimum Days for Evaluation conditions are both satisfied. - OR: The evaluation remains active until either of the Minimum Signals for Evaluation or Minimum Days for Evaluation conditions is satisfied. - Default value: OR |
A/B Testing Evaluation Execution table
The A/B Testing Evaluation Execution [evaluation_execution] table stores details of individual operations executed as part of A/B testing evaluations for live search traffic.
| Column | Description |
|---|---|
| Number | Autogenerated number to identify the evaluation operation. |
| Evaluation | Reference to the record for the evaluation on the A/B Testing Evaluation \[evaluation\] table. |
| Artifact ID | Sys\_id for the record analyzed by this evaluation operation. |
| State | State for the record evaluation execution.- Supported values: - Queued: The evaluation operation is queued for execution. - In Progress: The evaluation operation is underway. - Scoring: The evaluation operation is in the scoring phase. - Complete: The evaluation operation completed successfully. - Errored: The evaluation operation failed with an error. - Canceled: The evaluation operation was canceled by the system. - Suspended: The evaluation operation was suspended by the system. - Default value: Queued |
| Start Date | Date and time when the evaluation operation started. |
| End Date | Date and time when the evaluation operation ended. |
A/B Testing Evaluation Parameter table
The A/B Testing Evaluation Parameter [evaluation_parameter] table stores details of individual search query parameters considered in A/B testing evaluations for live search traffic.
| Column | Description |
|---|---|
| Evaluation | Reference to the record for the evaluation on the A/B Testing Evaluation \[evaluation\] table. |
| Name | Name and description for the evaluation parameter. |
| Use Artifact's Assigned Parameter |
|
| Parameter Values Traffic Percentage | Percentage of search queries to apply the specified Parameter Values to during the A/B evaluation.- Type: integer - Default value: 0 - Maximum value: 100 |
| Parameter Values | JSON list of values to compare to the current search query parameter value when Use Artifact's Assigned Parameter is true. |
| Parameter Type | Type for the search query parameter. The selected value determines how the Parameter Values are used during and after an A/B evaluation. Supported values:
|
| Score Calculation Type | Form of calculation used to compute search query parameter value scores and find the best value. Supported values:
Note: AI Search computes the average click position score by dividing the sum of all selected search result ranks by the number of searches. The highest-ranked search result has rank 1, the next-highest has rank 2, and so on. As an example, if you perform two searches, selecting the first result in one case and the second result in the other, your average click position score is (1 + 2) / 2 = 1.5.
Note: The helpfulness score is a metric that indicates whether Genius Result answers were helpful in the context of your search.
|
| Signal Provider | The provider for search signals needed to score the search query parameter. Search Event Signal Provider is the only supported value. |
A/B Testing Evaluation Parameter Result table
The A/B Testing Evaluation Parameter Result [evaluation_parameter_result] table stores calculation results for individual search query parameters considered in A/B testing evaluations for live search traffic.
| Column | Description |
|---|---|
| Evaluation Execution | Reference to the record for the evaluation execution on the A/B Testing Evaluation Execution [evaluation_execution] table. |
| Parameter Evaluation | Reference to the record for the search query parameter on the A/B Testing Evaluation Parameter [evaluation_parameter] table. |
| Best Value | Best value for the search query parameter, as determined by the Winning Score. |
| Winning Score | Numeric score for the search query parameter, determined using the Score Calculation Type. |
| Score Metadata | Metadata from the score computation for the search query parameter. |
Exclude a search profile from the search query parameter evaluation framework
Exclude a search profile from A/B testing evaluations of live AI Search traffic. This procedure prevents AI Search from using A/B testing evaluation results when publishing the search profile's search result relevancy model and its Q&A Genius Result answer validation model.
Before you begin
Role required: ais_admin
About this task
The search query parameter evaluation framework performs A/B testing evaluations of search configuration settings using live search traffic. By default, AI Search evaluates configuration settings for all search profiles.
Search administrators can exclude individual search profiles from the search query parameter evaluation framework. Excluding a search profile from the framework prevents AI Search from performing A/B testing evaluations for live search traffic that uses the excluded search profile.
Note: When you exclude a search profile from evaluations, AI Search no longer considers evaluation results when updating the machine learning relevancy and machine reading comprehension (MRC) models for that search profile. As a result, relevancy scoring settings and Q&A Genius Result answer filtering settings for the search profile may be less reflective of your search traffic. For more information on how AI Search uses A/B evaluation testing results when publishing these models, see Machine learning relevancy in AI Search and Q&A Genius Results.
Procedure
Navigate to All > AI Search > Search Experience > Search Profiles.
Open the record for the search profile that you want to exclude from A/B testing evaluations of live search traffic.
If the Search Profile form doesn't already display the Exclude from evaluation field, configure the form layout to make the field visible.
For details on configuring a form layout to make fields visible, see Configuring the form layout.
Select the Exclude from evaluation option.
Select Update.
Result
AI Search no longer performs A/B testing evaluations for traffic that uses the excluded search profile. Machine learning relevancy no longer updates the relevancy model for the search profile.