Log details for classic data collector
Performance Analytics score collection follows the process described here. To aid troubleshooting, a mapping between job steps and log entries is provided.
Collection overview
The data collection job involves executing an SQL query for each indicator source that uses the data collector. The query repeats for every collection time from the start date to the stop date. Then queries run for the next indicator source. Each step of executing a query is documented in the data collection job log.
Image omitted: data-collection-subprocesses.png
Flow chart showing the relation between the different levels of processes in a data collection job
Flow chart showing the relation between the different levels of processes in a data collection job
Optimization properties shown in log
The data collector has several optimizations that can be turned on or off in the System Properties. The beginning of the job log includes the state of these properties.
Note: Do not confuse these optimization properties with the new, optimized data collector. These properties apply only to the classic data collector. The new data collector is optimized at a more fundamental level.
| Optimization | Description | System property |
|---|---|---|
| Nested collection | Replaces flat mapping, to reduce redundant objects. For example, instead of creating identical indicator objects for every breakdown and element on that indicator, only one object is created for the indicator. | com.snc.pa.dc.nested_collection Default: true |
| Flush snapshots | The ServiceNow AI Platform stores a maximum number of Performance Analytics snapshots in memory before offloading them to the database. | com.snc.pa.dc.flush_snapshot_count Default: 1000 |
| Use hashcode for count distinct | Stores count distinct aggregations as a hashcode instead of as separate values. | com.snc.pa.dc.use_hashcode_for_distinct Default: true |
Starting and completing collection job
Follow the data collection process as revealed in log messages. The log messages in each subprocess have been separated into their own tables.
Note: This log is for the original data collector. If your instance uses this data collector, the system parameter com.snc.pa.dc.hsql exists and is "false." Otherwise, you are using the optimized data collector as described in Optimizing data collection. For more information about that log, see Log details for optimized data collector.
| Collection job step | Example of resulting log entry |
|---|---|
| Shows the domain of the collection job Run As user. | User <admin> belongs to domain global |
| Collection starts. Job start specifies whether scores, text index, or both are being collected. Messages also specify the settings of the data collection optimization properties. | Starting collection of Both scores and text index. Value of properties in system -Flush Snapshot Count: 1,000 Is nested collection: true Using hashcode for count distinct: true |
| Shows the domains for which scores are collected. Also shows other relevant details about the domains. | DomainConfig: Testing Domain Config Collection logs - 3d359877b77000106886e664de11a97c, Type: conditions, RollUpType: roll\_up\_children, collectAggregate? Yes, collectChildren?: Yes, AggregateDomain: Name: 3d359877b77000106886e664de11a97c - SysID: d1659c77b77000106886e664de11a91e, Domains: Default,MSP,ACME,Cisco,Initech,Oceanic Airlines |
| Lists indicator sources that the job collects data for, including their unique IDs. | Indicator Sources to collect: Incidents.New: b54f2c23d7030100b96d45a3ce61032f, Incidents.Open: b5cf2c23d7030100b96d45a3ce6103cc, Incidents.Closed: 2ac8dd31ff2302001e68ffffffffff3d, Incidents.Resolved: 4c207c23d7030100b96d45a3ce610308 |
| Lists each indicator associated with the job and the indicator properties that are related to data collection:- Are records collected in a snapshot? - Are any breakdowns excluded? - Are the collection periods in the indicator source overridden? - Is the breakdown matrix collected? - If the breakdown matrix is collected, are any breakdown combinations excluded? | JobIndicator: Number of open incidents - de537dc3d7131100b96d45a3ce610305 Job Indicator Properties: collectIndicator?: Yes, BreakdownCollectionType: WITH\_EXCLUDE Excluded Breakdowns: Assignment Group, State Indicator Properties => CollectingRecords: Yes, Overridden Collection Periods: No, Scripted?: No, Breakdown Combinations?: Yes, Excluded Breakdown Combinations: No combination is excluded. |
| Iterate steps for each indicator source | See table Steps iterated for each indicator source. |
| Show statistics for the data collection job | Statistics: inserts 184, updates 0, deletes 92, errors 0, warnings 0 |
| Complete job | Collection ended |
Steps iterated for each indicator source
| Collection job step | Example of resulting log entry |
|---|---|
| Retrieve indicator source. | Processing indicator source Incidents.Open |
| List indicators that have additional conditions, beyond the conditions inherited from the indicator source. | Indicator: Number of reassigned open incidents - ffb59561ff2302001e68ffffffffff33, Additional Conditions: reassignment\_count>0 |
| Indicates whether score collection is following the nested data structure instead of using flat mapping. Nested collection creates fewer redundant objects in memory.Non-nested collection supports batching of scores. | Using nested collection.Note - Batch inserts of scores are not supported with nested collection. |
| Iterate steps for each collection period. | See table Steps iterated for each collection period |
| Retrieve next indicator source. | Processing indicator source Incidents.New |
Steps iterated for each collection period
| Collection job step | Example of resulting log entry |
|---|---|
| Start date of collection job. | Collecting for 20191103 on Indicator Source Incidents.Open |
| Fetch the following fields:- The unique fields of the breakdown sources \(typically sys\_id\) - The indicator fields used in breakdown mappings - The table fields used in scripted breakdown mappings - The indicator source fields used in text analytics, if the text index is collected \(typically includes short\_description\) | Fetching "short\_description,sys\_id, opened\_at, assignment\_group, description, priority, category" from "incident" |
| Generate SQL based on the conditions that are specified in the indicator source.Note: If the indicator source specifies Today in one of the conditions, Today is considered relative to the period for which the data collection job is executed. For example, the Incidents.New indicator source includes the condition [Opened][on][Today]. With days defined to start at 07:00:00, when data is collected for 2019-10-28, the job produces the SQL script on the right. | SELECT task0.\`sys\_id\` FROM task task0 WHERE task0.\`sys\_class\_name\` = 'incident' AND \(task0.\`opened\_at\` >= '2019-10-28 07:00:00' AND task0.\`opened\_at\` <= '2019-10-29 06:59:59'\) |
| Display the number of records fetched by the SQL. | Fetched 150 rows from Indicator Source Incidents.Open \(facts table: incident\) |
| Run the map/reduce function to calculate indicator scores. | See the table Steps of the map/reduce function |
| If text indexing is active and has been configured for the indicator source, the data collector stores the resulting text index. | Storing Text Index for indicator source Incidents.Open |
| Bytes used by text index: 41,984 for: Incidents.Open | |
| Begin storing newly collected results for the indicator source. | Scores to be stored: 374,842 \(excluding lazily evaluated level 2 count distinct scores for indicators - ID - All Incidents with All 3 Breakdowns\) |
| Any indicator with a null value is logged. | Stored 'Value when nil': 0.0 as score for indicator: Number of incidents deflected by knowledge |
| Finish storing results. | Scores stored: 1,125,057 |
| Stored collected results | |
| Specify which indicators the data collector does not collect scores for. | Not collecting for Indicator: Summed age of open incidents with excluded Breakdown: Assignment Group |
| Finish collecting data for that indicator source for that period. | Collection for 20200519 on Indicator Source Incidents.Open finished |
| Display the average amount of time spent on a record to process additional conditions that were set on indicators. | Average time taken to process additional indicator condition for each record: 138 ms |
Steps of the map/reduce function
| Collection job step | Example of resulting log entry |
|---|---|
| The map/reduce function runs. | Applying map/reduce function for indicator source Incidents.Open |
| Warning if text indexing is active for the job but cannot run because of a missing or invalid text index configuration. | Skipping text index collection as there is missing or invalid text index configuration |
| Delete previous scores for the indicators and breakdowns that use the indicator source. | Deleting previous results for indicator source Incidents.Open |
| Deleted previous results 38 for indicator: External Logins | |
| Deleted previous results 21 for indicator: Failed Logins | |
| Process each record on the indicator source to calculate scores. Log messages track running total of processed records and memory usage. Processing for each record involves:- Execute or evaluate any scripts. - Validate indicator conditions. - Calculate the indicator score. - For each breakdown: 1. Calculate the breakdown score or execute the breakdown script. 2. Retrieve all breakdown unique values. 3. Create or update the array for scores or snapshots. The array is: `Indicator, Breakdown 1, Artifact 1, Breakdown 2, Artifact 2, Domain, Value, Array of [sys_id]` | Processed 18%% \(1\) records, Result Object size: 5 MB |
| Processed 42% \(2\) records, Result Object size: 9 MB | |
| Processed 76% \(3\) records, Result Object size: 14 MB | |
| Processed 100% \(4\) records, Result Object size: 20 MB | |
| The map/reduce function completes. | Applied map/reduce function |
Parent Topic:Data collection process and logging