Handling import failures for large GCP resources
Manage the data source failure when importing resources from GCP.
When importing large GCP resources, the data sources in the Service Graph Connector for GCP might fail because of a Direct REST response exceeding the 1024-MB limit. For more information, see REST step.
You can use one or more of the following options to handle the REST response size limitation error in the Service Graph Connector for GCP:
Save the response as an attachment
With the save response as an attachment feature, the response of each data source is temporarily stored as an attachment and after the response is read, the attachment is deleted automatically. The save response as an attachment feature is activated by default and is triggered by the sn_gcp_integ.save_rest_response_as_attachment system property.
As a user with the admin role, you can deactivate the save response as an attachment feature by setting the sn_gcp_integ.save_rest_response_as_attachment property to false. When the feature is deactivated and the response size is within the limit, the data sources run normally.
Modify the batch size
By default, the maximum batch size of the environment variable for a data source is set to 100 records. You can reduce or increase the batch size based on the data volume. To manage the batch size of a data source, you must add a system property of integer type for the data source as described in the following table.
| Data source | System property |
|---|---|
| SG-GCP Cloud Database | sg_gcp_cloud_db_batch_size |
| SG-GCP Cloud Function | sg_gcp_cloud_function_batch_size |
| SG-GCP Cloud Object Storage | sg_gcp_object_storage_batch_size |
| SG-GCP Kubernetes Cluster | sg_gcp_kubernetes_cluster_batch_size |
| SG-GCP Kubernetes Cluster Role | sn_gcp_integ.sg_gcp_kubernetes_cluster_role_batch_size |
| SG-GCP Kubernetes Cluster Role Binding | sn_gcp_integ.sg_gcp_kubernetes_cluster_role_binding_batch_size |
| SG-GCP Kubernetes Deployment | sg_gcp_kubernetes_deployment_batch_size |
| SG-GCP Kubernetes Namespace | sg_gcp_kubernetes_namespace_batch_size |
| SG-GCP Kubernetes Node | sg_gcp_kubernetes_node_batch_size |
| SG-GCP Kubernetes Pod | sg_gcp_kubernetes_pod_batch_size |
| SG-GCP Kubernetes Replicaset | sg_gcp_kubernetes_replicaset_batch_size |
| SG-GCP Kubernetes Service | sg_gcp_kubernetes_service_batch_size |
| SG-GCP Load Balancer | sg_gcp_load_balancer_batch_size |
| SG-GCP Load Balancer Health Service | sg_gcp_lb_health_service_batch_size |
| SG-GCP Load Balancer Pool | sg_gcp_lb_pool_batch_size |
| SG-GCP Load Balancer Pool Member | sg_gcp_lb_pool_member_batch_size |
| SG-GCP Load Balancer Service | sn_gcp_integ.sg_gcp_lb_service_batch_size |
| SG-GCP Network | sg_gcp_network_batch_size |
| SG-GCP Kubernetes Node Pool | sn_gcp_integ.sg_gcp_kubernetes_node_pool_batch_size |
| SG-GCP Security Group | sg_gcp_security_group_batch_size |
| SG-GCP Software Inventory | sg_gcp_software_inventory_batch_size |
| SG-GCP Storage Volume | sg_gcp_storage_volume_batch_size |
| SG-GCP Storage Volume Snapshot | sg_gcp_storage_vol_snapshot_batch_size |
| SG-GCP Subnet | sg_gcp_subnet_batch_size |
| SG-GCP VM Instance | sg_gcp_vm_instance_batch_size |
Note: You must add these system properties as they aren't available by default. You must have the admin role to add or modify a system property. For more information, see Add a system property.