Data Collection

Data collection settings impact how data is stored in the database.

It means that regardless of the option the user selects, all low-level raw events are being collected on the endpoints, sent to the server and processed by the rule engine, which generates detections when appropriate.

When detection is generated for the specific event, this particular event is also stored in the database regardless of the selected data collection option. For the events processed by the rule-engine without triggering detection, data collection settings apply as follows:

Store all available data

All collected low-level raw events are stored in the database. This option creates a vast database but allows detailed investigation of possible incidents because the analyst can see everything that happened on the system, regardless of whether it was previously flagged as suspicious by the product or not.

This option allows using all the product's features, such as retroactive search, execution of Threat Hunting queries or a re-run of existing or custom rules on the data.

Store most important data

Stores all the data related to the processes (i.e. you will see all processes executed on the endpoints along with its properties, such as command-line, etc.) but limits the storage of low-level events generated by the processes only to those generating the detection. It means the analyst will see a complete process tree during the investigation (be aware that also data retention setting applies here). Still, for the actions of the processes (such as writing to the disk, writing to the registry, network connection etc.), the analyst will see only those which were explicitly caught by the rule.

This may turn into a situation where you will see a network connection (detected by the rule) but will not see that the downloaded file was written to the disk (unless another rule does not detect a particular file write). Similarly, you will not be able to retroactively search for IOCs that are not connected to the process/file and were not detected by the rule previously. For example, i.g. you will find file hash or command-line fragment, but not the registry write anymore.

In a case you will be missing some important events using this option, you can still customize this setting by creating custom rules (typically with low severity, which you will ignore during the monitoring) that will detect events of your interest, solely to save these low-level events to the database.

Store only data directly related to detections

This option stores only those low-level event data explicitly caught by the rule, so the smallest database in this mode is created. It applies to the actions (such as writing to the disk, writing to the registry, network connection etc.), as well as to the processes themselves (process execution, command-line, etc.). However, when detection is triggered for the process, process-related data for the process itself, along with the process-related data about all the processes upwards, the process tree and direct child processes will be stored. (Storing process-related data for these additional processes will not store any other data such as „action-related “ events.) Information about all the other processes will not be stored in the database.

That means that similarly to the option „Store most important data“ you may not see some actions and, in this case, also processes that were not explicitly caught by the rule (and are not in that stored part of the process tree mentioned above) with the corresponding consequences. Processes for which data were not stored will not be shown in the process tree.

The ability to retroactively search for IOCs would be minimal since you will see neither IOCs related to actions nor IOCs related to process properties (command-line, etc.) unless they were detected by the rule previously.

As mentioned in the previous option, you can also customize this setting by creating custom rules (typically with low severity, which you will ignore during the monitoring) that will detect events of your interest (including process execution) solely to save these low-level events to the database.

important

Important

As mentioned previously, the storage of low-level event data is significantly affected by which events are being detected. It means that lowering the number of enabled rules will, in turn, also decrease the amount of stored data (available for investigation and retroactive search). This should be taken into account when lowering the Data Collection settings and disabling some rules simultaneously.

For example, selecting „Store only data directly related to detections“ and disabling all the rules will lead to storing no data at all and thus causing the product to be dysfunctional.

Store most important data/Store only data directly related to detections, some features are limited:

Events view

Aggregated events view

Background tasks

Scripts view

Search

The goal is to control the database size. The user is making a trade-off between database size and some advanced options.

Store only data directly related to detections option is recommended for IT Administrators.

To change what data is stored in the database, go to Admin->Server Settings in the Database Collection section.

 

Data Retention

Select how long should be data in the database stored. The longer the period, the more extensive database will become before old data are purged.

Choose for how long you want the data in the database to be stored. By default, it is three months.

Choose for how long you want the low-level data in the database to be stored. By default, it is one month. Low-level data accounts for most of the database size and should be kept as brief as possible. When it is removed, it only limits detailed investigations of data that was not identified as suspicious by the product.

This setting can be changed in Admin->Server Settings in the Database cleanup section.