Feed Utility configuration parameters
22 August 2024
ID 171650
Feed Utility reads the configuration parameters, feed rules, filtering rules, and parsing rules for feeds from the configuration file. This file is in XML format and has several groups of parameters.
The paths in the configuration file must contain only the characters used in the operating system locale, otherwise Feed Utility will not work.
Feed (feed rules, filtering rules, and parsing rules)
The Feed
parameter contains rules for a particular feed. This element has several types of nested parameters:
- Feed rules specify how this particular feed must be processed by Feed Utility.
- Filtering rules are criteria that Feed Utility uses to filter the original feed files. Filtering rules are a part of feed rules for each feed.
- Parsing rules are rules for custom feeds (OSINT feeds, and other feeds that are not Kaspersky Threat Data Feeds). These parameters specify how each feed must be parsed by Feed Utility.
This parameter has the following attributes:
enabled
Specifies whether Feed Utility must download and process this feed.
If
enabled
is true, Feed Utility downloads and processes the feed. Ifenabled
is false, Feed Utility skips this feed.
The following example demonstrates how feed rules, filtering rules, and parsing rules are nested in the configuration file.
<Settings> ... <Feeds> ... <Feed enabled="true"> <Name>Malicious_Hash_Data_Feed</Name> <!-- Other feed rules for this feed --> <Filters> <Field name="popularity" value="4;5"/> <!-- Other filtering rules for this feed --> </Filters> </Feed> <Feed> <Name>Botnet_CnC_URL_Data_Feed</Name> <!-- Other feed rules for this feed --> <!-- This feed has no filtering rules --> </Feed> ... </Feeds> ... </Settings> |
FeedsDir
The FeedsDir
parameter specifies the directory where Feed Utility puts processed feed files.
WorkDir
The WorkDir
parameter specifies the directory where Feed Utility puts the downloaded and unpacked feed files.
If this parameter is not specified, Feed Utility uses the default temporary directory of the operating system.
WorkDir
cannot be equal to FeedsDir
.
CertFile
The CertFile
parameter specifies the path to the certificate file. This certificate is used by Feed Utility to download feeds.
The certificate file must be in the PEM format.
SourceIPs
The SourceIPs
parameter specifies the IP addresses that are used by Feed Utility to download feeds.
This parameter is optional. If it is omitted or has an empty value, Feed Utility resolves Kaspersky server addresses by their domain names.
You can specify one or more IPv4 addresses in this parameter. To specify several IP addresses, use the semicolon (";"
) as a delimiter.
The following example demonstrates specifying IP addresses in the SourceIPs
parameter.
<SourceIPs>192.0.2.1;192.0.2.2</SourceIPs> |
SourceDomains
The SourceDomains
parameter specifies the domain names that are used by Feed Utility to download feeds.
You can specify one or more domain names in this parameter. To specify several domain names, use the semicolon (";"
) as a delimiter. Feed Utility will attempt to download feeds from the specified domain names in the order they appear in the configuration file.
When SourceDomains
and SourceIPs
parameters are used together, domains specified in the SourceDomains
parameter are used before IP addresses specified in the SourceIPs
parameter. If all attempts to download feeds fail, Feed Utility will generate an error message.
You can use Unicode symbols in this parameter.
The following example demonstrates specifying IP addresses in the SourceDomains
parameter.
<SourceDomains>updates1.example.com;updates2.example.com</SourceDomains> |
CreateExternalFeedInfoList path="PATH"
This parameter is obsolete. It is ignored in the current version of Kaspersky CyberTrace.
The CreateExternalFeedInfoList
parameter specifies whether a list of supported OSINT feeds must be generated. This parameter is mandatory.
If this parameter is 1
, Feed Utility creates a list of supported OSINT feeds, osint_feed_list.conf, in a directory specified in the path
attribute. If you added any custom or third-party feeds to Kaspersky CyberTrace, Feed Utility also creates a list of these feeds, custom_feed_list.conf, in the same directory as osint_feed_list.conf.
If this parameter is 0
, Feed Utility does not create a list of supported OSINT feeds.
The following example demonstrates specifying a path where the list must be created. In this example, the list will be created in a directory where Feed Utility binary is located.
<CreateExternalFeedInfoList path=".">1</CreateExternalFeedInfoList> |
NotifyKTFS path="PATH"
The NotifyKTFS
parameter specifies whether Kaspersky CyberTrace Service must be notified about the feed updates.
This parameter can be used only with json
output format.
If this parameter is 1
, Feed Utility notifies Kaspersky CyberTrace Service that the feeds must be reloaded. A path to the Kaspersky CyberTrace Service binary file must be specified in the path
attribute of this parameter.
If this parameter is 0
, Feed Utility does not notify Kaspersky CyberTrace Service.
EULA
The EULA
parameter specifies whether the terms of the End User License Agreement (EULA) were accepted by a user.
If this value is accepted
, the terms of the EULA were accepted.
If this value is rejected
, the terms of the EULA were not accepted. In this case, Feed Utility cannot be used.
RetryCount
The RetryCount
parameter specifies the number of attempts to download a Kaspersky Threat Data Feed. Feed Utility tries to re-download a feed when a connection timeout, partial downloading, and other errors occur.
If the specified number of attempts were unsuccessful, Feed Utility displays an error message and continues its operation.
This parameter is used only for Kaspersky Threat Data Feeds. OSINT feeds and other custom feeds will not be re-downloaded by Feed Utility.
This parameter is optional. If this parameter is not specified, Feed Utility uses the default value of 10
.
If this parameter is 0
, the number of attempts is not limited.
SequentialDownload
The SequentialDownload
parameter specifies whether Feed Utility must download feeds in sequential or parallel mode.
If this value is 1
or true
, Feed Utility downloads feeds in sequential mode, one by one.
If this value is 0
or false
, Feed Utility downloads feeds in parallel mode, all feeds at the same time.
By default, this parameter has the value of 0
.
OutputFormat
The OutputFormat
parameter defines the output format for all feeds. This parameter can have the following values:
json
The feeds are in JSON format. The feed files have a .json extension.
This is the default value. If the
OutputFormat
parameter is omitted, this value is used to define the output format.txt
The feeds are in plain text format (UTF-8 with BOM). The feed files have a .txt extension.
delimiter
attributeIn this format, record fields are separated with a delimiter. The default delimiter is
";"
. To specify a custom delimiter, use thedelimiter
attribute as follows:<OutputFormat delimiter="%delimiter%">txt</OutputFormat>
Here, substitute
%delimiter%
with a symbol that must be used as a delimiter.indicatorPerLine
attributeTo output one record field per line, set the
indicatorPerLine
attribute to 1 as follows:<OutputFormat indicatorPerLine="1">txt</OutputFormat>
If you use this attribute, subfields specified in the
RequiredFields
feed rule must have the same parent field. For example,"files/MD5;files/SHA1"
is valid, while"files/MD5;whois/domain"
is invalid and will result in an error.
If this output format is specified, all feed rules in the configuration file must include a
RequiredFields
parameter. TheRequiredFields
parameter specifies the order in which the fields are written to the output feed.csv
Same as
txt
. The feed files have a .csv extension.You can use
delimiter
andindicatorPerLine
attributes.openioc
The feeds are in OpenIOC format. The feed files have an .ioc extension.
You can specify the version of the OpenIOC format in the
version
attribute: it can be either1.0
, or1.1
. If the attribute is omitted, version1.1
is used.Converting feeds to OpenIOC 1.0 format has some restrictions. Phishing URL Data Feed and Malicious URL Data Feed cannot be converted to OpenIOC 1.0 format; an error message is printed instead. For other feeds, only hash and IP address fields are converted. Converting feeds to OpenIOC 1.1 format has no such restrictions.
It is not recommended to use the
RequiredFields
element. In this case, the feed will be missing the fields that are mandatory for converting to OpenIOC format.It is not recommended to use the
RecordsCount
parameter, since it is not intended for this format. The results of using this parameter may be unpredictable.Feeds in OpenIOC format take significantly more hard drive space than the original feed files.
stix
The feeds will be in STIX™ format. The files will have an .xml extension.
For STIX format, feeds with URL masks must have the
type
field.You can specify the version of the STIX format in the
version
attribute: it can be1
,2.0
or2.1
. If value1
is specified, the feed will be in STIX 1.1 format. If the attribute is omitted, value1
is used.It is not recommended to use the
RequiredFields
element. In this case, the feed will be missing the fields that are mandatory for converting to STIX format.It is not recommended to use the
RecordsCount
parameter, since it is not intended for this format. The results of using this parameter may be unpredictable.Feeds in STIX format take significantly more hard drive space than the original feed files.
The following example demonstrates how the OutputFormat
parameter is nested in the configuration file.
<Settings> ... <Feeds> <OutputFormat>json</OutputFormat> ... </Feeds> ... </Settings> |
CreateDiff
The CreateDiff
parameter specifies whether Feed Utility must create feed diffs. Feed diffs are files that contain differences between the old and new version of a processed feed file. This parameter affects all feeds created by Feed Utility as follows:
- If this parameter is
0
, Feed Utility does not create feed diffs. This is the default value. - If this parameter is
1
, Feed Utility creates feed diffs.
If CreateDiff
is 1
, and new versions of feeds are downloaded, two additional files are created for each feed (%feed_name%
is the name of the feed file):
The %feed_name%_new.json
file contains records that were added to the new version of the feed file.The %feed_name%_del.json
file contains records that were deleted in the new version of the feed file.
Feed diffs can be created only for feeds in JSON format that are contained in a single file:
- The
OutputFormat
parameter must have thejson
value. - For each feed, the
UrlMatcherField
parameter must be omitted or have an empty value. - For each feed, the
RecordsCount
parameter must not have theperFile
attribute, or this attribute must have a value of0
.
To create feed diffs, Feed Utility uses a key field in the old and new version of the feed:
- If this feed contains non-nested
id
,MD5
,ip
,url
, ordomain
field, this field is used as a key field. - If none of the fields above are present, Feed Utility attempts to search for a field with unique values across the feed. If this attempt is not successful, a warning is generated.
The following example demonstrates how the OutputFormat
parameter is nested in the configuration file.
<Settings> ... <Feeds> ... <CreateDiff>0</CreateDiff> ... </Feeds> ... </Settings> |
ProxySettings
The ProxySettings
parameter specifies proxy settings for Feed Utility. If you specify a proxy server, Feed Utility will download feeds using the specified parameters.
The user name and password for the proxy are stored in the Feed Utility configuration file. This information is not provided to Kaspersky.
Proxy settings are specified in the following parameters:
Host
Host of the proxy server.
You can specify a domain name or an IP address in this parameter. Both IPv4 and IPv6 addresses are supported.
Port
Port of the proxy server.
User
Encrypted user name for proxy server authentication.
If a proxy server does not require authentication, leave this parameter empty.
This parameter is stored encrypted. Use the
--set-proxy
command-line option to set this parameter. If you do not use this option and enter your user name as plain text, connection to the proxy server will not be established.Password
Encrypted password for proxy server authentication.
If a proxy server does not require authentication, leave this parameter empty.
This parameter is stored encrypted. Use the
--set-proxy
command-line option to set this parameter. If you do not use this option and enter your password as plain text, connection to the proxy server will not be established.
The following example demonstrates how proxy settings are nested in the configuration file.
<Settings> ... <ProxySettings> <Host></Host> <Port></Port> <User></User> <Password></Password> </ProxySettings> ... </Settings> |
LogSettings
The LogSettings
parameter defines how Feed Utility logs its activity.
If you enable logging, Feed Utility can write to the log files any of the following information that can be considered private, security-related, or sensitive: Feed Utility configuration parameters, proxy host and port, and operations performed while downloading and processing feeds.
If logging is enabled, Feed Utility writes to log files the information about free hard drive space that available for the work and feed directories. Also, starting from this version, an average speed that the feeds have while loading will be written to logs.
Log files are regular text files. All information written to the log files is not encrypted. The log files have standard inherited access rights. We recommend that you assign the directory for storing log files the appropriate rights so that only the administrator can read the log files.
Log files are stored until they are explicitly deleted by a user.
Feed Utility does not send log files or any data contained in them to Kaspersky. For technical support purposes, your Technical Account Manager (TAM) can ask you to provide log files.
Logging settings are specified in the following parameters:
EnableLog
Enables logging.
If this value is
1
ortrue
, Feed Utility logs its activity.If this value is
0
orfalse
, Feed Utility does not log its activity.LogsDir
Directory where Feed Utility stores its log files.
CleanOldLog
Enables removal of old log files.
If this value is
0
, upon initialization, Feed Utility keeps old log files.If this value is
1
, upon initialization, Feed Utility deletes old log files.
The following example demonstrates how logging settings are nested in the configuration file.
<Settings> ... <LogSettings> <EnableLog>0</EnableLog> <LogsDir>logs</LogsDir> <CleanOldLog>1</CleanOldLog> </LogSettings> </Settings> |