Feed rules
22 August 2024
ID 171651
Individual feed rules in the Feed
elements specify how each feed must be processed by Feed Utility.
By default, the configuration file has entries containing feed rules for all feeds. Entries for commercial feeds are disabled. If you use a commercial certificate to download feeds, uncomment the entries for feeds that are available with your certificate.
Example of feed rules
The following is an example of feed rules. Feed rules are specified individually for each feed.
<Settings> ... <Feeds> <Feed enabled="true"> <Name>Botnet_CnC_URL_Data_Feed</Name> <FeedID>65</FeedID> <Filters> <Field name="geo" value="RU"/> </Filters> <UrlMatcherField toRegex="false">mask</UrlMatcherField> <RequiredFields>mask;geo;first_seen;last_seen</RequiredFields> <RecordsCount perFile="100" total="1000" /> <FeedFields/> </Feed> <Feed enabled="false"> <Name>CustomFeed</Name> <Path>./custom_example/example_feed.json</Path> <Parsing type="json"> <MD5 type="MD5">files/MD5</MD5> </Parsing> <FeedFields/> </Feed> ... </Feeds> ... </Settings> |
Feed
This parent element contains feed rules for a feed.
Name
This element specifies the name of the downloaded feed file.
After Feed Utility unpacks downloaded feeds, it searches for a file with a name that begins with a specified string. If several file names begin with the specified string, Feed Utility prints an error message and stops processing feeds. In this case, you must manually resolve this conflict, for example, by deleting the extra files from the directory where Feed Utility unpacks them. This directory is specified by the WorkDir
parameter.
This parameter is case-insensitive.
This parameter must have a unique value. No two Feed
elements can have the same value in this parameter.
FeedID
This element applies only to Kaspersky feeds.
The element specifies the identifier of a feed. Feed Utility uses this parameter to download feeds from the update servers.
The enabled
attribute specifies whether the feed must be processed by Feed Utility:
- If this attribute is
true
, Feed Utility processes this feed. - If this attribute is
false
, Feed Utility skips this feed.
Path
The Path
parameter specifies the path and the file name for a custom or third-party feed.
The following value types are supported:
- Path and filename on the computer where the Feed Utility binary is located.
- FTP or FTPS path and file name.
- HTTP or HTTPS URL.
- Network path (supported only in the Windows operating system).
This parameter has the following attributes:
- cert
Specifies the path to a certificate that will be used to download this feed via HTTPS or FTPS. This attribute is optional. If it is not specified, Feed Utility will not use a certificate.
Authentication type is provided in the Settings > Feeds > Feed
parameter of the Feed Utility configuration file.
For a feed of the email type, select a connection protocol by specifying POP3 or IMAP as a value for type attribute.
If a connection protocol is selected, specify the IPv4/Host name and the port of email server in the Path element.
Authorization
Basic authentication settings for custom or third-party feeds.
This parameter has two nested elements:
User
Encrypted user name for basic authentication on a server from which the feeds are downloaded.
Use the
--set-basic-auth
command-line option to set this parameter. For STIX feeds that are downloaded from the TAXII™ server, use the--set-taxii
option.This element cannot be empty.
Password
Encrypted password for basic authentication on a server from which the feeds are downloaded.
Use the
--set-basic-auth
command-line option to set this parameter. For STIX feeds that are downloaded from the TAXII server, use the--set-taxii
option.
If this type of authentication is not required, do not specify this parameter.
TAXII
The TAXII parameter specifies the location of a STIX feed. This element must contain the address of a Poll service of a TAXII server.
This parameter has the following attributes:
collection_name
The name of the data collection for this STIX feed.
version
The version of the TAXII server. The possible values of this attribute are
1
,2.0
, and2.1
. For a STIX 2.0 feed, specify a TAXII 2.0 server. For a STIX 2.1 feed, specify a TAXII 2.1 server.By default, the value of this attribute is
1
.
The following example demonstrates specifying the location of a STIX feed:
<TAXII collection_name="example-collection" version="2.1">http://192.0.2.10:9000</TAXII> |
TAXII server authorization settings
Credentials for TAXII server authorization.
This parameter has two nested elements:
User
User name for authorization on a TAXII server.
This parameter is stored encrypted. Use the
--set-taxii
command-line option to set this parameter. If you do not use this option and enter your user name as plain text, connection to the TAXII server will not be established.Password
Password for authorization on a TAXII server.
This parameter is stored encrypted. Use the
--set-taxii
command-line option to set this parameter. If you do not use this option and enter your password as plain text, connection to the TAXII server will not be established.
The following example demonstrates usage of this parameter:
<Feed> <Name>TAXII</Name> <TAXII collection_name="example-collection" version="2.1">http://192.0.2.10:9000</TAXII> <Authorization> <User>zQYq33rAY7dgImLtk8W0jQ==</User> <Password>OUYWpkPDoH+vv/IFfCrshw==</Password> </Authorization> </Feed> |
Filters
This element specifies filtering rules for the feed. Each filtering rule is defined in a Field
element.
The Filters
element is optional. If a Filters
element contains no nested Field
elements, Feed Utility treats this situation as if the Filters
element is omitted. If there is no Filters
element, no filtering is performed.
UrlMatcherField
This element defines how feeds with URL masks are processed by Feed Utility.
If you use Feed Utility as a part of Kaspersky CyberTrace together with Kaspersky CyberTrace Service, feeds that contain URL masks must be converted to binary format.
If you use Feed Utility without Kaspersky CyberTrace Service, you do not need to compile masks, so the UrlMatcherField
element is not required in the Feed Utility configuration file.
This element has a value and an optional toRegex
attribute:
- The value of this element specifies the name of the field in the feed that contains the URL mask.
- The optional
toRegex
attribute of this element defines how the utility must process the feed.- If the
toRegex
attribute is0
,false
, or is omitted, Feed Utility compiles the feed and creates binary files that can be used by Kaspersky CyberTrace Service. - If the
toRegex
attribute is1
, ortrue
, Feed Utility processes URL masks in the feeds so that they are converted to regular expressions.For each URL mask, Feed Utility creates a regular expression that matches all URLs covered by the URL mask. For example, if a URL mask is
botnet.example.com
, the regular expression for it is((\/{2}|^)botnet\.example\.com(\/.*|$))
. Like the URL mask, this regular expression matches all URLs in thebotnet.example.com
domain.The
toRegex
attribute can be set to1
only for those feeds that have thetype
field. Also, if theRequiredFields
element is present among the feed rules, it must contain thetype
field.
- If the
For json
output format, you can do the following:
- Compile feeds with URL masks to a binary file that can be used by Kaspersky CyberTrace Service.
If you use Kaspersky CyberTrace Service, the
UrlMatcherField
element is mandatory for feeds that contain URL masks: Malicious URL Feed, Phishing URL Feed, Botnet C&C URL Feed, and IoT URL Feed.<UrlMatcherField>%field_name%</UrlMatcherField>
- Convert URL masks to regular expressions.
<UrlMatcherField toRegex=1>%field_name%</UrlMatcherField>
- Do not convert URL masks.
Omit the
UrlMatcherField
element.
For csv
and txt
output formats, you can do the following:
- Convert URL masks to regular expressions.
<UrlMatcherField toRegex=1>%field_name%</UrlMatcherField>
- Do not convert URL masks.
Omit the
UrlMatcherField
element.
For openioc
and stix
output formats, omit this element. You cannot use this element with these output formats.
RequiredFields
This element specifies fields that are included in the processed feed. This element is mandatory for txt
and csv
output formats. If the RequiredFields
element is omitted, all fields of a record are written to the processed feed.
Field names are separated by a semi-colon (";"
). The slash ("/"
) in a field name indicates a nested field (in terms of JSON format).
This element defines fields in the resulting feed; it does not work like a filtering rule. For example, if a <RequiredFields>id;mask</RequiredFields>
feed rule is defined for a feed, the records in the processed feed will have only id
and mask
fields. Records that have at least one of the specified fields (id
or mask
) will also be included. Records that do not have at least one of the specified fields will be excluded because the absence of specified fields results in an empty record written to the processed feed. If you want to filter a feed so that only records with all the specified fields are included in resulting feed, you must use filtering rules. For information about using the RequiredFields
element together with filtering criteria, see subsection "Excluding records with missing fields" in the "Filtering rules" section.
Record fields are written to csv
and txt
formats in the order they are listed in the RequiredFields
element. Record fields are written in json
format in the order in which they appear in the source feed. For openioc
and stix
formats, the order of records is not defined; records are written in the order of processing.
FeedFields
This element lists all fields present in a feed.
Do not change this parameter. Feed Utility automatically writes field values to it.
RecordsCount
This element specifies the maximum number of records that will be included in the processed feed.
This element has two attributes:
- The
total
attribute of this element defines the total number of first-level records that must be included in the processed files. Feed Utility includes only those records that match the filtering rules. If this value is0
, this number is not limited, and all records are included.Note that the
total
attribute refers to first-level records, so the processed feed can contain more nested records than the value oftotal
. For example, the original feed contains the following data:[
{
"time": "23.02.1992 15:43",
"urls":[
{
"url" : "http://url.biz/sample1"
},
{
"url" : "http://url.biz/sample2"
}
]
},
{
"time": "24.02.1992 15:43",
"urls":[
{
"url" : "http://url.biz/sample3"
},
{
"url" : "http://url.biz/sample4"
}
]
}
]
You can specify the following settings for this feed in the Feed Utility configuration file:
<OutputFormat>csv</OutputFormat>
<RequiredFields>urls/url</RequiredFields>
<RecordsCount total="1" />
The processed feed will contain two records because they are nested in one first-level record.
http://url.biz/sample1
http://url.biz/sample2
- The
perFile
attribute of this element defines the maximum number of records that must be included in a single file. If the total number of records in one processed feed file is greater than this value, Feed Utility creates an additional feed file.For example, if
total
is1000
,perFile
is300
, and Feed Utility finds 650 records that match filtering rules, then three files are created (two files with 300 records and one with 50 records).This attribute cannot be used in the following cases:
- The
perFile
attribute cannot be used for feeds with URL masks that are compiled into binary format. - The
perFile
attribute cannot be used with custom feeds that have parsing rules with thetype
attribute set tourl
ordomain
types. Such feeds are also compiled into binary format by Feed Utility. To use theperFile
attribute for these types of fields, you must change the field type tocontext
.
- The
Parsing
This element contains parsing rules for custom feeds.
Rules for feeds of email type
Feed Utility can download third-party feeds of email type from a mail server.
Configure mail server connection settings in the following elements of the kl_feed_util
configuration file:
MailboxConnection
This element is obligatory, if POP3/IMAP are enabled in the
Path
element.This element has the following attributes:
use_tls
Specify
true
orfalse
values foruse_tls
attribute to enable or disable use of TLS/SSL for interacting with the mail server. The preset value istrue
.receive_period
The maximum value for this attribute is 360, the preset value is 7, the minimum possible value is 1.
Feed Utility processes only emails received after the last email feed update.
If an email feed had not been updated before, Feed Utility processes all the emails over the last number of days set in the
receive_period
attribute.
MailboxConnection/User
The value for this element is encoded. For connecting to mail server, Feed Utility encodes user name using the
--set-mailbox
parameter.MailboxConnection/Password
The value for this element is encoded. For connecting to mail server, Feed Utility encodes password using the
--set-mailbox
parameter.For checking connection with a mail server, use --check-mailbox-connection attribute.
MailboxConnection/Filters
This section allows to specify one or more filtering rules for mail server messages. This section is optional. For more information, see section "Filtering rules".
Parsing
To set the parsing rules for a feed of email type, see section "Parsing rules".
The following example demonstrates mail server connection settings:
<Feed enabled="true"> <Name>Email_OSINT</Name> <Path type="pop3">myemailserv.com:3125</Path> <MailboxConnection use_tls="false" receive_period="7"> <User>==asd</User> <Password>==ewq</Password> <Filters> <Filter field="from" condition="not_match">mywife@ya.ru</Filter> <Filter field="subject" condition="contains">Best IoCs ever</Filter> </Filters> </MailboxConnection> <Parsing type="messageBody"> <ip type="ip">(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})</ip> </Parsing> <Parsing type="messageAttach"> <Attach type="csv"> <ip type="ip">1</ip> </Attach> <Attach type="pdf"> <url type="url">1</url> </Attach> </Parsing> <Excluded> <Item>VALUE1</Item> <Item>VALUE2</Item> </Excluded> </Feed> |