Data Fields

the document below details the various fields that are extracted from the data that flare collects these fields can be used with the lucene query syntax example queries activities that have the domain name acme com in their title title "acme com" listings that have a price between $100 and $200 price >100 and price <200 retrieve an actor and their publications by their username author name\ lapifia specifying multiple keywords your organisation and fraud using time field metadata estimated created at >= now 1d actors from specific threads index\ forum topic and title\ actor name and keyword levels of support some documents have three levels of support each of them is documented in the following table level description 1 base attributes of the entity as they are part of the document’s identity, fields on this level have the highest chance to be present 2 addition to the level 1 attributes fields on this level add more value from an analysis point of view as they are not part of the document’s minimal identity, they can be omitted in some cases 3 complementary information fields on this level may be unavailable for some sources or unnecessary for basic analysis their presence depends on the source common fields each document contains a metadata field with subfields describing the origin of the data and some other attributes to diagnose the source of the document interesting fields are metadata name description id unique id of the document usually composed of the source and an internal id at the source metadata source name of the source or website from which the raw data was downloaded from since a site can have multiple domains that change over time, using names such as dream or wall st is more reliable metadata first crawled at date the document was first found useful for having an approximate date of the creation of a document metadata last crawled at date the document was last seen useful for having an approximate date of deleting a document on a site metadata estimated created at some documents may have their exact date of creation this field estimates the creation date using the date provided by the source if possible or metadata first crawled at this field therefore provides a better accuracy for the creation date than metadata first crawled at author name name of the author associated with the document in question this may be the seller name for a listing, the repository host on github, etc documents also contain a features field with subfields describing extracted keywords that might identify digital identities features name description features urls list of urls extracted from the document this can be urls from any scheme, including http, ftp, ssh, etc features emails list of email addresses extracted from the document features domains list of domains extracted from the document the list contains every valid variant of a domain, such that a document with the www flared io domain to have a list with both www flared io and flared io features reversed domains list of domains extracted from the document and the components reversed a document with www flared io will have a list with both io flared www and io flared this field is useful to index and quickly find documents matching a domain’s subdomains with suffix wildcard query features reversed domains\ io flared features ip addresses cidr list of ip addresses extracted from the document support both ipv4 and ipv6 the field type supports queries with cidrs such as features ip addresses cidr 127 0 0 0/8 features btc addresses list of bitcoin public addresses extracted from the document features cc numbers list of potential credit card numbers extracted from the document while the numbers have not been validated, each number matches a known bin and has a valid luhn identification number documents seller seller fields level 1 name description about descriptive or biographical text of the seller created at creation date of the seller’s account username pseudonym of the seller seller fields level 2 name description feedbacks list of comments or ratings posted on the seller’s page contact info seller’s information rating rating (out of 5 stars usually) given to the seller rating count number of ratings given to the seller rating score ratings given to the seller usually the sum of positive and negative feedback ship from regions from which the seller delivers the goods use ship from norm for normalized region across data sources ship to regions where the seller delivers the goods use ship to norm for normalized region across data sources public pgp fingerprint pgp key impression list associated with the seller public pgp uid list of pgp key ids associated with the seller transactions count number of sales made by the seller seller fields level 3 name description last 7 days rating rating average for the last 7 days given to the seller last 7 days rating count number of ratings given to the seller in the last 7 days last month rating rating average for the last month given to the seller last month rating count number of ratings given to the seller during the last month transactions amount total amount of sales made by the seller in usd finalize early enabled boolean indicating whether the seller requires payments at the time of purchase or via an escrow title title of the seller on the market generally an internal title earned by the experience or reputation of a seller last active at last activity date of the seller listing listing fields level 1 name description title ad title description description of the ad seller id internal id of the seller seller name descriptive name of the seller creation date creation date of the ad price price of the ad in usd currency price’s currency listing fields level 2 name description category id internal id of the category of the ad use category for normalized id across all data sources category name descriptive name of the category of the ad use categoy name norm for normalized name across all data sources ship to list of regions where the goods can be delivered use ship to norm for normalized regions across data sources ship from regions from where the goods are shipped use ship from norm for normalized regions across data sources rating rating given to the merchandise or the seller rating count number of ratings given to the merchandise or the seller feedbacks list of comments or ratings posted on the ad see below for the fields at feedback sold count number of sales of the ad listing fields level 3 name description escrow boolean indicating whether the transaction is via an escrow managed by the market platform last 7 days rating rating for the last 7 days given to the merchandise or the seller last 7 days rating count number of ratings given to the merchandise or the seller in the last 7 days last month rating rating for the last month given to the merchandise or the seller last month rating count number of ratings given to the merchandise or the seller in the last month shipping options list of shipping options available for the listing stock count remaining quantity of the goods view count number of times the listing has been seen forum profile forum fields level 1 name description username username of the user personal text descriptive or biographical text registered at approximate date of account registration forum fields level 2 name description contact info profile’s contact info last posted at approximate date of the last message last active at approximate date of the last connection public pgp fingerprint fingerprint list of the pgp keys associated with the account public pgp uid list of pgp key ids associated with the account usually contains a list of email addresses associated with the key rating user rating rating pos number of positive ratings given to the user rating neg number of negative ratings given to the user signature signature attached to each user’s message forum fields level 3 name description age age of the user avatar url to the user’s avatar comments count number of comments from the user following category ids categories followed by the user following user ids other users followed by the user location location of the user unlikely to be accurate use location norm for normalized value across data sources posts list of posts left on the user profile by other users see below for the fields at forum post posts count number of messages from the user realname real name of the user unlikely to be an actual real name timezone offset time zone used by the user title title of the user (member, moderator, administrator, etc ) tags tags associated with the user website website url of the user forum topic forum topic fields level 1 name description author id internal id of the original author of the thread author name nickname of the original author of the thread posted at date of creation of the thread title title of the thread forum topic fields level 2 name description category id internal id of the thread’s category use category id norm for normalized value across data sources category name name of the thread’s category use category name norm for normalized value across data sources category path internal id of the thread’s category and parent's categories forum topic fields level 3 name description first post preview preview of the first post content last reply at date of the last reply in the thread profile id related profile if the thread is on a profile page tags tags associated with the thread forum post forum post fields level 1 name description author id internal id of the author of the message author name nickname of the author of the message content content of the message posted at date of creation of the message parent post id parent post to which this post reply topic id internal id of the thread containing the message topic title title of the thread containing the message forum post fields level 2 name description there are no support level 2 fields for this type of document forum post fields level 3 name description post title title of the post rating pos number of positive ratings given to the user driller driller fields name description html url url of the original document cache url cache url from google is dork whether the document came from a dork query title title of the document author id internal id of the author of the message author name nickname of the author of the message author email email of the author of the message created at creation date of the document project name project’s name content content of the document size size in bytes snippets snippets of the document sha sha of the document is truncated whether the document is truncated host host url of the document (ex github com) fileype type of the file filename name of the file dirpath dirpath of the document user document’s user issue document’s issue project the project that the document comes from commit document commit code extracted code from the document paste paste documents are mostly raw data coming from paste sites such as pastebin paste fields name description author id internal id of the author author name username of the author content raw content of the paste document can be truncated if actual content is too large expire at expiration date of the paste document documents remain in flare's database even after the expiration date is truncated large documents are truncated in flare's database if true, content is a truncated version of the actual document posted at date the paste document was created size actual size of the paste document (bytes) syntax document’s syntax defined by the author title title given to the paste document title en english title given to the paste document domain domain fields name description name domain’s name registered at certificate registration date feed certificate transparency feed name identifier domain domain identifier cert data contains details about the issued certificates and the issuers subject domain ssl certificate’s subject issuer certificate issuer host host fields name description vulnerabilities the name of a vulnerability present on the host (e g cve 2018 15919) country code the country code of where the ip is located e g country code\ us service service running on an ip (e g http, ssh, ftp, mongodb, elastic, etc) tags other tag that give information about what this ip is doing possible values cloud,cdn,starttls,self signed,database,vpn,honeypot,iot,devops,videogame,ics,compromised,cryptocurrency,medical,tor, proxy leak leak fields name description links \[optional] name the email address associated with the leak passwords list of passwords leaked associated with the email for each entry in password list domain \[optional] domain name of breached company extra \[optional] details on the intrusion method if available hash clear password or hash hash type \[optional] hash function id unique series of numbers for the leaked credential imported at timestamp at which the password was added to our database this does not correspond to the date at which the password was leaked breached at (under source) \[optional] date of the breach (not always available) description (under source) description of the breach in english description fr (under source) description of the breach in french hash description (under source) \[optional] description of the hash function of the source hash type (under source) \[optional] hash function of the source if the leaked passwords are the same type if not, the hash type will be “none” id (under source) internal source name and year of breach leaked at (under source) date when the data became publicly available name (under source) \[optional] name of the company that was breached related urls (under source) \[optional] news articles related to the breach, if available url (under source) \[optional] breached website url (original source not always available) source id internal source name source params line line number where the leaked password was found inner objects feedbacks feedback are usually posted by buyers of an ad and is an essential part of the black market reputation system feedbacks fields name description feedbacks rated at date of the rating or comment feedbacks rating rating given by the buyer feedbacks comment comment posted by the buyer feedbacks username full or partial pseudonym of the buyer feedbacks purchase amount price of the purchase associated with the comment feedbacks purchase currency currency used by purchase amount feedbacks author id internal id of the author user user fields name description email email company company full name full name location location followers count number of followers contact info contact info fields name description contact info fdw forum fdw forum internal id contact info jabber jabber address contact info email email address contact info skype skype username contact info icq icq contact info irc irc nickname contact info ricochet ricochet contact info bitmessage bitmessage contact info btc bitcoin wallet contact info telegram telegram contact info discord discord shipping options shipping options options fields name description shipping options description description of the option shipping options price price of the option shipping options currency currency of the price of the option project project fields name description owner name name of the author owner id internal id of the author owner type author’s association with the document tags associated tags last activity at approximate date of the last activity language author’s language followers count number of followers forks count number of forks there are of this repository in the whole network is fork identifies if the repository is a fork commit commit fields name description committer name name of the committer committer id internal id of the committer committer email email of the committer author email email of the author sha sha related articles