Open Web

the open web category includes data collected on various sources on the internet, also known as "clear web" for most open web sources, any identifiers you create are used to run periodic searches the identifier data is disclosed to third parties, but these are reliable platforms that support our collection, such as shodan or github source types pastes the pastes subcategory includes various websites that allow for anonymous sharing of text content, including pastebin (pastebin com) these websites are typically used by malicious actors looking to share content between each other (between different computers for example) there are two main paste collection processes certain paste websites have apis or historical paste lists that our systems monitor, download and store when paste links are found on any document in the entire system, they are added to a queue and crawled as well all pastes found through the 2 processes are collected, stored and searchable in flare web accounts the web accounts category monitors a list of websites that mirrors the coverage of the sherlock project's osint tool a list of sites currently covered by the tool can be reviewed https //raw\ githubusercontent com/sherlock project/sherlock/master/sherlock project/resources/data json we only match usernames with web accounts to learn more about how to set up usernames as part of identity identifiers, see https //docs flare io/configure identifiers#sx xy source code the source code subcategory includes data collected on platforms that include mainly source code, such as github and docker hub github flare collects data from github in the following manner docker hub flare collects data from docker hub in the following manner secret detection secret detection is the process of identifying sensitive information related to your organisation that may be unintentionally exposed on platforms like github or dockerhub we scan all github and dockerhub events that flare finds using a set of rules designed to detect patterns that match known secret formats some secrets are classified as generic these are common patterns that can easily trigger false positives, making them noisier and less reliable indicators of a real leak because of this, events are scored differently depending on whether the detected secrets are generic or not more details can be found https //docs flare io/source code secret detection stack overflow coverage discontinued with the advent of ai chat bots the use of stack overflow had reduced to such a level that we were no longer seeing valuable events in the application flare used to monitor stack overflow and historical events from this source are still available in the application events were created if any keyword or domain identifier are disclosed in a question on the forum google the google subcategory is used based on specialized queries (dorks) our experts maintain a large list of dorks, that are adjusted based on the domain and keyword identifiers all results are stored in the database hosts the hosts subcategory includes data related to an organization's ssl certificates and open ports it uses mainly third party services such as shodan searches are run on these services based on the domain and ip identifiers shodan shodan is a tool that discovers and indexes any devices that are connected to the internet, where they are located and who is using them it allows you to keep track of all the computers on your network that are directly accessible from the internet the tool is known to be used by malicious actors that investigate potential victims and their vulnerabilities more granularly, for every ip, shodan will look at most ports open, the service running on the port, versions of systems, associated vulnerabilities, and more depending on the use case here is a full list of all ports shodan scans, and the amount they have in their database for each https //www shodan io/search/facet?query=%2a\&facet=port in flare, shodan data is integrated to your identifiers monitoring as long as you check the hosts box in either one of the following ways a) domain identifiers whenever you create a domain identifier and check the hosts category, we will automatically send queries to shodan on a daily basis to see if the domain has appeared in any of its indexed hosts b) subdomain identifiers whenever you create a domain identifier and enable subdomains, we will query shodan using the domain and subdomain names we also resolve the ip addresses ourselves and query shodan c) ip address identifiers whenever you create an ip identifier, we will query shodan to return the data it has regarding that specific ip address d) ip address range whenever you create an ip range as an identifier, we will query shodan to return the data it has for each ip address in the range shodan banners shodan collects what it calls banners in practice, the information contained in a banner will differ depending on the port and service running for example, for an http/https service, the data in the banner will be the content of the response headers for a port 22 / ssh service, at a minimum it will contain the ssh service version, but could also provide the host key algorithms and compression algorithms used for more on banners you can use the search query port \<port number>in flare search bar to explore different ports, or look at this shodan article on banners buckets the buckets subcategory queries a database that indexes all publicly available cloud buckets from azure and aws we use this tool with identifiers of type domain and keyword, and search sensitive items particularly, given the large amount of noise present in cloud buckets data enrichment regroupment by similarity if items are identical, they are merged in a single card with the and x similar activities mentioned the factors to consider activities identical vary by source but generally include the content regroupment by project for sources that have specific grouping mechanisms, such as github repositories, results in the same group or repository are merged in a single card dork types if a certain activity was found with a specialized query ("dork"), the name of the dork will be indicated in a tag as well as on the right hand side panel, as shown in the image above tags include themes such as private pem certificate or django secrets scoring scoring on open web activities varies by source for anything related to network infrastructure, each port\ ip combination will be assigned a score an array of factors are considered in order to assign the score, including type of service running, port, associated vulnerability, associated ssl certificate, etc for example, unless the operating system version on the host contains known vulnerabilities (cves), a port 443 with an up to date certificate will likely be rated at risk score of 1 , which is the minimum at the other end, an elasticsearch service with its port open to the web, or a microsoft smb service will likely be rated at a risk of either 3 or 4 for any document or code, the system tries to extract sensitive information such as api keys, encryption keys, etc if successful, the score is adjusted to 3 or 4 depending on the severity of the finding related articles