Rate-limiting of Identifier Event Feeds

when you set up an identifier, both past events from historical data in flare and new events as they are ingested are evaluated against the identifier's parameters matching events are added to your events feed in some cases, the volume of matched events is unusually large, which risks flooding the feed with noisy or irrelevant events and can cause performance issues when the feed loads to prevent this, a rate limit is applied to the number of events added to that identifier's feed in a specific time period rate limiting thresholds rate limiting happens at two stages, when an identifier is first created and during ongoing event discovery initial identifier creation when an identifier is created or its parameters are modified, all past events are evaluated and matching events are added to the events feed two separate limits apply when matching these past events a per identifier limit of 50k events for each data source category docid\ bcuhc 7rp5zrfmdiuqpvb this is the maximum number of past events an identifier can pull per selected category when it is first created since each category has its own limit, an identifier that draws from several sources can pull in more than 50k past events overall a per tenant limit of 250k events per hour this caps the total volume of past events matched across all new identifiers in a tenant within a given hour this protects the tenant from being overloaded when many identifiers are created or imported in a short span this limit resets every hour continuous event discovery and matching after an identifier is created, new events are continuously evaluated as they are ingested and are matched against the identifier the following rules apply to these newly discovered events a per identifier limit of 1,000 events per hour applies for each data source category this is the maximum number of newly discovered events an existing identifier can add to its events feed each hour any events beyond this limit will be dropped and won’t appear in the identifier’s events feed the limit resets every hour this rate limit only applies to newly discovered events each identifier is treated separately, so if you have duplicate identifiers, or the same identifier used in different tenants, each one is rate limited on its own identifying a rate limited identifier if one of your identifiers is rate limited, you will see an indicator in the flare application highlighting the identifier and the event category being rate limited what should i do if i have an identifier event feed being rate limited? if you have an identifier that is being rate limited, it is likely that the identifier needs tuning you can reach out to your se (if you are in trial/poc) or your csm (if you are a customer) to set up a call to tune your identifiers however here are some questions to consider when trying to make your identifiers more specific is the term you are using very generic? eg , a keyword identifier for github this will get a lot of hits and likely most of those events are noise consider choosing a more specific term are you only interested in specific severities? eg , you don't care about low severity events for this identifier are you seeing a lot of noisy events coming from look alike domains? if you have a short domain then there is a risk that the number of permutations we send for look alike detection is vast this will result in a lot of irrelevant look alike domain events in this case consider removing the ‘low’ severity filter from their identifier, this will filter a lot of the look alike event noise does your situation require to consistently monitor an identifier matching a high volume of events? our regular identifier event feeds might not be the best approach in that case we do offer other approaches based on api access to support these types of situations, please contact your flare representative to discuss options