Rate-limiting of identifier event feeds

when setting up an identifier, flare considers both past events from historical data on the flare platform, as well as continuously monitor events being ingested in the flare platform events matching the identifier's parameters will be added to its event feed in some cases, the volume of events matching an identifier may be unusually large when this happens, there is a risk that too many noisy or irrelevant events could flood the identifier’s event feed, which may cause performance issues when loading the feed to prevent this, we’ll apply a rate limit to the number of events added to that identifier’s feed what are the rate limiting thresholds? initial identifier creation when an identifier is created (or its parameters are modified), flare will evaluate all past events and add matching events to its event feed there is a limit of 50k events per source type (sub category) when generating past events for a given identifier continuous event discovery and matching if an identifier generates more than 1,000 events per hour in any source type (sub category) https //docs flare io/data sources , we’ll apply a rate limit this means any events beyond the 1,000 per hour limit will be dropped and won’t appear in the identifier’s event feed the limit resets every hour this rate limit only applies to newly discovered events each identifier is treated separately so if you have duplicate identifiers or the same identifier used in different tenants, each one will be rate limited separately how will i know if i have an identifier event feed being rate limited? you will see an indicator in the flare application, highlighting the identifier and the event category being rate limited what should i do if i have an identifier event feed being rate limited? if you have an identifier that is being rate limited, it is likely that the identifier needs tuning you can reach out to your se (if you are in trial/poc) or your csm (if you are a customer) to set up a call to tune your identifiers however here are some questions to consider when trying to make your identifiers more specific is the term you are using very generic? eg , a keyword identifier for github this will get a lot of hits and likely most of those events are noise consider choosing a more specific term are you only interested in specific severities? eg , you don't care about low severity events for this identifier are you seeing a lot of noisy events coming from look alike domains? if you have a short domain then there is a risk that the number of permutations we send for look alike detection is vast this will result in a lot of irrelevant look alike domain events in this case consider removing the ‘low’ severity filter from their identifier, this will filter a lot of the look alike event noise does your situation require to consistently monitor an identifier matching a high volume of events? our regular identifier event feeds might not be the best approach in that case we do offer other approaches based on api access to support these types of situations, please contact your flare representative to discuss options