Rate-limiting of Identifier Event Feeds
When you set up an Identifier, both past events from historical data in Flare and new events as they are ingested are evaluated against the Identifier's parameters. Matching events are added to your Events feed. In some cases, the volume of matched events is unusually large, which risks flooding the feed with noisy or irrelevant events and can cause performance issues when the feed loads. To prevent this, a rate limit is applied to the number of events added to that Identifier's feed in a specific time period.
Rate-limiting Thresholds
Rate limiting happens at two stages, when an Identifier is first created and during ongoing event discovery.
Initial Identifier Creation
When an Identifier is created or its parameters are modified, all past events are evaluated and matching events are added to the Events feed. Two separate limits apply when matching these past events:
- A per-Identifier limit of 50k events for each data source Category. This is the maximum number of past events an Identifier can pull per selected Category when it is first created. Since each Category has its own limit, an Identifier that draws from several sources can pull in more than 50k past events overall.
- A per-Tenant limit of 250k events per hour. This caps the total volume of past events matched across all new Identifiers in a Tenant within a given hour. This protects the Tenant from being overloaded when many Identifiers are created or imported in a short span. This limit resets every hour.
Continuous Event Discovery and Matching
After an Identifier is created, new events are continuously evaluated as they are ingested and are matched against the Identifier. The following rules apply to these newly discovered events:
- A per-Identifier limit of 1,000 events per hour applies for each data source Category. This is the maximum number of newly discovered events an existing Identifier can add to its Events feed each hour. Any events beyond this limit will be dropped and won’t appear in the Identifier’s Events feed. The limit resets every hour.
- This rate limit only applies to newly discovered events.
- Each Identifier is treated separately, so if you have duplicate Identifiers, or the same Identifier used in different Tenants, each one is rate limited on its own.
Identifying a Rate-limited Identifier
If one of your Identifiers is rate-limited, you will see an indicator in the Flare application highlighting the Identifier and the Event Category being rate-limited.
What should I do if I have an identifier event feed being rate-limited?
If you have an identifier that is being rate-limited, it is likely that the identifier needs tuning. You can reach out to your SE (if you are in trial/POC) or your CSM (if you are a customer) to set up a call to tune your identifiers.
However here are some questions to consider when trying to make your identifiers more specific:
- Is the term you are using very generic? eg., a keyword identifier for GitHub. This will get a lot of hits and likely most of those events are noise. Consider choosing a more specific term.
- Are you only interested in specific severities? eg., you don't care about low severity events for this identifier.
- Are you seeing a lot of noisy events coming from Look-alike domains?
- If you have a short domain then there is a risk that the number of permutations we send for look-alike detection is vast. This will result in a lot of irrelevant look-alike domain events.
- In this case consider removing the ‘low’ severity filter from their identifier, this will filter a lot of the look-alike event noise.
- Does your situation require to consistently monitor an identifier matching a high-volume of events? Our regular identifier event feeds might not be the best approach in that case. We do offer other approaches based on API access to support these types of situations, please contact your Flare representative to discuss options.