Bugsink manages event storage efficiently using a combination of ingestion rate limits and smart retention. This approach ensures that you stay within system limits while keeping the most important data for debugging and analysis.
Ingestion rate limits control the rate of event ingestion, ensuring projects don’t exceed predefined limits within specific timeframes. Bugsink implements rate limits such as:
When a rate limit is exceeded, Bugsink returns an HTTP 429 status code, signaling to clients that they should back off temporarily. On the client side, this is typically implemented with a 60-second backoff, which helps alleviate pressure on the system. This allows ingestion to stabilize and prevents overwhelming the system.
Rate limits are configurable at the Project level, allowing system administrators to allocate resources according to the importance of each project. This flexibility ensures that more critical projects can be prioritized with higher limits, while less essential projects are restricted, optimizing overall system performance and preventing any single project from monopolizing resources.
Rate limits are essential for several reasons:
Fairness: Rate limits ensure that all projects receive a fair share of system resources, preventing any single project from monopolizing resources to the detriment of others.
Resource management: By controlling the rate of ingestion, rate limits help manage system resources effectively, ensuring that the system remains stable and responsive.
Preventing abuse: Rate limits help prevent abuse of the system by malicious actors or misconfigured clients.
This is against the backdrop of a system that is designed to be highly scalable and reliable.
Smart retention in Bugsink ensures that only the most relevant events are kept, without overwhelming storage capacity. Bugsink’s retention algorithm considers multiple factors:
Number of events: The relevance of each new event decreases as more similar events are generated. When an issue has a high number of occurrences, each subsequent event usually offers less new information. Bugsink reduces the implied sampling rate in such scenarios, retaining only the most significant data points.
Event age: Bugsink assigns an irrelevance score to events, influenced by their age. Unlike systems that use a fixed retention period (e.g., “keep everything for 30 days”), Bugsink’s approach considers the age of an event as just one of the factors affecting its retention. The algorithm applies a decay model, which means that older events are not immediately discarded but are assessed for relevance over time.
Bugsink’s retention algorithm is designed to be dynamic. When event volumes are low, retention periods may extend, allowing data to be retained for longer periods—sometimes years—if it remains relevant. This approach ensures that your most valuable data is kept while older, less significant data is discarded as needed.
Rate limits and retention serve complementary purposes in Bugsink:
Ingestion rate limits ensure that your project stays within limits for incoming events, acting as a first level of filtering to prevent overwhelming the system during bursts of activity or under heavy error loads. This ensures fairness across projects and keeps the system responsive.
Retention takes over after rate limiting, determining which events are worth keeping long-term. It balances event frequency and age-based analysis to decide what data is essential to retain. Bugsink’s retention strategy is more nuanced than simple time-based or count-based models, prioritizing high-value data rather than simply retaining or discarding based on age or count alone.
These mechanisms help avoid the pitfalls of simpler models used by many error tracking tools:
Traditional systems often use fixed limits (e.g., “10,000 events per month”). Once that limit is hit, you stop receiving potentially critical data, risking the loss of important information. Bugsink, by contrast, allows all events to be ingested while managing storage intelligently with retention.
Other systems may employ a fixed retention period (e.g., “30 days”), risking the loss of older data that might still be relevant. Bugsink’s model factors in context and relevance, meaning data can be retained for longer periods if warranted by its ongoing value.
By combining rate limits to manage input flow with smart retention to govern long-term storage, Bugsink provides a robust solution that ensures your most relevant data is preserved without overwhelming system resources or requiring manual intervention. This dual approach helps you focus on fixing issues rather than dealing with storage constraints.
