Use Stripe Efficiently: Rate Limiting and Data Caching

Successful integration of payment processing platforms requires careful considerations about performance and reliability of your service. Whether you are gearing up for a major sales campaign or experiencing unexpected spikes in payment or billing-related requests, optimization techniques and strategic approach to caching are likely at the top of your todo list. This guide breaks down how rate limiting works with Stripe and how to handle peak traffic smoothly. We will also explore data caching practices with respect to PCI DSS compliance, ensuring your application runs efficiently without breaking privacy and security standards.

Not sure which platform best caters for your needs? Read a detailed and pragmatic comparison that helps you choose between Stripe and PayPal.

Stripe is designed to support high volume transactions via robust APIs. The platform ensures security and compliance, especially when it comes sensitive financial data. Nevertheless, it is always a good idea to limit the dependency on a third-party system and plan ahead. Duplicating Stripe data in your system helps you mitigate the risk of Stripe being inaccessible – no matter how unlikely that seems to be. A smart combination of rate limiting and data caching helps you grow your customer base at scale.

Table of Contents

• Rate Limiting
• Graceful Handling of Excessive Calls
• Data Caching
• Summary

Rate Limiting

Stripe has a generous basic limit of 100 read and 100 write operations per second. Bear in mind, this applies to live data. In the sandboxed environment, the limits are significantly lower – a quarter of the live data limits to be exact.

The general advice is to treat these limit as hard boundaries. In fact, some endpoints have much stricter rate limiting – Search API, for example, is highly useful, but it is also more strictly shielded against excessive use.

API requests vary not only in frequency, but also in time they take to serve. Some requests take longer to process and, if accumulated, could put a lot of pressure on CPU and other server resources. As a counter measure Stripe provides a concurrency limiter. If you provide pay-as-you-go plans to your customers, or simply charge them based on how much they use your products or services, then be aware that metered usage is limited to a single concurrent request per customer (per meter).

Since Stripe is highly flexible, allowing you to increase the base limit up to 100 000 requests per second. However, that requires contacting their sales team. I have never tried this option and I am not sure what the cost implications are.

See the documentation for full details.

Graceful Handling of Excessive Calls

Stripe uses the standard 429 response status whenever your request gets rejected due to rate limiting. It is always a good idea to have a built-in retry mechanism with an exponential backoff. Curious about implementation details? This article got you covered.

Ideally, you proactively implement a strict rate limiting on your end to prevent being blocked by Stripe in the first place. A Token Bucket algorithm is a common solution to this problem.

Stripe deals with concurrency issues on your behalf as well. That is why you might occasionally encounter an object lock timeout – Stripes sends a response code lock_timeout. Apart from retries with increased delays, it is best to avoid making concurrent modifications altogether. Instead, queue up modification requests to make them run sequentially – consider using reliable queues such as Amazon’s SQS, Google’s PubSub or RabbitMQ. This also implies that updates should be idempotent.

Beware of asynchronous processing, which applies to metered usage requests, for example. In this case, it is best to listen for webhook events (v1.billing.meter.error_report_triggered) to intercept processing failures and act accordingly (send an event to a dead-letter queue and schedule a retry).

Reporting and data migrations can too lead to massive spikes in API requests, especially if your customer base grows large. Except for the established pattern of rate limiting on your end, you can also consider tapping into designated infrastructure provided by Stripe. Namely, the Stripe Data Pipeline. You can try the service for free. After that, there will be a constant charge per transaction event.

Data Caching

Let me start with the obvious. Do not store credit card details. While Stripe provides general guidance on the subject, it is not worth the hassle.

How about customer data? Well, if possible, don’t store any personal details – such as emails, names, addresses etc. This is especially important if your business has to comply with GDPR or similar regulations.

Here is what you can safely store in your own database or cache to the tiniest level of detail:

• Product catalogue: This is very much static information about your products and services represented in Stripe. Since they are generic and typically don’t change too often, it is safe to make a copy in your own system, including full product and price details.
• Subscriptions: They keep track of which products or service a particular customer subscribed for without revealing any sensitive information. If you are in a subscription business, storing this information and keeping it up-to-date is arguably the most important part of your solution.
• Metered usage: By design the metrics come from your system and it should indeed remain the source of truth.
• Transaction history: Stripe provides detailed export of processed transactions, including refunds and failed payments. Just make sure to not store any sensitive information, such as credit cards or personal details.

There are no restriction for how long you can store or cache the data. Provided you adhere to privacy and security standards you are free to store as much or as little as required. For the sake of scalability, and to reduce dependency on Stripe, it is advisable to duplicate Stripe records in your own cache or database. Storing static data about products or prices is straightforward. The hard part is keeping dynamic data updated – typically subscriptions.

Let me conclude with a suggestion on how your data model could look like. Take it with a grain of salt and question whether it addresses your unique needs.

Customer:

• Entity ID: This is a business key, that identifies either a user or another entity in your system. You can always propagate this information into the Customer object in Stripe as metadata.
• Stripe ID: Customer ID in Stripe.
• Status: Active or churned, for example.

I would skip storing any personal or financial information. On that note, you most likely have some sort of personal information about your customers already. The Customer record in Stripe is likely a result of a new sign-up on your end, collecting the user’s email address and other personal information.

Product:

• Code: A unique code or a human readable identifier of the product.
• Stripe ID: Product ID in Stripe.
• Name: This can be a copy of what is in Stripe.
• Description: Provides additional context.

Price:

• Price ID: An internal unique identifier.
• Stripe ID: Price ID in Stripe.
• Product Code: There can be several prices associated with the same product. This is useful when you want to make a distinction between a monthly and an annual plan, for example.

Pro tip: If your business requires a fine grained control over which features or services are accessible based on the subscription plan, consider leveraging the concept of entitlements. I will unpack on this topic in a separate post.

Subscription:

I could come up with a rich model comprising various fields that might be of interest. However, a simpler and a more powerful solution could be storing an entire subscription object (JSON) literally as it came from Stripe. For this to work you need to ensure correct synchronization with the subscription state in Stripe, as it might change at any moment (upgrades, cancellations, failed payments, change in quantity, adding other products, discounts ..).

Tools to use:

• Subscribe to webhook events, such as customer.subscription.created / updated / deleted etc.
• Set expiry on each object you store in your cache / database an re-fetch it from Stripe once expired.

Summary

Integration with payment processing platforms like Stripe requires thorough planning around performance, reliability, and compliance aspects. This is especially crucial when preparing for high-traffic events or scaling up operations to accommodate growing customer demands.

Key takeaways:

• Duplicate Stripe data in your system: Decrease dependency on third-party systems like Stripe, and duplicate data within your system. This includes static data, such as a product catalogue. Avoid storing sensitive information, such as credit cards or personal details.
• Cache dynamic data: Subscriptions are a prime example of data that might change often or unpredictably. Store subscription details, but know when to refresh them. Employ webhooks and cache eviction to re-fetch the fresh state from Stripe every so often.
• Plan for peak traffic: Implement retries with exponential backoff to manage peak traffic smoothly and comply with Stripe’s rate limiting. It’s also beneficial to preemptively enforce rate limiting on your end through algorithms like Token Bucket. Prevent concurrent modifications – ensure data integrity via sequential processing through reliable queues, such as Amazon SQS, Google PubSub or RabbitMQ.

In conclusion, successfully integrating Stripe within your service demands a strategic approach to rate limiting and data caching. By carefully planning these aspects and adhering to best practices and compliance standards, you can achieve seamless payment processing that scales with your business needs.