API Rate Limit inconsistency

I performed a simple test to showcase the problems with the current implementation of the API rate limiter, and to demonstrate that the claimed rate limit of 60 requests per minute, as per the documentation, is wrong or misleading.

Here’s what I did:

  1. Reset the webflow api key (to eliminate any past contamination)
  2. Performed a simple HTTP GET on a collection = 1 request
  3. Repeat that single request every 5 minutes (every 300 seconds)

The expected outcome:
The API rate limiter should return a value of “remaining” of 59 - on all requests.

The actual outcome is charted below, but shows that the rate limit is not reset every minute and in fact is depleting over time :exclamation:

3 Likes

Hi @fbcto,

our client eventually made use of their Premium Support with Webflow, but all we learned that way is that the servers use a “bucketed approach” to rate limiting and essentially said “well, it’s the way it is”. I had performed very similar tests to yours, though not as well structured and without pretty graphs, and in those it seemed even less reliable with sudden jumps.

The only “solution” we found, which Webflow’s own premium support did not deny, was to flood the server with requests until a 429 response is encountered, then pause-and-retry until a request makes it through. Rinse and repeat.

Hope that helps :slight_smile:

3 Likes

Thanks for your input, @DukeDiamond. I will look into a"flood-and-pause" solution.

For the record, here’s the answer from Webflow “support” regarding this issue:

Thanks for reaching out to Webflow.

Here’s a link to an explanation of how we do bucket rate-limiting at Webflow:

Rate limiting with Redis | Chris O'Hara

The rate limit is not strictly 60 requests per minute on the exact minute boundaries, which is why it doesn’t reset back to 60 at the top of the minute.

If you ensure that your app or server is making requests at that rate then you shouldn’t encounter any further issues.

Let me know if you have any other questions, and I’ll be happy to assist you further!

That did not answer my question at all… and I’m really disappointed with the level of “support” received from Webflow. If this is the norm I would much rather setup and maintain a wordpress installation * ugh* :frowning_face:

1 Like

I did some additional testing to see how the API responded to different workloads, to better design a possible solution or work-around to this rate limit madness.

  1. I reset my API token
  2. I set up a new timed function to perform a single HTTP GET every 1 minute
  3. I ran it for a few days and collected response data

Here’s the result:

The API will depleted over time, eventually resulting in a rate limit hit error. That is an established fact now, also based on my observations described above.

However, what’s really troublesome, is that I observed an unavailability period, where the API is returning “rate limit hit” error instead of resetting the counter, on average 28,58 minutes long. That is unavailability of 28,5 minutes on every rate limit reset. Let that just sink in for a moment… Throttling is okay, but mind you, I did 1 request per minute which is well within the rate limitations, so this kind of unavailability on a claimed-to-be production ready API is UNACCEPTABLE. :rage:

2 Likes

My observation over the last few years is that Webflow has little interest in supporting developers. I guess designers, not developers, are their target market and therefore they focus their energy where they will get the biggest bang for buck. This is an understandable business decision.

However this still doesn’t change the fact that working with the CMS API is a frustrating experience. There seems to be no appetite to improve it either. Their JS API wrapper library hasn’t had a meaningful commit in 3 years and open issues on the library go ignored. I have seen pull requests where community members have taken the time to add functionality which go ignored.

I’ve worked on a few projects that required integration with the CMS API. I learnt my lesson, and now categorically advise against touching the CMS API when anyone asks me. It’s just not worth the pain when there are so many other developer friendly platforms out there.

I do hope to see Webflow shift to supporting developers in a more meaningful way. Making it easier to integrate and develop with Webflow would win over more of the tech guys within teams and lead to increased adoption IMO. This would only benefit Webflow.

Just my 2 cents.

9 Likes

Thanks for your input, @jasondark
Which of the developer friendly platforms would you recommend over Webflow then?

It wouldn’t be good form for me to leave that sort of a post on Webflow’s own public forum :slight_smile:
But feel free to message me if you have a specific question that I might be able to help you with.

It’s not just developers. Full stack design hackers like me using zapier and integromat are hitting big walls. Just this week - can’t create an ecommerce product with IPAAS tools, can’t upload a file creating an item (webflow module is crashing integromat dev tools extension.

2 Likes

Yep I see your problem. It’s actually quite strange - most development platforms encourage integration and try to make life easy. Webflow seems to be doing the opposite? Maybe there is a business reason for it, I’m not sure. Or maybe this is just way down their priority list. It would be good to hear something from them but I certainly won’t be holding my breath.

1 Like

I’ve had a few mails back and forth with Webflow support and I’m starting to get the feeling that they dont even know what’s going on with the API themselves. Or maybe they do, but don’t know how to fix it…

Here’s the latest Webflow support answer, directing me to a Redis throttle implementation issue (on the API end). If you can make any sense of this, please let me know… :slight_smile: :see_no_evil:

This GitHub post is public: Ratelimit issue · Issue #38 · chriso/redback · GitHub so it is not private to just us.

Basically it tells us that since you are only performing a call once every minute, you are staying within the same polling bucket and therefore the bucket is not resetting. What you could do is switch your polling frequency to less than 10 seconds instead of 5 minutes that the rate limiter will most likely fix itself. Within that GH issue is a link to what someone else did to fix this rate limit issue.

Now, this is a one-off case for your issue and likely that it is specific to the polling frequency that you are using. It does appear that some adjustment has to be done on your end within the code you are using.

So “the fix” is to change my polling strategy to prevent falling into the same bucket on every request. So if you need to do a single request every minute (e.g. polling CMS data) you must accept a 28-minute unavailability period after every 60 cycles - meaning 7 hours of downtime every day (as I described and documented earlier in this thread).

I will continue my tests…

In answer to your original question about a solution, the best way I have found is to use a backoff and retry mechanism with a retry that persists for a long time.

I have used GCP pubsub to make this work. This is a (very simplified) overview of what I set up and the order of events that take place:

  1. My own database has a new entry.
  2. This triggers a webhook to a cloud function.
  3. The cloud function creates a pubsub message that a new entry has been created.
  4. Another cloud function listens to messages to this topic
  5. On receipt of a new message, this cloud function updates Webflow accordingly.
  6. If the rate limit is hit, the process fails.
  7. If the process fails, it continues to retry until it succeeds for up to 1 week (this is a cloud function setting you can enable in GCP).

Not the most elegant solution, but it does work pretty well. You could end up in a situation where a newer update gets through before an older one that is being retried, so I probably should write something to handle that edge case when I get time…

Thanks for a description of your workaround, @jasondark

I got this from support last night:

I wanted to start by saying thanks for the constructive and thoughtful feedback. I’ve shared this with our team.

I also want to say thanks for reporting this behavior. This is an issue that was unknown to us, and you’re reporting it will help improve our API not only for yourself but for many other users as well.

The team is planning on rolling out a fix for this, but I don’t have a time frame for when that might be available. In the meantime, the only workaround when using the API is to use the methods mentioned by our engineers in the previous message.

As soon as we have an update on a solution, I’ll circle back with you and make sure you are notified.

Based on your experience with Webflow support an actual fix of this issue is not going to happen any time soon, so I’m stuck with trying to make a workaround for our setup. GCP pubsub is not an option in our current setup, so I will have to come up with another way to handle the rate limit.

I have tried different request patterns to find the “sweet spot”, but the remaining count has been very inconsistent up until now (going from 1 to currently 5 requests per minute).

Similar to what @DukeDiamond suggested, I’m also trying to perform N<60 requests on a 5-minute cycle, and once N requests have been submitted to the API “flood” the API with the remaining count to hopefully reset the polling bucket.

I will keep you posted on my progress…

While we are at it. The get items limit of 100 max is riDiculArse!

3 Likes

Interesting to hear that they had no idea about the rate limit issues that have been getting raised on these forums and the wishlist for years :joy:

Another option you could look at is building a queuing mechanism for your requests. Then you can try to execute your requests in sequential order. If one fails, you pause your queue until the first one succeeds, and so on so forth. I may have to move to something like this myself if I get into issues with GCP pubsub’s asynchronous nature meaning that a failed request may not get through before newer ones do.

4 Likes

I am sure glad I found this thread after seeing these issues I was baffled.

I’m using https://github.com/SGrondin/bottleneck to limit my queries with mixed results on different endpoints.

The main issues are with using the GET /collections/:collection_id/items

The docs say the limit is 60 requests per minute and these headers that come back.

X-RateLimit-Limit & X-RateLimit-Remaining

On my requests I am seeing X-RateLimit-Limit is 120 not 60

But when it gets anywhere near 50-60 without a hard 1.2 second bottleneck it fails even though its no where near the 120 limit.

You cannot even rely on X-RateLimit-Remaining as its not accurate. It will say there are 59 left and then throw a 429.

Even if I wait an hour and try again without any activity is starts with X-RateLimit-Remaining at low numbers like 45.

What is really weird is I do not have this issue with the PUT /collections/:collection_id/items/:item_id

This PUT endpoint handles just fine. I can index 6000 items to Webflow smoothly at 120 per minute and never have any failures. X-RateLimit-Remaining gets between 5-15 but never fails.

The same cannot be said for the GET items endpoint.

This inconsistency between the GET and PUT requests seems odd.

3 Likes

I experience the same problems as described above. The API rate limits are very obviously and clearly broken. The inconsistencies make it nearly impossible to use the API in any serious integration.

I would be happy to hear a response from Webflow support here.

3 Likes

Hi @andyjames can you tell me what plan are you on?

The proper approach seems to be like @jasondark suggested on April 9.

If you, like me, feel that setting up a queue and auto-failover system is a bit too much, I have created this small script that you can plug into your existing code and handle any rate limit errors for you with very little change to your existing code.

The biggest disadvantage with this script is that it has the risk of timing out, if the webflow API is not providing a new pool of requests within the script timeout you have set up.

1 Like

Hi @QA_Brandon the site is on a full hosting package. We are testing with maximum records of up to 10,000

I’ve spent much more time with this and now have a working solution. I am using bottleneck to handle the requests and sending lots of requests initially and then slowing it down after that using bottlenecks reservoir feature. This is to ensure that smaller batches are run faster. It includes a formula that works out the bottleneck settings based on the collection size so it remains performant at all collection sizes.

The simple solution is using an async await with forced buffer of 1 - 1.2 seconds in between each request.

I believe this is similar to the solution posted by @fbcto recently. I see await sleep(1000)

This time delay is needed for large collections but it is no good for people with smaller collections as it can run a lot faster taking advantage of 60 requests * 100 items a minute. (Weirdly the items api X-RateLimit-Remaining says it allows 120 but never gets close and looks more like its 60… see below)

One thing I have done is to use a recursive function that does one call to Webflow to get the remaining items. If it is below 90 it will wait until it is > 90 to run it again. This is working perfectly to get around issues that occur with the limits, especially when you get data from one collection and then another. The second collection will have a lower rate to start so this checking becomes important.

The issue I see based on hours of testing is that something really seems off with the count of the GET /collections/:collection_id/items and its returning value of X-RateLimit-Remaining It says 120 on a fresh start but I am not convinced.

The only way I can get this endpoint to work is to make sure I never drop below 60 remaining. If its 59 when the job finishes it will bomb! It’s like it’s counting 60 more than it should, or its not allowing for the 120 is says it is

One last thing. I want to re iterate. The above is all about the get items API. When I use the API to put data INTO Webflow I can run at 120 requests a minute and never have issues I can go all the way down to 1 item remaining. It seems strange to me that I do not experience the same issues with both API’s

2 Likes