API Rate Limit inconsistency

[andyjames (Andy)]

I’ve spent much more time with this and now have a working solution. I am using bottleneck to handle the requests and sending lots of requests initially and then slowing it down after that using bottlenecks reservoir feature. This is to ensure that smaller batches are run faster. It includes a formula that works out the bottleneck settings based on the collection size so it remains performant at all collection sizes.

The simple solution is using an async await with forced buffer of 1 - 1.2 seconds in between each request.

I believe this is similar to the solution posted by @fbcto recently. I see await sleep(1000)

This time delay is needed for large collections but it is no good for people with smaller collections as it can run a lot faster taking advantage of 60 requests * 100 items a minute. (Weirdly the items api X-RateLimit-Remaining says it allows 120 but never gets close and looks more like its 60… see below)

One thing I have done is to use a recursive function that does one call to Webflow to get the remaining items. If it is below 90 it will wait until it is > 90 to run it again. This is working perfectly to get around issues that occur with the limits, especially when you get data from one collection and then another. The second collection will have a lower rate to start so this checking becomes important.

The issue I see based on hours of testing is that something really seems off with the count of the GET /collections/:collection_id/items and its returning value of X-RateLimit-Remaining It says 120 on a fresh start but I am not convinced.

The only way I can get this endpoint to work is to make sure I never drop below 60 remaining. If its 59 when the job finishes it will bomb! It’s like it’s counting 60 more than it should, or its not allowing for the 120 is says it is

One last thing. I want to re iterate. The above is all about the get items API. When I use the API to put data INTO Webflow I can run at 120 requests a minute and never have issues I can go all the way down to 1 item remaining. It seems strange to me that I do not experience the same issues with both API’s

[Fonsume]

This weird and totally unneccessary API behavior is wreaking havoc on our site. We use the Wordpress plugin “Webflow Pages”, which apparently uses the API for fetching the static and dynamic pages. This has resulted in us hitting the rate limits over and over again, taking the entire site offline. Not only that, but the API key is decoupled from the plugin and we have to re-integrate the entire site manually again.

To make matters worse, we’re using the API to offload older articles from the webflow cms into our wordpress database. Webflow themselves told me that to search through our 1000s of articles we should run 50-70 queries with offsets just to list all articles - but that also hits the rate limit.

This is so frustrating, Webflow.

[fbcto (Anders)]

I think the reason why we’re having issues with the GET method is due to a faulty redis caching mechanism on Webflow’s part (also in connection with support statements about Redis caching here: API Rate Limit inconsistency - #10 by jasondark).

There’s no caching in place for the other methods so we’re not seeing issues here.

[RDaneelOliwav]

I took note

[QA_Brandon (Brandon Mosco)]

@Fonsume,

Just an FYI, the WP Plugin does not connect to anything dynamic on Webflow. It is strictly for use with static pages.

I have also been informed that the Business Hosting plan does have a 120 rate limit, @andyjames you are correct there. We were not informed of this change until recently.

Have a Great Day and Happy Designing,
~ Brandon

[Jeandcc (Jean Costa)]

Just dropping a useful link here: bottleneck - npm

This lib helped me deal with Webflow rate-limit pretty well. Very solid interface. Here’s how I’m using it:

import Bottleneck from "bottleneck";
import Webflow from "webflow-api";

const wfLimiter = new Bottleneck({ minTime: 1000 });

wfLimiter.on("failed", async (error, jobInfo): Promise<void | number> => {
  const queuedJobs = wfLimiter.jobs("QUEUED");
  const runningJobs = wfLimiter.jobs("RUNNING");
  const executingJobs = wfLimiter.jobs("EXECUTING");

  const idxOfThisJob =
    (executingJobs.findIndex((id) => jobInfo.options.id === id) || 0) +
    runningJobs.length +
    queuedJobs.length;

  // Only retry if it's rate-limit error, and only up to 10 times
  if (error.code === 429 && jobInfo.retryCount < 10) {
    const msOfBackOff = 2 ** jobInfo.retryCount * 1000 + idxOfThisJob * 1000;
    return msOfBackOff;
  }
});

To make calls to Webflow, just use it like so:

const api = new Webflow({ token: _API-TOKEN-HERE_ });
const siteData = await wfLimiter.schedule(() => api.site({ siteId }));