I’ve spent much more time with this and now have a working solution. I am using bottleneck to handle the requests and sending lots of requests initially and then slowing it down after that using bottlenecks reservoir feature. This is to ensure that smaller batches are run faster. It includes a formula that works out the bottleneck settings based on the collection size so it remains performant at all collection sizes.
The simple solution is using an async await with forced buffer of 1 - 1.2 seconds in between each request.
I believe this is similar to the solution posted by @fbcto recently. I see await sleep(1000)
This time delay is needed for large collections but it is no good for people with smaller collections as it can run a lot faster taking advantage of 60 requests * 100 items a minute. (Weirdly the items api X-RateLimit-Remaining says it allows 120 but never gets close and looks more like its 60… see below)
One thing I have done is to use a recursive function that does one call to Webflow to get the remaining items. If it is below 90 it will wait until it is > 90 to run it again. This is working perfectly to get around issues that occur with the limits, especially when you get data from one collection and then another. The second collection will have a lower rate to start so this checking becomes important.
The issue I see based on hours of testing is that something really seems off with the count of the GET /collections/:collection_id/items and its returning value of X-RateLimit-Remaining It says 120 on a fresh start but I am not convinced.
The only way I can get this endpoint to work is to make sure I never drop below 60 remaining. If its 59 when the job finishes it will bomb! It’s like it’s counting 60 more than it should, or its not allowing for the 120 is says it is
One last thing. I want to re iterate. The above is all about the get items API. When I use the API to put data INTO Webflow I can run at 120 requests a minute and never have issues I can go all the way down to 1 item remaining. It seems strange to me that I do not experience the same issues with both API’s
