Yay, me too \o/ Webflow is awesome 
3 Likes
All engines are working now 
4 Likes
Agreed -
I have made the decision today to abandon WEBFLOW CMS which allows me to host on multiple servers (for backup) and repoint DNS records when this happens. Also allows me to manage backups myself.
1 Like
Any suggestions for contacting support for future emergencies?
3 Likes
WEBFLOW WEBSITES ARE DOWN - no response from WEBFLOW …
Not worried about pricing this second - worried that I am losing customers today …
Respond WEBFLOW
Ours are okay - testing from France.
When something happens our servers send an auto notification to our on call staff. So yes, we have been working on this . But I dont mind if you send me a tweet. @thepixelgeek
4 Likes
Thanks for the offer PixelGeek -
However, we need a contact help desk that is staffed full time that can respond as OUTAGE issues arise. Please understand that this would be a minimal expectation from your customer if your product is to be relied upon by companies of any size.
I am struggling to keep a sizable company on your service. Today hurt a great deal.
Outages happen, but, my inability to convey to my customer updates on outage remedies is inexcusable … I find that I am in a position of making excuses for WEBFLOW’s lack of rudimentary support processes …
Please accept this as a sincere effort to get WEBFLOW to realize what is happening from just one customer’s
perspective. I really sold WEBFLOW and now …
Hi everyone,
Please accept our sincere apologies for this unexpected outage, and the early lack of transparency as we began to investigate this issue.
What Happened
A distributed denial-of-service attack was started against one of our CMS Hosting customers at 10am PST, and overwhelmed several servers responsible for rendering dynamic content. Some sites that were rendered by that server cluster, but were not yet cached by our CDN, were seeing an error page render instead of the actual site. This lasted for approximately 2 hours.
Impact
This affected some sites that had pages with dynamic CMS-powered content, and was limited to sites and URLs that were either recently published or had not been visited recently. The vast majority of sites were not affected, as that traffic was served via cached responses from our content delivery network.
Current Status
We’ve updated our servers to slow down traffic to the specific site being attacked, and have added more servers to handle the higher load. All sites should be working properly again, with no action needed on your part.
If your website is still experiencing this issue, please contact our support team at support@webflow.com
Next Steps
- We are analyzing all layers of our CMS Hosting infrastructure to ensure that DDoS spikes of this nature are quickly detected, and do not negatively affect the delivery of other domains across our servers.
- We are working on extending our Webflow Status site to include multiple health checks on published sites of various types. Currently, the status page only reflects the health of the Webflow Designer and Dashboard - which clearly doesn’t cut it for situations like this when hosted sites are affected.
- We’re reviewing our customer support and notification procedures to come up with a faster and more transparent way to deal with urgent issues of this nature. Like @rayjnorris said, outages do happen, but we need to do much better to keep all of you informed in real time about what’s happening.
Thank you all immensely for notifying us and alerting us of the errors which were coming through, and for your patience and understanding while we worked to resolve the issue.
Sincerely,
Vlad
11 Likes
Even with the price hike on hosting you cannot expect 100% uptime. End-users (clients of us designer and developers) need to know this and it is our job to notify our clients about this before signing them on the Webflow hosting. Especially something like DDoS is almost impossible to defend against unless you spend a extremely huge amount of money on DDoS solutions. As long as I know Webflow is doing their best, for example keeping us updated on their status page, then I hold no ill feelings towards Webflow.
1 Like
I would like to know more about the DDoS strategies and systems WEBFLOW uses now (or will in the future) to pass along to my clients.
Most of the angst written above, and most certainly by me, is about Transparency and communication during an extreme interruption to our clients whom we sell Webflow services to.
I believe Vlad has taken note and will be addressing … Would love to know the DDoS prevention system in place now.
I hope so too, Webflow is really good at adapting and updating themselves. So I’m sure it’s coming soon
Hi Vlad -
I have some questions that I need to share the answers to with my clients who are paying for your CMS Service.
-
What DDoS mitigation systems or services are you currently using to ensure availability of services? Please discuss ‘non-cached CDN’ stage in particular.
-
What steps is Webflow taking, moving forward, with regard to #1 (and when will that go into place?)
-
What changes can we expect Webflow to make to accurately tie service interruption incidents to status.webflow.reporting. To many of us, the outage experienced would most certainly be considered an ‘incident’ (to me, a serious incident) and it is still not reported as such. Very mis-leading to be sure …
-
What steps will Webflow be taking to ensure there is someone that we can talk to in the future during future incidents. I am not working with any other companies who do not offer a live help desk.
Hi everyone -
Wanted to provide a post mortem of this downtime, and explain why our status page did not reflect this downtime. on 2/11 7pm CET a certain breed of DoS attack hit our infrastructure, and took down our render cluster’s ability to render CMS content. A certain fleet of servers were not being tracked on status.webflow.com, which is why that site was not updated during this outage.
This is explained in the post mortem, and we’ve taken steps to ensure that this won’t happen again, including beefing up our infrastructure monitoring and auto-posting to status.webflow.com when there are production issues.
Thanks @brryant -
Can you shed any light on who we can call during a serious service interruption? Many folks tried contacting Webflow thru the forum as well as emails to support@… - I sent numerous, but it took 50 minutes to receive a reply. During that 50 minutes, we all hoped that the system was being attended to, but did not know. I was at a loss as to what to tell my customer (your customer as well) during that time. Very strange not having anyone to communicate with …
I have two customers with mission critical systems using Webflow, one of which I am currently removing from Webflow as a result. I hate it … Webflow offers great promise once the ‘transparency’ challenge is resolved …
Hey @rayjnorris,
I’m really a kid here, but I don’t really understand how a company who can really not afford to have their site down for 2 hours is not hosting on their own server with a team of dedicated engineers being on alert 24/7 if anything happens ?
And ok, maybe Webflow should have posted something on the forum to say “hey guys, we’re working on it”, but apart from that, do you prefer people figuring out what the problem is or people wasting their time on the phone telling you they are working on it ?
1 Like
This is how I handle it…
I have 4 mission critical clients on a dedicated servers ($350 / mth).
Of them…
I have 3 clients were one dedicated and a 4th on their own dedicated.
So that’s 2 dedicated servers ($700/mth).
All 4
also have alternate server setup ($150) as a backup - just in case.
If and when something happens…
a simple dns switch at the registrar fixes the problem.
That’s ok. Agreed. My bad completely. I understood from Webflow’s marketing materials that they did have availability of service processes and systems in place. This issue is not a cost issue. I now know that Webflow is not a good place to host websites for companies that provide high availability to their customers.
To answer your question: “do you prefer people figuring out what the problem is or people wasting their time on the phone telling you they are working on it ?”
I require both. It’s not a tall order. I believe Vlad agrees.
Same, I use Rackspace as primary and lighter weight Mediatemple as backup. Thought I would give Webflow a try. If rackspace goes down (which they do), I am alerted via Pingdom , I switch DNS point-to’s to mediatemple, contact rackspace. They are always there and helpful.
Et all - not trying to beat up on Webflow. I think they have a great product in their Designer.