Webhooks fail to execute on configured operation for resource

  • 1
  • Problem
  • Updated 2 months ago
We have had several instances on 8/8 (first observed at 11 AM CST and last observed at 3:30 PM CST) where a file upload on a folder with a File Upload webhook failed to post to the configured webhook URL.

When we deleted the webhook and allowed the same automated process to recreate it, and uploaded an additional file, the webhook successfully posted to the configured webhook URL.

We have several hundred webhooks and our concerned about the reliability of them executing as expected.
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes

Posted 2 months ago

  • 1
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 732 Posts
  • 102 Reply Likes
Hi Michael,

We'll take a closer look at what happened here. What is the folder id where the webhook failed to post?

Thanks,
Eli
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes
Is there a way for me to message you privately with those details? 
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 732 Posts
  • 102 Reply Likes
You can upload a text file with that info to this secure link: https://eliezeren.sharefile.com/r-raadbefc07144ba28.

Could you also include your account subdomain, and the API client Id you are using?

Thanks!
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes
Done.
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 732 Posts
  • 102 Reply Likes
Thanks, we'll look into it and get back to you as soon we find out what happened

Eli
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 732 Posts
  • 102 Reply Likes
Hi Michael,

Here's what we could find. It is not clear that there was an error delivering any of the Webhooks you've subscribed to, so we are planning to add extra monitoring to catch anything that could've happened, as well as any possible delays in the sending of events. We apologize for the issues you saw, we'll use this opportunity to further improve the reliability of our service.

Best,
Eli
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes
Thanks. It looks like we have another failed webhook on a different folder id this AM that wasn't in the set I provided to you earlier. Could I upload the list of all the folder Ids we have webhooks for inclusion in the extra monitoring? 
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 732 Posts
  • 102 Reply Likes
Sure
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes
Done.
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes
Done.
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes
We have a few more upload events that failed to trigger the webhook. I'll upload the resource ids for the folders to you now. Hopefully, some of the additional monitoring you have in place detected some anomalous behavior.
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 732 Posts
  • 102 Reply Likes
Thanks Michael, I'll forward this to the team and see what we can find.
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes
We've seen about a 50% failure rate in the last week for our webhooks. Was your support team able to identify any issues in the logs?
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 732 Posts
  • 102 Reply Likes
I'm sorry to hear this Michael. I haven't heard back from the previous errors but will bump this up in priority to see what we have found.

So we have more information, how is your system determining that a webhook is not received? 
Photo of Michael Ericksen

Michael Ericksen

  • 12 Posts
  • 2 Reply Likes
Sure. We have two sets of notifications configured: an email notification configured through the portal (legacy workflow); and webhook notifications created via the REST API (new workflow).

We're testing the production readiness of the new workflow and getting feedback from users that they're receiving email notifications that don't have any corresponding entry via the new workflow.
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 732 Posts
  • 102 Reply Likes
Thanks Michael. Wanted to let you know we've started looking at this and should get back to you no later than Tuesday next week. We'll let you know if we have any additional questions.