Webhook file upload notifications are 2 days late.

  • 2
  • Problem
  • Updated 2 months ago
We are not getting webhook notifications on new file uploads for folders that we have webhook subscriptions on.  It looks like this problem started on teh 4th of this month.  I am seeing a 2 day delay in the notifications.  Is anyone else seeing this issue?
Photo of James

James

  • 3 Posts
  • 0 Reply Likes

Posted 2 months ago

  • 2
Photo of Cláudio Batista

Cláudio Batista

  • 1 Post
  • 0 Reply Likes
We are currently experiencing similar issues. Sometimes the webhooks are not working at all. Funny enough, sometimes for on specific folder the webhook works and for other folder doesn't work  - sometimes the inverse. 


Photo of Dale Smith

Dale Smith, Employee

  • 137 Posts
  • 19 Reply Likes
Hi there James/Claudio,

Can you please give us some more information about the specific subscription that you're having issues with. For example the following would be of great help

Webhook Subscription Id (this is the id generated when you created the webhook)
Your OAuth Client Id
The Url the webhook is going to
The specific event details that was late (for instance if it was File Upload event, the file name and folder id the event happened in)
A time frame that this was happening.

You can upload these details in a text file via: https://citrix.sharefile.com/r-r93c380b257c4fe5a

We're currently unaware of any issues with the webhooks infrastructure and haven't made any changes to it recently(though there were some outages within the ShareFile infrastructure in the past week or two that could have impacted it). There are also some gotcha's with the webhooks that may also be at play here. Either way, some more detailed information will help us narrow this down for you guys.

Thanks,
Dale Smith

Photo of James

James

  • 3 Posts
  • 0 Reply Likes
Thanks Dale for looking into this.  I have uploaded the information that you requested.  Let me know if you need further details.
Photo of Dale Smith

Dale Smith, Employee

  • 137 Posts
  • 19 Reply Likes
Hey James,

Thanks for sending that in. Just wanted to give you an update. So I do see some uncaught exceptions in our logs for sending webhooks to your url for those dates, however, I'm unable to correlate them specifically to the examples you gave me. It does look like one particular instance was responsible, so its distinctly possible that that instance gets an event into its memory, attempts to send it, fails, and keeps retrying (we use a back off strategy here). Eventually the instance is either recycled and the message gets put back on the queue for another instance to handle, or that instance corrects itself and on the next retry is able to successfully send the message. Meanwhile, other instances are handling your webhook events as well and so most events are going through fine. 

I'm going to dig into the stack traces for these exceptions and see what I can find (and hopefully they will be the root cause of your delayed messages). I'll update this thread if I find anything. 

Thanks,
Dale Smith
Photo of James

James

  • 3 Posts
  • 0 Reply Likes
Thanks for the update Dale.  I appreciate you taking the time to dig into the issue.  I had some more reported issues on Friday for 1 specific webhook as well.  I asked my client for an example of the issue.  If they do provide me more example I will pass them on to you. 

You are correct that my other webhooks are working just fine.  It just seems to be random for specific subscriptions.

Thanks again!

James