Sharefile account webhook isn't firing sometimes

  • 1
  • Problem
  • Updated 2 months ago
Hello,

A bit of a backstory with the webhook:
We're using an account webhook (we didn't use the folder one since it doesn't trigger for subfolders) to keep files in sync with a local zip that we keep.
The hook worked perfectly until i'd say 2 weeks ago when it sometimes would not trigger at all, or we'd get the events with either a few minutes delay, or the next day.
We know that some didn't trigger since we keep both a db log and a file log of all hooks (we log the event before we even begin to process it).

Is there a place to see all queued/failed events? And if not, can we provide some account/hook/folder ids to be checked?

Thanks,
Andrei
Photo of Sirbu Andrei

Sirbu Andrei

  • 3 Posts
  • 1 Reply Like
  • worried

Posted 9 months ago

  • 1
Photo of Dale Smith

Dale Smith, Software Engineer

  • 180 Posts
  • 23 Reply Likes
Hi Sirbu,

We have had a few other customers report the same issue and have created a ticket for this issue to be looked into. While we do have data points from the other customers, we would welcome your examples as well, as the more data we have, the easier it will be to find the issue. Feel free to upload your examples in a text file to https://citrix.sharefile.com/r-r07cbe7cc3cc4521a

It's best to include the following if you have it:
Webhook Subscription Id (this is the id generated when you created the webhook)
Your OAuth Client Id
The Url the webhook is going to
The specific event details that was late (for instance if it was File Upload event, the file name and folder id the event happened in)
A time frame that this was happening.

In particular if you have examples of the hooks that did not trigger at all, as that is new in comparison to the reports we have had.

Thanks, and apologies for the inconvenience.
Dale Smith
Photo of Sirbu Andrei

Sirbu Andrei

  • 3 Posts
  • 1 Reply Like
Thank you for your reply Dale.
I've just uploaded the txt with the details.

Photo of Dale Smith

Dale Smith, Software Engineer

  • 180 Posts
  • 23 Reply Likes
Thanks Sirbu,
 
I've received the file and attached it to the ticket.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
Any update on this one? I am experiencing a similar issues.
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
The issue should be addressed early next week.
Apologies again for the inconvenience.

Robert
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
@Robert - was this addressed already?
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
A fix for the issue is pending a release. The release is planned for this week. I will let know here once it is deployed.

I'm sorry for the delay.

Robert

Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
Sirbu, Lukasz

The fix has been deployed. Let us know if you still experience any issues.

Thanks,
Robert
Photo of Sirbu Andrei

Sirbu Andrei

  • 3 Posts
  • 1 Reply Like
Thanks.
Wii do.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
Reopening that issue - i am experiecing exactly the same problem for the past week. Are there some problems again on sharefile end? I am not receiving all of those webhook notifications.
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
Thanks for bringing it up.
We have identified an intermittent issue that occurred end of last week which should be resolved now.  Sorry for that.

Are you still experiencing the problem?
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
@Łukasz, could you please provide the support ticket number?
I will follow up with our support channel. Unfortunately I am not able to help more at the moment. 
And sorry again for the inconvenience.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
@Robert - case number is 78819328 - it was closed within a couple hours with no resolution on your end.
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
Thanks. I have requested internally to have the case revisited.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
Hey Robert - it was not reopened. Could you please look into this?
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
Hi Łukasz,

I have raised your issue several times but my request is still pending.
I will add a note that you just asked.
My apologies again.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
I will be monitoring this this week and let you know, however i still believe i was not receiving all of the events Today. I will provide an update tmrw.
Photo of Matthew

Matthew

  • 5 Posts
  • 0 Reply Likes
Hi Łukasz,

We are unable to reopen your prior support case, however, we will be investigating the root cause of your webhooks issue. I will provide an update when we have identified the cause and when a fix is in place.
(Edited)
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
Matthew,

Any update on this?
Photo of Anuja Gogate

Anuja Gogate

  • 6 Posts
  • 1 Reply Like
Hey Lukasz,

Could you provide us with the following information: 

Webhook Subscription Id (this is the id generated when you created the webhook)
Your OAuth Client Id
The Url the webhook is going to
The specific event details that was late (for instance if it was File Upload event, the file name and folder id the event happened in)
A time frame that this was happening.

You can upload these details in a text file via: https://citrix.sharefile.com/r-r4089026e08b4b199
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
It looks like it's working now, even tough I did not change anything. Will monitor it again.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
@Anuja - it was working correctly up until i activated a second webhook. That caused the same issue to reappear, which means that im only getting some of the webhook events back to my server. I added a logging to it, so i know that the requests do not even hit my servers.
Photo of Cláudio Batista

Cláudio Batista

  • 7 Posts
  • 2 Reply Likes
Hello there,

Any reports about webhooks issues since last Thursday? It was working fine until then, but now even what we call the Webhooks subscription endpoint (https://companyname.sf-api.com/sf/v3/WebhookSubscriptions) we got the follow response:

{
    "code": "NotFound",
    "message": {
        "lang": "en-US",
        "value": ""
    },
    "reason": "NotFound"
}

Same response when we do POST, when trying to recreate subscriptions. 


Thanks,
Photo of Anuja Gogate

Anuja Gogate

  • 6 Posts
  • 1 Reply Like
Claudio and I have exchanged communication on another thread and this issue is resolved.
Photo of Adam Cole

Adam Cole

  • 5 Posts
  • 0 Reply Likes
Any updates here? We are experiencing this issue since Thursday 4/18 such that we are no longer receiving a webhook for every upload on an account-wide webhook subscription. This is very problematic because action based on these uploads is SLA bound.

Thanks,

Adam
Photo of Adam Cole

Adam Cole

  • 5 Posts
  • 0 Reply Likes
Sagar, Anuja,

The issue is not on the receiving end, as nothing has changed there, and a response is provided immediately. The implementation conforms to your recommendation.

To be sure, we cleaned up our Webhook Subscriptions list and re-subscribed to the route in question using another URL that would hit the same destination on Friday 4/26 @ ~5:30 PM Pacific Time. That same URL was verified independently as working via Postman tests. Since Friday we are not receiving any inbound webhooks. This happened a few months ago in the same fashion, and the issue was squarely on your end.

What is going on here? We need a fix fast.

Thank you,

Adam
Photo of Adam Cole

Adam Cole

  • 5 Posts
  • 0 Reply Likes
Update? Still no webhooks being received.
Photo of Sagar Thirumala

Sagar Thirumala, Software Engineer

  • 6 Posts
  • 0 Reply Likes
Adam,

My apologies for the delay in response, we should have the fix available since last couple days, the fix will ensure that we retry calling the registered url with exponential back-off strategy for timeout scenarios.

Let us know if you notice any further issue with not receiving webhook calls.
Photo of Adam Cole

Adam Cole

  • 5 Posts
  • 0 Reply Likes
Sagar,

We have uncovered more Webhook misses from the past few days, with a significant number yesterday evening. I just uploaded examples to the ShareFile upload drop previously provided + details requested about the webhook subscription.

What's going on there? This is how the issues began previously.

Thank you,

Adam
Photo of Sagar Thirumala

Sagar Thirumala, Software Engineer

  • 6 Posts
  • 0 Reply Likes
Adam,

My apologies for the delay in response, could you confirm if you guys saw any further upload drops since 11th May?