Sharefile account webhook isn't firing sometimes

  • 2
  • Problem
  • Updated 6 days ago
Hello,

A bit of a backstory with the webhook:
We're using an account webhook (we didn't use the folder one since it doesn't trigger for subfolders) to keep files in sync with a local zip that we keep.
The hook worked perfectly until i'd say 2 weeks ago when it sometimes would not trigger at all, or we'd get the events with either a few minutes delay, or the next day.
We know that some didn't trigger since we keep both a db log and a file log of all hooks (we log the event before we even begin to process it).

Is there a place to see all queued/failed events? And if not, can we provide some account/hook/folder ids to be checked?

Thanks,
Andrei
Photo of Sirbu Andrei

Sirbu Andrei

  • 3 Posts
  • 1 Reply Like
  • worried

Posted 1 year ago

  • 2
Photo of Dale Smith

Dale Smith, Software Engineer

  • 187 Posts
  • 26 Reply Likes
Hi Sirbu,

We have had a few other customers report the same issue and have created a ticket for this issue to be looked into. While we do have data points from the other customers, we would welcome your examples as well, as the more data we have, the easier it will be to find the issue. Feel free to upload your examples in a text file to https://citrix.sharefile.com/r-r07cbe7cc3cc4521a

It's best to include the following if you have it:
Webhook Subscription Id (this is the id generated when you created the webhook)
Your OAuth Client Id
The Url the webhook is going to
The specific event details that was late (for instance if it was File Upload event, the file name and folder id the event happened in)
A time frame that this was happening.

In particular if you have examples of the hooks that did not trigger at all, as that is new in comparison to the reports we have had.

Thanks, and apologies for the inconvenience.
Dale Smith
Photo of Sirbu Andrei

Sirbu Andrei

  • 3 Posts
  • 1 Reply Like
Thank you for your reply Dale.
I've just uploaded the txt with the details.

Photo of Dale Smith

Dale Smith, Software Engineer

  • 187 Posts
  • 26 Reply Likes
Thanks Sirbu,
 
I've received the file and attached it to the ticket.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 26 Posts
  • 0 Reply Likes
Any update on this one? I am experiencing a similar issues.
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
The issue should be addressed early next week.
Apologies again for the inconvenience.

Robert
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 26 Posts
  • 0 Reply Likes
@Robert - was this addressed already?
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
A fix for the issue is pending a release. The release is planned for this week. I will let know here once it is deployed.

I'm sorry for the delay.

Robert

Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
Sirbu, Lukasz

The fix has been deployed. Let us know if you still experience any issues.

Thanks,
Robert
Photo of Sirbu Andrei

Sirbu Andrei

  • 3 Posts
  • 1 Reply Like
Thanks.
Wii do.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 26 Posts
  • 0 Reply Likes
Reopening that issue - i am experiecing exactly the same problem for the past week. Are there some problems again on sharefile end? I am not receiving all of those webhook notifications.
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
Thanks for bringing it up.
We have identified an intermittent issue that occurred end of last week which should be resolved now.  Sorry for that.

Are you still experiencing the problem?
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
Robert - this is still happening - I just checked it and I am not receiving webhook events when i drop a file.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
Any update on this Robert? I tried openning a supprt ticket, but they closed it within acouple of hours with no resolution... I don't know what I should do at this point and it's affecting our live systems.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
@Robert - following up on this once again.
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
@Łukasz, could you please provide the support ticket number?
I will follow up with our support channel. Unfortunately I am not able to help more at the moment. 
And sorry again for the inconvenience.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
@Robert - case number is 78819328 - it was closed within a couple hours with no resolution on your end.
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
Thanks. I have requested internally to have the case revisited.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 14 Posts
  • 0 Reply Likes
Hey Robert - it was not reopened. Could you please look into this?
Photo of Robert Jakubowski

Robert Jakubowski, Software Engineer

  • 25 Posts
  • 4 Reply Likes
Hi Łukasz,

I have raised your issue several times but my request is still pending.
I will add a note that you just asked.
My apologies again.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 26 Posts
  • 0 Reply Likes
I will be monitoring this this week and let you know, however i still believe i was not receiving all of the events Today. I will provide an update tmrw.
Photo of Matthew

Matthew, Product Manager

  • 7 Posts
  • 0 Reply Likes
Hi Łukasz,

We are unable to reopen your prior support case, however, we will be investigating the root cause of your webhooks issue. I will provide an update when we have identified the cause and when a fix is in place.
(Edited)
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 26 Posts
  • 0 Reply Likes
Matthew,

Any update on this?
Photo of Anuja Gogate

Anuja Gogate, Employee

  • 6 Posts
  • 1 Reply Like
Hey Lukasz,

Could you provide us with the following information: 

Webhook Subscription Id (this is the id generated when you created the webhook)
Your OAuth Client Id
The Url the webhook is going to
The specific event details that was late (for instance if it was File Upload event, the file name and folder id the event happened in)
A time frame that this was happening.

You can upload these details in a text file via: https://citrix.sharefile.com/r-r4089026e08b4b199
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 26 Posts
  • 0 Reply Likes
It looks like it's working now, even tough I did not change anything. Will monitor it again.
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 26 Posts
  • 0 Reply Likes
@Anuja - it was working correctly up until i activated a second webhook. That caused the same issue to reappear, which means that im only getting some of the webhook events back to my server. I added a logging to it, so i know that the requests do not even hit my servers.
Photo of Cláudio Batista

Cláudio Batista

  • 7 Posts
  • 2 Reply Likes
Hello there,

Any reports about webhooks issues since last Thursday? It was working fine until then, but now even what we call the Webhooks subscription endpoint (https://companyname.sf-api.com/sf/v3/WebhookSubscriptions) we got the follow response:

{
    "code": "NotFound",
    "message": {
        "lang": "en-US",
        "value": ""
    },
    "reason": "NotFound"
}

Same response when we do POST, when trying to recreate subscriptions. 


Thanks,
Photo of Anuja Gogate

Anuja Gogate, Employee

  • 6 Posts
  • 1 Reply Like
Claudio and I have exchanged communication on another thread and this issue is resolved.
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Any updates here? We are experiencing this issue since Thursday 4/18 such that we are no longer receiving a webhook for every upload on an account-wide webhook subscription. This is very problematic because action based on these uploads is SLA bound.

Thanks,

Adam
Photo of Matthew

Matthew, Product Manager

  • 7 Posts
  • 0 Reply Likes

Thank you for sharing issues you are experiencing with our service. We are unable to record customer issues and communicate the progress of fixes on this forum. Please contact support to open a case here: https://www.sharefile.com/support.

Support will need to securely capture your account information to troubleshoot the issue.

Photo of Anuja Gogate

Anuja Gogate, Employee

  • 6 Posts
  • 1 Reply Like
Adam, 

Like Matt mentioned, if possible please do open a support case. In the meantime,  could you provide us with the following information: 

Webhook Subscription Id (this is the id generated when you created the webhook)
Your OAuth Client Id
The Url the webhook is going to
The specific event details that was late (for instance if it was File Upload event, the file name and folder id the event happened in)
A time frame that this was happening.

You can upload these details in a text file via: https://citrix.sharefile.com/r-r4089026e08b4b199
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Matthew, Anuja -

I have already submitted a support case (78882083). Support was unable to assist, stating they "do not support" anything related to API nor webhooks. They directed me to post here. I would strongly suggest Citrix figure out how to support this product going forward. Terrible customer experience.

Anuja - Submitted information as zip. Events are not simply late: we are not receiving events at all for a significant number of file uploads.

Thanks,

Adam
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 23 Posts
  • 0 Reply Likes
Adam,

I had exactly the same issue like you, but it magically started working yesterday for every upload. Before that it would work for exactly every second upload for an account wide subscription.
Also like Adam noted I had the same issue with support, went with them over email back and forth for ~4 weeks just to have an answer that they do not support this. It was really a terrible experience.
Photo of Sagar Thirumala

Sagar Thirumala, Software Engineer

  • 6 Posts
  • 0 Reply Likes
Adam,

Thanks for submitting the details. Based on the investigation, we do see few calls to the URL you have provided succeeding. But for the failure cases, the URL doesn't seem to be responding back with in 10 seconds.

As per the documentation

The ShareFile Platform expects a 2XX response back in 10 seconds to indicate a success. If we do not receive a successful response within that timeframe, we will consider it a failure.
In general, consumers are recommended to accept callbacks quickly and process them asynchronously, hope this information helps for further troubleshooting on your end. Let us know if you need any more information.
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Sagar, Anuja,

The issue is not on the receiving end, as nothing has changed there, and a response is provided immediately. The implementation conforms to your recommendation.

To be sure, we cleaned up our Webhook Subscriptions list and re-subscribed to the route in question using another URL that would hit the same destination on Friday 4/26 @ ~5:30 PM Pacific Time. That same URL was verified independently as working via Postman tests. Since Friday we are not receiving any inbound webhooks. This happened a few months ago in the same fashion, and the issue was squarely on your end.

What is going on here? We need a fix fast.

Thank you,

Adam
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Update? Still no webhooks being received.
Photo of Sagar Thirumala

Sagar Thirumala, Software Engineer

  • 6 Posts
  • 0 Reply Likes
Adam,

My apologies for the delay in response, we should have the fix available since last couple days, the fix will ensure that we retry calling the registered url with exponential back-off strategy for timeout scenarios.

Let us know if you notice any further issue with not receiving webhook calls.
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Sagar,

We have uncovered more Webhook misses from the past few days, with a significant number yesterday evening. I just uploaded examples to the ShareFile upload drop previously provided + details requested about the webhook subscription.

What's going on there? This is how the issues began previously.

Thank you,

Adam
Photo of Sagar Thirumala

Sagar Thirumala, Software Engineer

  • 6 Posts
  • 0 Reply Likes
Adam,

My apologies for the delay in response, could you confirm if you guys saw any further upload drops since 11th May?
Photo of PK

PK

  • 1 Post
  • 0 Reply Likes
Howdy,

I've been developing a webhook integration with the ShareFile API over the past few weeks with pretty good success.  I recently cleared the SF subscriptions for a baseline test and since 8/21/2019 no longer receive any notifications.  I am able to enumerate the webhook subscriptions from the ShareFile API so I know they are entred in the SF system.  Our public endpoint is still up and responding to basic Postman POSTs, just no activity from SF. 

I created a ticket this evening but have yet to receive a confirmation email. 
Can someone take a look to see if there is a system issue?

Thanks!
PK
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Hello,

We are AGAIN experiencing this, apparently coinciding with a change made and communicated yesterday regarding authentication. No webhooks received since 11:33 AM Pacific Time on Monday, November 4, 2019.

This is affecting our ability to respond within our SLA to our customers.

Please advise ASAP.

Thank you,

Adam
Photo of Leo

Leo, Official Rep

  • 437 Posts
  • 43 Reply Likes
Hi Adam,

Please upload a text file with the below information to here: https://citrix.sharefile.com/r-rf31e214c8114961a
  • Webhook Subscription Id (i.e. the id generated when you created the webhook)
  • Your OAuth Client Id
  • The Url the webhook is going to
  • The specific event details that was missing (for instance if it was File Upload event, the file name and folder id the event happened in)
  • A time frame that this was happening. 
I'll get the content over to our developers.

-Leo
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Leo,

Looks like the webhook is working again today 11/5 as of ~1:44 PM Pacific Time. It also sent in previously missed notifications. Would you please provide us with root cause analysis to understand what went wrong?

This has happened before (see this thread for context), and it is highly disruptive to our business.

Thank you,

Adam
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Hello,

We are again experiencing this issue today. Last webhook received was 2:32 AM Pacific Time. What is going on?

This is affecting our ability to respond within our SLA to our customers.

Please advise ASAP.

Thank you,

Adam
Photo of Łukasz Trzewik

Łukasz Trzewik

  • 26 Posts
  • 0 Reply Likes
Same here around the same time.
Photo of Daniel Macias

Daniel Macias

  • 9 Posts
  • 0 Reply Likes
Same, please advice, we need this working ASAP.
Photo of Austin Nafziger

Austin Nafziger

  • 4 Posts
  • 0 Reply Likes
All of our webhooks stopped working on 11/28 and have not recovered. This is a critical issue for an enterprise application and has created significant disruptions to our internal process and client services.

Please provide an update on the solution.
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
Same here since 11/27, we are again out of order with our customer facing SLA in jeopardy, and my team yet again having to waste time assisting our operations teams with reconciliation.
Photo of smart web

smart web

  • 4 Posts
  • 1 Reply Like
is there any alternative way instead of this not working webhook? We are also in trouble as the system is not working and it's out of our control.
Photo of Adam Cole

Adam Cole

  • 14 Posts
  • 1 Reply Like
No webhooks since 11:06 AM Pacific Time today. We are AGAIN broken and dead in the water, placing our customer facing SLAs in jeopardy and wasting our internal resources during the busiest time of year.

Please provide an update ASAP. This is absurd and unacceptable.