Changes with the "Download Item Content" 's behaviours

  • 1
  • Problem
  • Updated 7 months ago
  • (Edited)
Hi,

I am not sure when exactly, but between now and September the behaviour of the "Download Item Content" has changed, making our IOS and Java apps failing when using the sharefile API and trying to download a file.

We were just about rolling-out those apps, but the issue is now jeopardizing the all project since the development phase has been closed and the developer is not available.

For almost a year in order to download files we have been using successfully the following API call : https://account.sf-api.com/sf/v3/Items(id)/Download?includeallversions=false&includeDeleted=false@redirect=false
If successful (HTTP answer 200) this call automatically generate the following call to the storage zone in order to download the file : https://storage-eu-XXX.sharefile.com/download.ashx?dt=XXX&h=XXX
Since recently the storage zone call fails, returning a 403 HTTP response.

If we change "redirect=false" for "redirect=true" it works again.
Unfortunately if I can change the java implementation, I can not for the IOS app, and to be honest reading the description of the "redirect" attribute I do not understand why it fails with a 403 HTTP response now.

I read all the release notes since September and I cannot find any notification about this change in behaviour of the "Download Item Content" API call.
Is it a bug or a new expected behaviour ? Can you explain why such change after almost a year of a different behaviour ?

Current project and purchase of extra sharefile licenses are on-hold since this bug.

Thanks in advance for an answer  !

Kind regards,
Guillaume
Photo of GuillaumeK

GuillaumeK

  • 4 Posts
  • 1 Reply Like
  • Annoyed

Posted 7 months ago

  • 1
Photo of Suprit

Suprit, Employee

  • 10 Posts
  • 2 Reply Likes
Hello Guillaume,

If you can provide sample of your java code implementation and downloadTokens for the failed request calls, we can provide better assistance in fixing the issue.

You can upload the sample code and downloadToken details at https://citrix.sharefile.com/r-r83119e9ba0c4f849

Thank you,
Suprit
Photo of GuillaumeK

GuillaumeK

  • 4 Posts
  • 1 Reply Like
Hi Suprit,

First thank you for the quick answer,even if it was just to ask for more information, it allowed me to keep my brain and attention "hot" on the topic.
By asking me the downloadToken it drove me toward finding the problem. I consider it as being a bug from Citrix's side which may have put in the light one on our side :)
It is not that I didn't look at the downloadToken before, but it is that I did not look at it carefully enough. Till now the URLs given in the download token and the one used to call the storage zone by our apps looked to be the same, but in fact they were not. I have been able to find a really small difference. All characters % from the downloadToken became %25 when we call the storage zone, which is not a surprising behaviour on IOS.
So if we call the storage zone with a different URL parameter than the one we received in the download token why is it a bug from your side ?
The problem is that sharefile is currently using in the downloadToken a character qualified as unsafe in URLs by the rfc1837 and other (https://www.ietf.org/rfc/rfc1738.txt) : 
The character "%" is unsafe because it is used for
   encodings of other characters.
Yes currently all the URLs generated in the DownloadToken finished with  "%3d" (part of the parameter "h"), but also sometimes the character % is used in the middle of the parameter "h" value.
Is it really not recommended to include unsafe and reserved characters in URL parameters values.
I guess you already avoid the other characters qualified as unsafe by the rfc 1738 in your generated parameter values, as the ones qualified as reserved, so why suddenly using the % or %3d (or even "=" corresponding to %3d from https://www.w3schools.com/tags/ref_urlencode.asp) when few weeks ago and for a while you were not ?
Without sharefile starting including unsafe characters in its URL parameters, our IOS apps using sharefile API will still be working.

Can you please let us know when you plan to fix this ?

Kind regards,
Guillaume
(Edited)
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 695 Posts
  • 98 Reply Likes
Guillaume,

Thanks for the thorough analysis on this issue. I do not believe this is a Citrix bug.

The storage center url returned by the Api is already percent encoded, which is a valid use case for including the "%" character. We do not include "%" characters that are meant to represent the actual percent sign; those "%" are actually percent encoding other characters. For instance, we often include "%2f" in the url, which encodes the "/" character.

Based on your description of the issue, your application is likely re-encoding an already encoded url, which then results in loss of the original value, and thus gets the error you're encountering. The storage center will accept either the encoded or non-encoded url, but it will not handle a double-encoded value.I do apologize if it seems this behavior is new, but we have been encoding our download urls for at least a year now, and we cannot return unencoded urls in order to continue abiding by best practices.

Do let me know your thoughts and what we can do to help.

Thank you!
Eli
Photo of GuillaumeK

GuillaumeK

  • 4 Posts
  • 1 Reply Like
Hi Eli,

I agree and recognized in my previous message that there was a bug in our IOS implementation. Some developers do not understanding completely encoding and it is a bigger reality than we think.
The REST answer we get providing us the storage zone URL is according to specifications. You have encoded some specific characters like they should be as part of the response. I keep in mind that the token given is then not reused in a REST call to access the file on the storage zone but in a simple URL call with parameters. I don't know if our developer got confused or just did not realise/understand what he was doing.

This behaviours from having specific encoded characters from the sharefile API response in the h parameter value is recent or at least have increased dramatically recently. We have been testing our app using the API since March last year and such behaviour was not seen until recently. Sometime it failed to download a file (we thought because of network issues and did not really check the logs) but it was really rare, and after trying a second time it always worked. Otherwise we will have spotted it earlier. Now it fails at every tries. So something as changed.

In fact my point in my previous message was not to qualify as a bug the encoding you have done sending back the URL but more the fact you use specific characters in what looks to be a token (the parameter h).
If some rfc and especially the rfc 1738 qualified certain characters as "unsafe" like % or "reserved" like / , you must have a really good reason to use them in URL parameter value. I make a significant difference between using those character within an URL path and an URL parameter value, especially when it is about creating a token to be exchanged between different systems.
I may miss some logic in your API implementation but from my point of view using such characters in a token his not a good practice and looking for troubles with your API consumers. That is the reason why I call it a bug, in case there is no real reason to use such characters, but you may have some, and if you do, their usage have significantly increased since September. You can not deny that.
But to be honest the only reason I can imagine to have such characters in a so long token is to add some logic in the token itself, and I see that as bad design.

I have found in some threads in this forum and others, people mentioning that downloads were failing sometimes but always. If you include such character encoding randomly that could explain why.  Just an idea...

The philosophy of rest APIs and services is to make it simpler for the API consumers. Using unsafe or even restricted characters in URL parameters (especially in a token) when it is not totally a necessity, is not helping the API consumers and is always creating issues.

Kind regards
Photo of Eliezer Encarnacion

Eliezer Encarnacion, Official Rep

  • 695 Posts
  • 98 Reply Likes
Guillaume,

I can definitely look into whether something changed in the "h" parameter value that would lead to more encoded parameters appearing, but the encoding itself is not new. I definitely understand your concerns about ambiguous characters in an important value that is mean to transport information between systems, and we will discuss this matter among the development team to consider making a change to how that url param is encoded. 

Thanks!
Eli
Photo of GuillaumeK

GuillaumeK

  • 4 Posts
  • 1 Reply Like
Hi Edi,

I am glad to read your acknowledgement.
In fact it is not just more often in the "h" parameter value, it is now every time. Such new behaviour can not be only bad luck, but the result of a voluntary change.
Thanks a lot for investigating it.

Kind regards,
Guillaume
(Edited)