Status: Open
Status: Answered
Status: Closed
Status: Duplicate

Downloading using CS REST API (V2) using Java HttpClient

0
Posted Mar 05 by Hugh Ferguson.

Hi,
I've been doing what should be a dead simple file download using Apache HttpClient. However, for whatever reason, I hit a limit of about 5 MB for the file transfer before the connection gets reset. Anything up to that size, and this method works just fine. I've tested this on Content Server 16 on Windows Server 2012, and both Oracle 12 and SQL Server 2014 databases. I'm wondering if this is an inherit limit of Apache HttpClient but nowhere on Stack Overflow can I find anyone citing 5 MB as the upper limit (unless it's a function of memory). Here is my code:

public void getDocument(long nodeID, String downloadLocation, String filename) throws Exception  {
    CloseableHttpClient client = HttpClients.createDefault();
    CloseableHttpResponse response = null;
    boolean ok = true;
    String  errMsg = "";
    InputStream is = null;
    OutputStream os = null;
    try {
                    // simply build a url of the form http://servername/OTCS/cs.exe/api/v2/nodes/<nodeID>/content
        String url = getAPIURL(2).concat(String.format("nodes/%d/content", nodeID));
        response = this.getResponseFromGet(client, url);
        //System.out.println("Got response");
        HttpEntity entity = response.getEntity();
        //is = entity.getContent();
        System.out.println("Got stream " + File.separator);
        System.out.println("Writing to " + downloadLocation + File.separator + filename);
        os = new FileOutputStream(new File(downloadLocation + File.separator + filename));
        entity.writeTo(os);
        System.out.println("Wrote to stream");
        byte[] buffer = new byte[FILE_XFER_BUFF_SIZE];
        int bytesRead = 0;
        int passCount = 0;
        int cumBytesRead = 0;
        System.out.println("Begin xfer");
        /*while((bytesRead = is.read(buffer)) > 0) {
            os.write(buffer, 0, bytesRead);
            cumBytesRead += bytesRead;
            passCount++;
            if (passCount % 10 == 0)
                System.out.println(String.format("Pass %d, bytes read %d", passCount, cumBytesRead));
        }*/


        //is.close();
        os.close();
        System.out.println("Download complete");
    } catch (Exception e) {
        ok = false;
        errMsg = e.getMessage();
    } finally {

        if (response != null)
            response.close();
        client.close();

        if (!ok)
            throw new RESTException(errMsg);
    }

}

I've tried reading a fixed number of bytes into a WHILE block as well as just using inputStream.read() to outputStream.write(byte), but regardless of which of these approaches I take, I always hit a limit of about 5 MB. I need to scale this to at least 2 GB. On the upload side, I can easily get much more than that.

Any suggestions?
Thanks
-Hugh Ferguson

3 Answers

0
BEST ANSWER: As chosen by the author.

Hi Hugh,

Silly question, but this may be a setting issue on the server that is hosting Content Server and nothing to do with the httpclient call. IIS or Tomcat (which ever you are using) may have a limit on the file size set somewhere in its configuration.

I know I've had issues with this in the past for memory stream objects in WCF calls to CS and I had to configure IIS to handle files larger that Xmb (I can't recall the default setting). Once I'd done this, we were good to upload and download without issues.

I don't have a server to hand right now to validate this, but perhaps look at the IIS or Tomcat config for your issue.

Hope this helps,

Anthony


0
BEST ANSWER: As chosen by the author.

Hi Anthony, <br/>
I ended up finding the answer to this on the Knowledge Centre. REST calls and webUI calls exhibited the same behaviour, meaning this wasn't REST. I found out that for CS 16, there is a patched bug (LPAD-53849) where the CGI app was crapping out at around 5 MB. Once I applied the appropriate patch, it started to work. The workaround was to use llisapi.dll instead of cs.exe<br>
-Hugh


0
BEST ANSWER: As chosen by the author.

Hi Hugh,

Well that's interesting little 'gotcha' to know. Thanks for updating.

Anthony


 You have subscribed and will receive email notifications of updates to this topic. To unsubscribe, uncheck the checkbox.

Statistics

Related categories

Related tags

Your answer

To leave an answer, please sign in.