- Managing Global Settings with OptionManager
- Collecting Logs Using DiagnosticManager
- Sharing Sessions Among Different Applications
- Using Single Sign-On from the VI SDK to CIM
- Downloading and Uploading Files Using HTTP Access
- Multithreading with the VI SDK
- Versioning
- Following Best Practices for Performance and Scalability
- Considering Internationalization
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Downloading and Uploading Files Using HTTP Access
In VI 3.5, VMware introduced a new feature to enable applications to download a file from and upload a file to the datastores of ESX servers.
The syntax of the URLs is as follows:
http(s)://<hostname>/folder[/<path>]?dcPath=<datacenter_path>[&dsName=<datastore _name>]
Following are some sample URLs:
https://18.17.218.228/folder?dcPath=Datacenter
which lists all the datastores in the datacenter whose path is Datacenter.
https://18.17.218.228/folder?dcPath=Datacenter&dsName=storage1%20(1)
which lists all the folders in the datastore named storage1 (1) whose path is Datacenter.
https://18.17.218.228/folder/SuSe_server10/SuSe_server10- flat.vmdk?dcPath=Datacenter&dsName=storage1%20(1)
which points to the vmdk file called SuSe_server10-flat.vmdk.
VMware has shipped a sample code with VI SDK 2.5 showing how to upload a virtual machine to an ESX server (com.vmware.samples.httpfileaccess.ColdMigration). The sample code works by uploading files whose sizes are 40MB or smaller. In most of the cases, virtual machine disk files are much bigger than 40MB; therefore, an OutOfMemoryError is almost always thrown for such an execution.
Given the average virtual disk size, the ColdMigration sample is almost useless if it cannot work with files bigger than 40MB.
An easy solution seems to be increasing the Java heap size using a parameter to the Java virtual machine. But that is not a good solution given the size of the virtual machines, sometimes 100GB or bigger. It would be hard to find such a large memory and assign it to a Java virtual machine. So let's pin down the root cause of the problem and find an alternative solution.
After debugging the sample, the following section of code is found to throw exceptions.
OutputStream out = conn.getOutputStream();
FileInputStream in = new FileInputStream(
new File(localFilePath));
byte[] buf = new byte[1024];
int len = 0;
while ((len = in.read(buf)) > 0) {
out.write(buf, 0, len);
}
conn.getResponseMessage();
conn.disconnect();
out.close();
Simply reading the code, it seems okay. To figure out in which iteration the problem happens, a conditional breakpoint is set to catch OutOfMemoryError.
The next run reveals the stack shown in Figure 18-4 when the exception happened.
)
Figure 18-4 Partial calling stack when OutOfMemoryError happens
The error was thrown at the following highlighted line. The argument newLength is 67,108,864. No wonder we have a problem with this huge array.
public static byte[] copyOf(byte[] original, int newLength)
{
byte[] copy = new byte[newLength];
System.arraycopy(original, 0, copy, 0,
Math.min(original.length, newLength));
return copy;
}
Further reading discloses that PostOutputStream(ByteArrayOutputStream).write() tries to buffer all the data to be sent. It's not going to work with big size uploading.
The question becomes whether it can use a different output class. The PosterOutputStream is returned from the getOutputStream() method:
public synchronized OutputStream getOutputStream()
throws IOException
{
return delegate.getOutputStream();
}
Because the source code of Sun's HttpURLConnection class is not available in the src.zip, a search on the Web finds code samples like the following. It shows that the getOutputStream() method can actually return subtypes of OutputStream other than PosterOutputStream depending on the variable fixedContentLength and chunkLength.
public synchronized OutputStream getOutputStream()
{
...
if(streaming())
{
if(fixedContentLength != -1)
strOutputStream = new StreamingOutputStream(
ps, fixedContentLength);
else if(chunkLength != -1)
strOutputStream = new StreamingOutputStream(
new ChunkedOutputStream(ps, chunkLength), -1);
return strOutputStream;
}
...
if(poster == null)
poster = new PosterOutputStream();
return poster;
}
By exploring the HttpURLConnection class, you can find the methods to set the two variables: setFixedLengthStreamingMode(int) and setChunkedStreamingMode(int), respectively.
Because the file size is known beforehand, just use the first method and get StreamingOutputStream. It works fine by inserting the following line before conn.getOutputStream():
conn.setFixedLengthStreamingMode(fileSize);
The issue with the sample code is not a bug of VMware Infrastructure per se. The root cause of the issue is the misuse of Java APIs.
You could argue that PosterOutputStream should not be the default OutputStream. In practice, uploading a huge file is not a typical use case of the HttpConnection. Most API users use it to download content of any size and upload a relatively small file. After all, the HttpURLConnection class has provided an alternative to solve the problem.
Note that the argument to setFixedLengthStreamingMode is an integer, meaning it can only hold up to about 2.14GB, which is not enough for a disk file. Uploading bigger files requires using setChunkedStreamingMode(int). As of SDK 2.5, the chunked streaming is not supported on the ESX server side.