[Cross posted from here: http://rob.gillenfamily.net/post/External-File-Upload-Optimizations-for-Windows-Azure.aspx]
I’m wrapping up a bit of the work we’ve been doing on data movement optimizations for cloud computing and the latest set of data yielded some interesting points I thought I’d share. The work done here is not really rocket science but may, in some ways, be slightly counter-intuitive and therefore seemed worthy of posting.
Summary: for those who don’t like to read detailed posts or don’t have time, the synopsis is that if you are uploading data to Azure, block your data (even down to 1MB) and upload in parallel. Set your block size based on your source file size, but if you must choose a fixed value, use 1MB. Following the above will result in significant performance gains… upwards of 10x-24x and a reduction in overall file transfer time of upwards of 90% (eg, uploading a 1GB file averaged 46.37 minutes prior to optimizations and averaged 1.86 minutes afterwards).
Detail: For those of you who want more detail, or think that the claims at the end of the preceding paragraph are over-reaching, what follows is information and code supporting these claims. As the title would indicate, these tests were run from our research facility pointing to the Azure cloud (specifically US North Central as it is physically closest to us) and do not represent intra-cloud results… we have performed intra-cloud tests and the overall results are similar in notion but the data rates are significantly different as well as the tipping points for the various block sizes… this will be detailed separately).
We started by building a very simple console application that would loop through a directory and upload each file to Azure storage. This application used the shipping storage client library from the 1.1 version of the azure tools. The only real variation from the client library is that we added code to collect and record the duration (in ms) and size (in bytes) for each file transferred. The code is available here.
We then created a directory that had a collection of files for the following sizes: 2KB, 32KB, 64KB, 128KB, 512KB, 1MB, 5MB, 10MB, 25MB, 50MB, 100MB, 250MB, 500MB, 750MB, and 1GB (50 files for each size listed). These files contained randomly-generated binary data and do not benefit from compression (a separate discussion topic). Our file generation tool is available here.
The baseline was established by running the application described above against the directory containing all of the data files. This application uploads the files in a random order so as to avoid transferring all of the files of a given size sequentially and thereby spreading the affects of periodic Internet delays across the collection of results. We then ran some scripts to split the resulting data and generate some reports. The raw data collected for our non-optimized tests is available via the links in the Related Resources section at the bottom of this post.
For each file size, we calculated the average upload time (and standard deviation) and the average transfer rate (and standard deviation). As you likely are aware, transferring data across the Internet is susceptible to many transient delays which can cause anomalies in the resulting data. It is for this reason that we randomized the order of source file processing as well as executed the tests 50x for each file size. We expect that these steps will yield a sufficiently balanced set of results.
Once the baseline was collected and analyzed, we updated the test harness application with some methods to split the source file into user-defined block sizes and then to upload those blocks in parallel (using the PutBlock() method of Azure storage). The parallelization was handled by simply relying on the Parallel Extensions to .NET to provide a Parallel.For loop (see linked source for specific implementation details in Program.cs, line 173 and following… less than 100 lines total). Once all of the blocks were uploaded, we called PutBlockList() to assemble/commit the file in Azure storage. For each block transferred, the MD5 was calculated and sent ensuring that the bits that arrived matched was was intended. The timer for the blocked/parallelized transfer method wraps the entire process (source file splitting, block transfer, MD5 validation, file committal). A diagram of the process is as follows:
We then tested the affects of blocking & parallelizing the transfers by running the updated application against the same source set and did a parameter sweep on the block size including 256KB, 512KB, 1MB, 2MB, and 4MB (our assumption was that anything lower than 256KB wasn’t worth the trouble and 4MB is the maximum size of a block supported by Azure). The raw data for the parallel tests is available via the links in the Related Resources section at the bottom of this post.
This data was processed and then compared against the single-threaded / non-optimized transfer numbers and the results were encouraging. The Excel version of the results is available here.
Two semi-obvious points need to be made prior to reviewing the data. The first is that if the block size is larger than the source file size you will end up with a “negative optimization” due to the overhead of attempting to block and parallelize. The second is that as the files get smaller, the clock-time cost of blocking and parallelizing (overhead) is more apparent and can tend towards negative optimizations. For this reason (and is supported in the raw data provided in the linked worksheet) the charts and dialog below ignore source file sizes less than 1MB.
(click chart for full size image)
The chart above illustrates some interesting points about the results:
- When the block size is smaller than the source file, performance increases but as the block size approaches and then passes the source file size, you see decreasing benefit to the point of negative gains (see the values for the 1MB file size)
- For some of the moderately-sized source files, small blocks (256KB) are best
- As the size of the source file gets larger (see values for 50MB and up), the smallest block size is not the most efficient (presumably due, at least in part, to the increased number of blocks, increased number of individual transfer requests, and reassembly/committal costs).
- Once you pass the 250MB source file size, the difference in rate for 1MB to 4MB blocks is more-or-less constant
- The 1MB block size gives the best average improvement (~16x) but the optimal approach would be to vary the block size based on the size of the source file.
(click chart for full size image)
The above is another view of the same data as the prior chart just with the axis changed (x-axis represents file size and plotted data shows improvement by block size). It again highlights the fact that the 1MB block size is probably the best overall size but highlights the benefits of some of the other block sizes at different source file sizes.
This last chart shows the change in total duration of the file uploads based on different block sizes for the source file sizes. Nothing really new here other than this view of the data highlights the negative affects of poorly choosing a block size for smaller files.
Summary
What we have found so far is that blocking your file uploads and uploading them in parallel results in significant performance improvements. Further, utilizing extension methods and the Task Parallel Library (.NET 4.0) make short work of altering the shipping client library to provide this functionality while minimizing the amount of change to existing applications that might be using the client library for other interactions.
Related Resources
Currently rated 1.0 by 1 people
- Currently 1/5 Stars.
- 1
- 2
- 3
- 4
- 5
The following represents data and results gathered from the second research institution connection cloud transfer test and compares results from Azure’s US North Central data center and Azure’s US South Central data center. The methodology applied during this test is detailed here and should be reviewed prior to considering the results or commentary below.
Test Overview:
- 05561 Cloud Transfer Tests: Research Institution Test 02
- Local Connection: Research Networks
- Started: February 9, 2010
- Finished: February 16, 2010
- Origination Point: Oak Ridge, TN
Disclaimer:
- Standard Disclaimer Applies
Test Objectives:
- Standard objectives apply
- Specific to this test: Test a research institution connection as the researcher’s “workstation” and gather data aimed at building a realistic expectation of performance
Test Setup
- Included File Sizes:
- 2KB, 32KB, 64KB, 128KB, 256KB, 512KB, 1MB, 5MB, 10MB, 25MB, 50MB, 100MB, 250MB, 500MB, 750MB, 1GB
- Network Connectivity - “research institution”
- Consists of a computer connected to a local network router via 100Mbps hard-wire.
- Multiple switches/routers/firewalls may exist between workstation and the public internet
- There may exist multiple high-speed networks that may be leveraged for connectivity to remote datacenters (ESNet, I2, NLR
- Reasonable effort has been made to ensure that no other applications or TSRs are running on the source computer for the duration of the test.
- For this test, a newly-installed Windows 7 Professional installation was used, fully patched, with no other applications (beyond the test harness) installed.
Test Execution:
- Standard execution approach applied with the exception of the fact that Azure was tested for both cases – simply different datacenters (see slides for details)
Report Generation
- Standard report generation approach applied
Conventions:
- Standard conventions apply
Resources:
- Standard resources apply - no test-specific customizations beyond adaptations for the specific file sizes included in the test
Results:
Similar to other tests, there is some variability displayed that is obviously a result of traffic issues. We are continuing to look into this.
In general, the data from the Azure US North Central data center proved better than that of US South Central which is not altogether surprising as we are physically closer to the USNC location.
Slides 171 and 172 remain disturbing as the download values for the 750MB file size continue to be outside of what would be expected.
Slide 172 in particular is of interest as it draws attention to some wide variability across file sizes for the USSC datacenter (not just the 750MB size).
Full results are available in slide form here:
PDF of results are available here: http://sciencecloud.us/media/05561_Xfer-Research_02.pdf
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
The following represents data and results gathered from the first research institution connection cloud transfer test. The methodology applied during this test is detailed here and should be reviewed prior to considering the results or commentary below.
Test Overview:
- 05561 Cloud Transfer Tests: Research Institution Test 01
- Local Connection: Research Networks
- Started: February 8, 2010
- Finished: February 16, 2010
- Origination Point: Oak Ridge, TN
Disclaimer:
- Standard Disclaimer Applies
Test Objectives:
- Standard objectives apply
- Specific to this test: Test a research institution connection as the researcher’s “workstation” and gather data aimed at building a realistic expectation of performance
Test Setup
- Included File Sizes:
- 2KB, 32KB, 64KB, 128KB, 256KB, 512KB, 1MB, 5MB, 10MB, 25MB, 50MB, 100MB, 250MB, 500MB, 750MB, 1GB
- Network Connectivity - “research institution”
- Consists of a computer connected to a local network router via 100Mbps hard-wire.
- Multiple switches/routers/firewalls may exist between workstation and the public internet
- There may exist multiple high-speed networks that may be leveraged for connectivity to remote datacenters (ESNet, I2, NLR
- Reasonable effort has been made to ensure that no other applications or TSRs are running on the source computer for the duration of the test.
- For this test, a newly-installed Windows 7 Professional installation was used, fully patched, with no other applications (beyond the test harness) installed.
Test Execution:
- Standard execution approach applied
Report Generation
- Standard report generation approach applied
Conventions:
- Standard conventions apply
Resources:
- Standard resources apply - no test-specific customizations beyond adaptations for the specific file sizes included in the test
Results:
Across both services there exists an interesting amount of variability that is likely due to intermediate traffic or traffic management issues. Even within the same test run (see various scatter plots) you can detect “walls” of change wherein a the values will be hovering around a certain value and subsequently they hover around a much higher/lower value (ex. slide 133, 134).
There is not a consistent “winner” in this report. for various file sizes one platform would clearly outperform the other only to have the tables completely reversed for the next file size. This hints at network routing issues. A brief conversation with some of our local networking team indicates that some traffic (in particular Amazon’s) appeared to generally leave via the router connected to ESNet whereas most of the Microsoft traffic would leave via the router connected to Southern Crossing with subsequent connections to I2 and NLR. It may well be that the insertion of some static routs may help address some of the stability issues here.
Of particular interest is the “hump” seen by both services in slide 170. This has been seen in a similar location on the chart in other runs (see slide #82 here: http://www.slideshare.net/rgillen/cloud-storage-upload-tests-02). We don’t yet have a good explanation for this shape in the curve and are hoping to track that down soon.
Further, the shape of the Azure curve in slide 171 is inconsistent with other tests – specifically the data points for the 750MB size. We will continue to compare with other sets/runs to see if this continues or was simply transient.
What remains consistent across all tests so far is that the level of variability tends to be greater with the S3 platform as compared to the Azure Blob storage.
Full results are available in slide form here:
PDF of results are available here: http://sciencecloud.us/media/05561_Xfer-Research_01.pdf
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
The following represents data and results gathered from the first consumer connection cloud transfer test. The methodology applied during this test is detailed here and should be reviewed prior to considering the results or commentary below.
Test Overview:
- 05561 Cloud Transfer Tests: Consumer Connection Test 01
- Local Connection: Comcast Residential
- Started: February 9, 2010
- Finished: February 14, 2010
- Origination Point: Knoxville, TN
Disclaimer:
- Standard Disclaimer Applies
Test Objectives:
- Standard objectives apply
- Specific to this test: Test a consumer/commodity connection as the researcher’s “workstation” and gather data aimed at building a realistic expectation of performance
Test Setup
- Included File Sizes:
- 2KB, 32KB, 64KB, 128KB, 256KB, 512KB, 1MB, 5MB, 10MB, 25MB, 50MB, 100MB
- Network Connectivity - “typical home network”
- Consists of a computer connected to a local router via 1GE hard-wire.
- Router is then directly connected to service provider’s modem
- Consumer has a “general” plan for internet connectivity
- Reasonable effort has been made to ensure that no other applications or TSRs are running on the source computer for the duration of the test.
- For this test, a newly-installed Windows 7 Professional installation was used, fully patched, with no other applications (beyond the test harness) installed.
Test Execution:
- Standard execution approach applied
Report Generation
- Standard report generation approach applied
Conventions:
- Standard conventions apply
Resources:
- Standard resources apply - no test-specific customizations beyond adaptations for the specific file sizes included in the test
Results:
In contrast to some other test runs on other networks, in this test Azure seemed to generally (if barely) out-perform the Amazon platform and, consistent with other tests, Amazon’s interaction with Amazon’s platform shows greater variability across a given file size.
The test was limited to file sizes up to and including 100MB so as to avoid being flagged by the residential ISP for poor traffic habits (an issue to be addressed for large-bandwidth users on consumer connections).
Full results are available in slide form here:
PDF of results are available here: http://sciencecloud.us/media/05561_Xfer-Consumer_01.pdf
Currently rated 5.0 by 1 people
- Currently 5/5 Stars.
- 1
- 2
- 3
- 4
- 5
The following describes the methodology applied to some of the data transfer tests we are performing for various cloud storage platforms. In each case, the following approach should be assumed with the exception of test-specific details which will be posted with each result set.
Disclaimer:
- The research team understands that any time the public internet is introduced into a test a number of non-controllable factors are introduced. It is the intent of this project to test various scenarios often enough and with enough variance to obtain a reasonable average and thereby allow the team to make general assumptions about the quality of service (given the constraints stated) that one can reasonably expect to encounter when utilizing a given service.
- It is similarly understood that there may exist environmental factors (i.e. routing paths, proxy servers, firewalls) that affect the transfer rates being tested. In general, it is believed that these factors should affect all tested platforms equally. However, in the case of various research institutes where specialty networks (i.e. ESNet, NLR, I2) exist, there may be routing configurations that particularly favor one service or endpoint over another. It is an objective of these tests to expose these anomalies with the goal of addressing them as appropriate.
- Baseline: For the various services tested, these initial tests were performed using no particular optimization techniques. We took the respective vendor’s shipping SDK, integrated it into a very similar wrapper (source code available for verification) and executed it. Subsequent work should focus on optimizations in the SDKs, or the methods in which the libraries are utilized, etc.
- Not A Stand-Alone Work: This data should not be considered in isolation. Rather, it is a portion of a larger data set (some of which may remain to be published) and should be interpreted for what it is – a portion of a larger collection that aims to provide a more complete view of the entire problem domain.
Test Objectives:
- General: Generate data to set expectations for users of various cloud services focusing on a scenario of local compute combined with cloud-hosted data (blob storage). Note: the reverse scenario as well as cloud-hosted compute/cloud-hosted data will be tested separately
- These tests and data are crucial to our overall objective of improving the experience of researchers interacting with cloud computing assets as they provide a baseline against which any optimizations or alterations may be compared.
Test Setup:
- Test Setup
- A collection of random-data files were generated (RandomFileGenerator.exe). For each of the following file sizes, 50 files were generated and stored on standard disks local to the test computer: Range is specific to each test set.
- Network Connectivity: specific to each test set
Test Execution:
- For each file size, AWS_Console_App1.exe was called to upload the files to Amazon’s US Standard Region and record the duration
.\amazon\aws_console_app1.exe .\data\2KB
- For each file size, DownloadFiles.exe was called to download the files just uploaded to Amazon’s US Standard Region and record the duration
.\downloader\DownloadFiles.exe -i .\amazon_2KB.csv -p 6 -m yes
- For each file size, AzureTesting.exe was called to upload the files to Azure’s US North Central region and record the duration
.\azure\azuretesting.exe .\data\2KB
- For each file size, DownloadFiles.exe was called to download the files just uploaded to Azure’s North Central region and record the duration
.\downloader\DownloadFiles.exe -i .\azure_2KB.csv -p 6 -m yes
- NOTE: immediately following each operation for each file size, the resulting file (log.csv) was renamed to represent the source, transfer direction, and file size
ren log.csv azure_ussc_upload_2KB.csv
Report Generation:
- For each service tested, and each file size tested
- For both Uploads and Downloads (separately)
- Scatter plot is generated showing the distribution for the transfer duration (seconds)
- Scatter plot is generated showing the distribution for the transfer rate (Mb/s)
- Transfer duration average (seconds) is calculated
- Transfer duration standard deviation (seconds) is calculated
- Transfer rate average (Mb/s) is calculated
- Transfer rate standard deviation (Mb/s) is calculated
- For each file size tested
- For both Uploads and Downloads (separately)
- A comparison chart (column) is generated showing the average transfer duration (seconds) and error bars indicating one standard deviation (seconds). Also plotted is a dot indicating the associated average transfer rate on the secondary Y axis (Mb/s)
- Summary Charts
- For both Uploads and Downloads (separately)
- A range chart is generated showing the band covered by one standard deviation (per service tested) for the transfer duration (seconds) across the tested file sizes
- A range chart is generated showing the band covered by one standard deviation (per service tested) for the transfer rate (Mb/s) across the tested file sizes
- Presentation
- Once the above charts have been generated, they are assembled into a PowerPoint file
- Once the power point file has been generated and saved, it is published as a PDF file
- Automation
- All of the above steps are automated via a script (ProcessTransferLogs.ps1)
Conventions:
- Naming Conventions
- Amazon_USSTD: Amazon’s US Standard region was specified when the bucket was created
- Azure_USNC: Azure’s US North Central region was selected when the storage account was created
- Error Handling
- In most runs, errors were displayed to the screen but not captured to logs.
- Existence of errors (all of which were network-related) are manifested in the logs as collections of data points less than 50 (the test source size)
- Due to the fact that the respective download tests are based on the upload source files, a download file containing less than 50 entries is not necessarily indicative of errors but may simply be tied to the fact that the input file had less than 50 entries. This being said, there were more errors on downloads than uploads.
Resources:
Results: Specific to each test set
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
I’ve been getting my test harness and reporting tools setup for some performance baselining that I’m doing relative to cloud computing providers and when I left the office on Friday I set off a test that was uploading a collection of binary files (NetCDF files if you care) to an Azure container. I was doing nothing fancy… looping through a directory, for each file found, upload to the container using the defaults for BlobBlock and then record the duration (start/finish) for that file and the file size. The source directory contained 144 files representing roughly 58 GB of data. 32 of the files were roughly 1.5 GB each and the remainder were about 92.5 MB.
I came in this morning expecting to find the script long finished with some numbers to start looking at. Instead, what I found is that, after uploading some 70 files (almost 15 GB), every subsequent upload attempt failed with a timeout error – stating that the operation couldn’t be completed in the default 90-second time window. I started doing some digging into what was happening and so far have uncovered the following:
- By default, the Storage Client that ships with the November CTP breaks your file up into 4 MB blocks (assuming you are using BlobBlock – which you should if your file is over the 64 MB limit.
- The client then manages 4 concurrent threads uploading the data. as each thread completes, another is started – keeping four active most the entire time.
- At some point Saturday afternoon (just after 12 noon UTC), the client could no longer successfully upload a 4 MB file (block) in the 90 second window, and all subsequent attempts failed.
- I initially assumed that my computer had simply tripped up or that a local networking event caused the problem so I restarted the tool – only to find every request continuing to fail.
- I then began to wonder if the problem was the new storage client library (not sure why) so I pulled out a tool to manage Azure storage – Cloud Storage Studio (http://www.cerebrata.com/Products/CloudStorageStudio/Default.aspx) and noticed that I was able to successfully upload a file. I remembered that CSS (by default) splits the file into fairly small blocks, so I cracked open Fiddler and began monitoring what was going on. I learned that it was using 256 KB blocks (this is configurable via settings in the app).
- I then adjusted my upload script to set the ServiceClient.WriteBlockSizeInBytes property (ServiceClient is a property of the CloudBlockBlob object) to 256k and re-ran the script. This time, I had no troubles at all (other than a painfully slow experience).
- So, I can upload data (not a service outage) but while 256K blocks work, the 4 MB blocks that worked on Friday no longer work – I’m assuming that there’s a networking issue on my end, or something in the Azure platform. To provide more clarity, I adjusted the tool again, this time using a WriteBlockSizeInBytes value of 1MB and re-ran the tool – again, seeing successful uploads.
While this last step was running, I thought it might be good to go back and do some crunching on the data I had so far. The following chart represents the uploads rate from the files that successfully were uploaded on Friday/Saturday followed by the a chart showing the probability density. The mean rate was 2.74 mbits/sec with a standard deviation of 0.1968. It is interesting to note that there was no upward drift at the end of the collection of successful runs, indicating that more than likely, the “fault” was likely caused by something specific rather than being the result of a gradual shift or failure based on usage (imagine a scenario wherein as more data is populated in a container, indexes slow down, causing upload speeds to trail off).
Upload Speeds [click image for full size]
Probability Density [click image for full size]
I then ran similar reports against the data I from this morning’s runs. I’m still in the process of generating a full report on the data, but a representative sample shows the following: The mean upload rate was 0.15 mbits/sec with a standard deviation rate of 0.0375. This is over 17x slower than Friday. This data points represented below are for three batches – the first batch used a WriteBlockSizeInBytes of 256K, the second used 1MB, and the third used 2MB (10 points per size). The file upload did not succeed with the 2MB size – only finished about 1/4th of the full file.
Upload Speeds [click image for full size]
Probability Density [click image for full size]
I’ve seen a few comments from others today that indicate the slow down may be widespread – My next course of action is to attempt to run the tests from a few different locations to hopefully eliminate my local network as the problem set and have more data with which to address the issue.
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
I’ve been working on moving a large collection data to, from, and around Azure as we are testing the data profile for scientific computing and large-scale experiment post-processing and, in order to verify the data we uploaded and processed turned out as we wanted tit to, I built a simple visualization app that does a real-time query against the data in Azure and displays it. Originally the app was built as a simple WPF desktop application, but I got to thinking that it would be particularly interesting on the Surface and therefore took a day or two to port it over. The video below is a walkthrough of the app – the dialog is a bit cheesy but the app is interesting as it provides a very tactile means of interacting with otherwise stale data.
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
It’s late on the Friday afternoon before Christmas week which means things are pretty quiet around the office. This quiet has the net-effect of allowing me to get quite a bit done. The last few days have been very productive with respect to our research project and Azure work (more on that coming soon) which is now in full swing. We are currently working on collecting performance data from our codes running in Azure (and soon in the Amazon cloud) and are also doing some testing of transfer speeds of data both to/from the cloud as well as between compute and storage in the cloud.
I’ve been working to automate much of this testing so we can do things in a repeatable fashion as well has have something that others could run (both other users like ourselves as well as possibly vendors should we come across something that requires a repro scenario). So far, running tests and generating data in CSV or XML format is pretty simple, but I found myself wanting to automatically generate charts/graphs of the data as part of the test process to allow a quick visualization of how the test performed. I spent a good bit of the day looking at old tools for command-line generation of charts (i.e. RDTool, etc.) and none of them were exactly what I was looking for – not to mention my proclivity to using C# and VS.NET tools and my desire to have something that looked refined/polished and not overly raw.
Thankfully, I stumbled upon something I should have remembered existed but simply hadn’t had the need to use before – the System.Windows.Forms.DataVisualization.Charting class. If you aren’t familiar with this assembly, it was released at PDC08 and has a companion Web class for performing similar operations in ASP.NET applications. In my basic testing I was able to build a console application that would ingest the CSV output from my testing harness and then generate some fairly nice looking charts based on that data. The following shows a chart (click the chart to see it full size) generated from ~1800 data points, and automatically generates a 50% band and 90% band allowing the viewer to very easily ascertain the averages and data points. This was generated using a combination of the FastPoint and BoxPlot chart types.
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
Last Friday I posted some initial results from some simplistic testing I had done comparing pulling data from Azure via ATOM (the ADO.NET data services client) and JSON. I was surprised at the significant difference in payload and time to completion. A little later, Steve Marx questioned my methodology based on the fact that Azure storage doesn’t support JSON. Steve wasn’t being contrary, but rather pushing for clarification to the methodology of my testing as well as a desire to keep people from attempting to exploit the JSON interface of Azure storage when none exists. This post is a follow up to that one and attempts to clarify things a bit and highlight some expanded findings.
The platform I’m working against is an Azure account with a storage account hosting the data (Azure Tables), and a web role providing multiple interaction points to the data, as well as making the interaction point anonymous. Essentially, this web role serves as a “proxy” to the data and reformats it as necessary. After Steve’s question last week, I got to wondering particularly about the overhead (if any) the web role/proxy was introducing and if, esp. in the case of the ATOM data, it was drastically affecting the results. I also got to wondering if the delays I was experiencing in data transmission were, in some part, caused by the fact of having to issue 9 serial requests in order to retrieve the entire 8100 rows that satisfied my query.
To address these issues, I made the following adjustments:
- Tweaked my test harness for ATOM to optionally hit the storage platform directly (bypassing the proxy data service).
- Tweaked the data service to allow an extra query string parameter to indicate that the proxy service should make as many calls to the data service as necessary to gather the complete result set and then return the results as a single batch to the caller. This allowed me to eliminate the 1000 row limit as well as to issue only a single HTTP request from the client.
- I increased the test runs from 10 to 20 – still not scientifically accurate by any means, but a bit longer to provide a little better sense of the average lengths for each request batch.
The results I received as follows and not altogether different than one might expect:
As you can see from the charts above, the JSON FULL option was the fastest with an average time to completion of 14.4 seconds. When compared to the regular JSON approach, you can infer that the overhead introduced from multiple calls is roughly 4 seconds (18.55 average time to completion).
In the ATOM category, I find it interesting that the difference between the ATOM Direct (directly to the storage service) was only marginally faster (0.2 of a second on average) than the ATOM FULL approach. This would indicate that the network calls between the web role and the storage role are almost a non-factor (hinting at rather good network speeds). Remember, in the case of ATOM Full, the web role is doing the exact same thing as the test client is doing in Atom Direct, but additionally bundling the XML response into a single blob (rather than 9) and then sending it back to the client.
The following chart shows the average payload per request between the test harness and Azure. Atom Full is different then Atom and Atom Direct in that the former is all 8,100 rows whereas the later two represent a single batch of 1000. It is interesting to note that the JSON representation of all 8,100 records is only marginally larger than the ATOM representation of 1,000 records (1,326,298 bytes compared to 1,118,892 bytes).
At the end of the day, none of this is too surprising. JSON is less verbose in markup than ATOM and would logically be smaller on the wire and therefore complete sooner (although I wouldn’t have imagined it was a factor of 9 difference). What is interesting, is that the transfer of data b/t the data layer and the web role is almost trivially fast (remember, that 9 MB of XML moved between the layers and was then reformatted as JSON and shoved down back to the client in 14 seconds). It further makes you wonder what the performance improvement would/could be if Azure storage exposed a native JSON interface…
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5
UPDATE 8/20/2009, 15:29 EST: There is some confusing content in this post (i.e. Azure storage doesn’t support JSON). A follow up to this post with further explanation/detail is available here
UPDATE 8/14/2009, 17:16 EST: @smarx pointed out that this post is a bit misleading (my word) in that Azure storage doesn’t support JSON. I have a web role in place that serves the data, which, upon reflection could be introducing some time delays into the Atom feed. I will test further and update this post.
---
I’m just really beginning to scratch the surface on my work on cloud computing and scientific computing but it seems that nearly every day I’m able to spend time on this I come away with something at least moderately novel. Today’s observation is, on reflection, a bit of a no-brainer but it wasn’t immediately obvious to me.
I’m kicking around some scientific data and have a single collection of data, with somewhere north of 40,000 subsets of data, with each subset containing roughly 8,100 rows. I’ve had an interesting time getting this data into Azure tables, but that’s not the point of this post. Once the data resided in Azure, I built a little ADO.NET client to pull the data down based on various queries (in my case, a single “subset” at a time, or 8,100 rows). In case you are wondering, the data is partitioned in Azure based on the key representing each subset, so I know that each query is only hitting a single partition. I proceeded to follow the examples for paging and data calls (checking for continuation tokens, etc.) and it wasn’t long before I had a client that would query for a particular slice of data, and then make however many individual data calls necessary until the complete result set was downloaded and ready for processing. I was, however, disappointed in the time it took to pull down a single slice of data… averages were around 55 seconds. Pulling down a number of slices of the dataset, at nearly a minute each, was a bit slow.
I spent some time poking around with Fiddler and some other tools and discovered that I was suffering from XML bloat. The “convenience” factor of having a common, easy-to-consume format was killing me. Each response coming back from the server (1000 rows) was averaging over 1MB of XML.
After a while of kicking around my options (frankly, too long), I decided to try pulling the data as JSON. I hadn’t used JSON previously, but had heard it touted as being very lightweight. I also found some nice libraries on CodePlex for de-serializing the response so I could use it as I had the results of the Atom feed. Once I made this change, I was shocked to see the amount of improvement (I expected some, but what I saw was much more than anticipated). My average time dropped to around 14 seconds for the entire batch and the average size of the response body dropped to about 163k.
I’ve included some charts below showing the results of my tests. I ran the tests 10 times for each of the approaches. Code base for each test harness is identical with the exception of the protocol-specific text handling. Time measurements are from the start of the query through the point that each response has been de-serialized into an identical .NET object (actually, a List<T> objects).
These first two charts show the time required to retrieve the entire slice of data. The unit of measure is seconds.
These charts show the average payload returned per request for the two different methods. In both cases, the unit of measurement is bytes.
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5