Search ESS
-
Jun 18, 2013 12:09 GMT
-
Jun 14, 2013 12:56 GMT
Scaling properties of common statistical operators for gridded datasets
| Title | Scaling properties of common statistical operators for gridded datasets |
| Publication Type | Journal Article |
| Year of Publication | 2007 |
| Authors | Zender, C. S., & Mangalarn H. |
| Journal | International Journal of High Performance Computing Applications |
| Volume | 21 |
| Pagination | 485-498 |
| Date Published | 12/2007 |
| Type of Article | Proceedings Paper |
| ISBN Number | 1094-3420 |
| Accession Number | http://apps.isiknowledge.com/InboundService.do?Func=Frame&product=WOS&action=retrieve&SrcApp=EndNote&Init=Yes&SrcAuth=ResearchSoft&mode=FullRecord&UT=000250718500009 |
| Keywords | analysis; climate system model; computational model; data; data access; geoscience; interface; netCDF; performance; scaling; Zender Modeling Lab |
| Abstract | An accurate cost model that accounts for dataset size and structure can help optimize geoscience data analysis. We develop and apply a computational model to estimate data analysis costs for arithmetic operations on gridded datasets typical of satellite- or climate model-origin. For these dataset geometries our model predicts data reduction scalings that agree with measurements of widely used geoscience data processing software, the netCDF Operators (NCO). I/O performance and library design dominate throughput for simple analysis (e.g. dataset differencing). Dataset structure can reduce analysis throughput ten-fold relative to same-sized unstructured datasets. We demonstrate algorithmic optimizations which substantially increase throughput for more complex, arithmetic-dominated analysis such as weighted-averaging of multi-dimensional data. These scaling properties can help to estimate costs of distribution strategies for data reduction in cluster and grid environments. |
| URL | pub/730 |
| Alternate Journal | Int. J. High Perform. Comput. Appl. |