Distributed Parallel Processing Based on Grid Datafarm Architecture

Abstract

In the Solar-Terrestrial Physics field, satellite observation data and computer simulation data have been tremendous increased. Since most of data files and computer resources are distributed over the Internet, analysis environments for data intensive processing are required. In this study, we propose a parallel distributed processing system with meta-data system and Grid Datafarm. A testing system is constructed with 8 file system nodes. As a result of small-data processing of observation data on the system, parallel processing is found effective using metadata file at local disk and hierarchical Gfarm file. As for parallel visualizations of simulation data, it was achieved high parallelization efficiency of 97.6% when using FIFO-type scheduling.