The RPC server smadata_svc provides several data services to process the data received from real-time computers (Crates and Hal). As soon as the server receives a wake up signal (that signifies the observing run to start) from a real-time computer, several FITS-IDI (Diamond 1997; and Flatters 1998) table files are created with a postfix indicating the type of data table (such as FILENAME.UV, FILENAME.AG, FILENAME.AN, FILENAME.FQ, FILENAME.SU...). The UV data transferred from the Crates will be assembled by the server process smadata_svc along with the random parameters for each integration and each baseline. These data are appended continuously into the FILENAME.UV as the observing and the real-time correlation process goes on. The ancillary data is appended into the associated tables if any updating information occurs either as the source changes or as the correlator configuration changes. Immediately after each observing track, these FITS tables are packed into a single FITS-IDI file. This FITS-IDI will be moved to the On-line Storage System. Meanwhile, in the process FITStoDB (Tsutsumi & Zhao 1999), the FITS header information along with the other ancillary data such as the setup parameters for the correlator configuration and the information for the FITS-IDI file location is stored into the SMA Astronomical Database (SMADB see Section 2.2) in Sybase. The details regarding the data transfer and data structure for the FITS-IDI standard have been discussed in the SMA Technical Memo 126 (Zhao 1998).
Two points that we would like to emphasize: (1) the storage format for the data taken from a single observing run must follow an international standard which is platform and medium independent. The FITS-IDI is the best standard for packing up the raw visibility data in real-time. A FITS binary table is a collection of row data fields, organized into columns. This structure is also similar to the table structure used in a relational database. The information contained in the FITS-IDI file can be easily organized into our RDBMS. (2) the reasons why we do not store all the interferometer data, including the visibility and other ancillary data into Sybase, is because the SMA data sets will be very large in particular if the integration time is shorter than 10 sec. The procedures and I/O within Sybase Servers would not be efficient to handle large data files and will make the archiving process complicated and slow. This will also cause difficulty and problems in database management. The data in the FITS header and some of the associated tables are stored in SMADB. The individual FITS-IDI data file can be stored on an on-line file system that requires special hardware devices. The information about and location of each FITS-IDI file will also go to the RDBMS. These data are sufficient for users (astronomers) to review the basic information for each SMA observing run and determine where to get the FITS-IDI data file that they want. Thus, with minimal I/O activity the archive data model for our system becomes simple and easy to understand. Also in this system, the individual FITS-IDI files can be put on a portable medium for manually delivery without interruption or slow down to the SQL server.
This storage design meets two basic requirements of (1) the flexibility in data retrieval process and (2) high efficiency for data management.