- SOFIA Overview
- Proposing & Observing
- Meetings and Events
SOFIA's science archive has recently completed its transition to the Infrared Science Archive (IRSA) hosted by the Infrared Processing & Analysis Center (IPAC) as its primary data archive. SOFIA data from Cycle 1 onwards are now searchable through IRSA SOFIA Archive, including:
- Archival data: data publicly available from programs that have passed their exclusive-use period and from SOFIA's Director's Discretionary Time programs, which have no exclusive-use period. More information about approved DDT programs and currently available DDT data can be found here. No log in is required to download archival data.
- Proprietary data: data which have not yet passed their exclusive-use period, available only to guest observers (GOs) and delegated users of programs. Note that co-Is who used their email registered with a DCS account when submitting the observing proposal are automatically considered as delegated users. To obtain delegated status, other users should ask the program PI to contact SOFIA's Help-Desk. An IRSA account is required to download proprietary data. GOs and delegated users of SOFIA data who did not have an existing IRSA account should have received an email informing them that an IRSA account was created on their behalf. If you did not get that information, you can always register for an IRSA account here - if you use the same contact email as for your DCS account, then all the proprietary permissions pertaining to your SOFIA data should be active.
For tips on SOFIA specific archive searches and taking advantage of IRSA visualization features, check out IRSA's video tutorials here. Note that the SOFIA Data Cycle System website is still online and active, and is the main portal for tasks related to SOFIA proposals and AORs (Phase I and II).
Gathering and Navigating SOFIA Data
Whether you have completed your own SOFIA observations or want to use publicly available datasets in the archive, below are a few tips to help you get started with exploring SOFIA data.
Jump to Section:
All raw and processed SOFIA data are stored in the IRSA SOFIA Archive. The archive can be searched by any user, without logging in. Archival data can be downloaded by any user, without logging in. Downloading proprietary data for a given project is only accessible to the Guest Observers on the project and delegated users, and an IRSA account is required. Register for an IRSA account here.
Launch a query using in the IRSA SOFIA Archive search form. Note that the default search queries by target, so unless you are defining a target name /coordinates in the 'Spatial constraints' section, you should tick the 'All Sky' option in that section. Some common queries include the following:
- by proposal ID: 'Plan ID' field in Proposal Constraints section, or by PI name
- by target: A target name can be entered in Object name in the 'Spatial constraints' section, and is automatically resolved to known coordinates using SIMBAD or NED. Alternatively, coordinates can also be entered directly (equatorial, galactic or ecliptic). It is recommended to enter a relatively large spatial search radius (for example, 600 arcseconds) to avoid excluding relevant observations.
- by instrument: in the drop down menu in the 'Instrument Constraints' section
- by data processing level, in the 'Data Products contraints' section: Generally speaking, Level 0 and Level 1 correspond to raw data, and Level 2 to data corrected for instrument artifacts. Level 3 (flux calibrated data) and Level 4 (higher level processing, such as combined observations, large maps) are usually the data processing levels most useful for scientific analysis. The default is both Level 3 and 4 being queried, and is the recommended way to proceed (Level 4 products are not produced for all observations). See the Data Processing webpage for information on processing levels.
Once the selected fields are defined, click search. This IRSA video tutorial provides a quick overview of the query features.
The query's outcome shows two panels: on the left, lists of items - AORs (in the AOR tab) and datasets (in the Instruments tabs). On the right, a description of the item currently selected on the left (header information, mosaic coverage plot, image plot).
In the AOR tab, each row corresponds to an AOR.
In the instrument tabs, each row corresponds to a data product. Level 3 and 4 data products usually correspond to a unique combination of AOR, Observation ID (time unit of data processed together, which can be as long as multiple flights), spectral element and product type. Files can correspond either to single-mission or multi-mission (combined) data. Note that if only single-mission files are available, they may be replaced in the archive by multi-mission data once multi-mission processing is performed.
There may be a large number of datasets listed in each instrument tab, spread over several pages in each tab: look for the left and right arrows to sort through pages. It may be helpful to only list the datasets which you may be interested in. Click on the funnel icon to enable filters at the top of each column. For example, scroll over to find the "Product Type" column, click on the down arrow, and select one of the available product types. For a detailed description of the products types available under each processing state for a given instrument, it is recommended to download the corresponding data handbook from the Documentation for Data Products webpage, which contains a detailed description of the pipeline calibration process and the pipeline products.
Once an instrument tab is selected, the tabs on the right change to include 'data' or 'preview' tabs. The 'data' tab shows the FITS file corresponding to the data product; some data products may not be displayable. GREAT and HAWC+ data may have the 'preview' tab populated instead of the 'data' tab. This IRSA video provides a more in-depth on how to use the coverage tab to visualize the data.
In the instrument tabs, click the desired boxes on the left hand side, then click 'prepare download'. You can select to have your data downloded in a zip file structured (with directories) or flattened (no directories).
If you selected a structured download, the data package is organized in a series of embedded directories, described here from upper level to lower level. This structure may vary slightly across instruments and observing cycles, and not all directory levels may be present.
- If both Level 3 and Level 4 data were downloaded, they will be stored in different upper level directories, labeled missions and level4 (L4) respectively.
- Under these upper level directories, one may find one directory per flight, named after the corresponding mission ID (for example 2018-07-05_HA_F481), as well as directories for products produced from different flights processed together.
- Within those, one can find subdirectories for each pipeline processing run, labeled pNNNN.
- Each pipeline processing directory is possibly further divided in several directories for each spectral element/AOR combination.
- The Level 3 and Level 4 products can then be found. Those are usually are in the .fits format. (For GREAT data, data in .great and .lmv format can also be found, as well as tarballs in .tar)
Filenames can give you information on the nature and content of the file even without looking at its header. Typically, a filename follows the template Flight_IS_MOD_AOR-ID_SPECTEL_Type_FN1-FN2.fits, where Flight is the SOFIA flight number, IS is the instrument identifier, SPE specifies the instrument mode, AOR-ID is the 8 digit AOR identifier for the observation, SPECTEL is the keywords specifying the spectral configuration, Type are three letters identifying the product type, FN1 is the file number corresponding to the first input file, and FN2 is the file number corresponding to the last input file.
Collecting Information on Data Quality
All processed data have been through a quality analysis examination to determine whether they are usable. Comments on the data quality and suggestions on the best way to analyze the data may have been recorded. Those comments can be essential to understanding the data. Such comments can be found in the headers of Level 3 and Level 4 products, at the bottom of the HISTORY key.
- Note that for GREAT data, the comments can be found in the data reduction pdf included in the dowloaded data package (note that not all GREAT Level 4 packages contain the data reduction pdf, and it is sometimes necessary to download the corresponding Level 3 packages to obtain that file).
- Known data issues associated with each flight (listed by mission ID) are also collected in the "Catalog of Known Data Product Issues Index" document, which is regularly updated.
Gather Information on Observations:
- Information on observation specifics can be found under the header keys in the fits products, such as EXPTIME (Total on-source integration time), WVZ_END and WVZ_STA (water vapor).
- For GREAT data, comments on observations can be found in the observing log pdf included in the downloaded data package (note that not all GREAT Level 4 packages contain the data reduction pdf, and it is sometimes necessary to download the corresponding Level 3 packages to obtain that file)
- If you are the guest observer of the proposal from which the observations were performed, we recommend examining the flight reports which were sent to you by the instrument scientists after each observing series. Those may contain useful information on observing conditions.
You have now collected all the available data and information to start data analysis. If you are not familiar with SOFIA data, you may find it useful to look at our data recipes, which were written to show how to make simple plots and carry out basic data analysis tasks.
Contact the Help-Desk for any question related to your data. The staff at the SOFIA Science Center is there to support you in your exploration of your data.