The International MODIS/AIRS Processing Package (IMAPP) is a suite of programs that lets anybody to process MODIS (and in future releases AIRS ) data from Level-0 to Level-1A and 1B products. The package has been derived at Space Science and Engineering Center (University of Wisconsin-Madison) from the operational MODIS processing software developed at NASA GSFC. It is distributed freely under the terms of GNU General Public License. Package description and source distribution can be found at the primary site at SSEC: http://cimss.ssec.wisc.edu/~gumley/IMAPP/IMAPP.html. The original IMAPP was designed to run on Unix. R&D center ScanEx has adopted it to run on Microsoft Windows platform. The last variant (let's call it IMAPPW) is hereafter described in this manual.
IMAPP can be used for entry levels processing of MODIS data level-0 received with direct broadcast (DB) station from Terra and Aqua satellites. This processing includes following operations
This paper tries to resolve a lack of comprehensive IMAPP users's manual for those who don't like to drill down into a source code. It does not provide full coverage of the topic but focuses on the questions that are important for productive operation work. So we highly recommend you to read the SSEC's IMAPP page too. For description of processing algorithms, data formats and data interpretation please refer to documentation available from NASA.
| ver 1.5R1 16 Jan 2004 |
This release contains calibration code patches that resolve problems with the nadir door state flags. The problems with Terra's MODIS might lead to the missing image data in MOD02 products. |
| ver. 1.5 12 Nov. 2003 |
Release is based on the IMAPP ver. 1.5.
|
| ver. 1.4 30 July 2003 |
Release is based on the IMAPP ver. 1.4
|
| ver 1.3 24 July 2002 |
Release is based on the IMAPP ver. 1.3.
|
Look at the IMAPP revision history (at http://cimss.ssec.wisc.edu/~gumley/IMAPP/IMAPP.html) for the changes details.
Thinking to install IMAPP on Windows platform? Check the following requirements:
Other computer's properties have no crucial meaning.
To perform installation unzip the downloaded distribution and run setup.exe. Automated installation will place most necessary files at proper places under a directory you specified and perform all necessary settings. It is good idea to specify during installation a valid directory you plan to use for storing level-1A files (have at least several GB free there). If you are installing the full package you are ready to run it (go to the next chapter). If you are installing distribution created more than 1-2 month ago you should also update two critical ancillary files (utcpole.dat and leapsec.dat) before you start using the IMAPP (see this manual further for how to find them). In order to simplify run-time operation all ancillary data by default are placed in the \ancillary subdirectory of the installation root but this can be customized on file-per-file basis.
Please, note that the installation package does not include large ancillary files (namely Land-Sea mask and DEM) required for geolocation. These two file groups require about 0.8 and 1.8 GB disk space respectively. They are not mandatory for running IMAPPW programs but if one needs Land-Sea or height fields filled with proper data he has to download and install these file separately. See Geolocation parameters description further in this text for installation details.
OK, so you've obtained a MODIS level-0 Production Data Set (PDS) with your EOS direct broadcast station. What? You don't have a a station yet? Why not consider an EOScan station from ScanEx.
Lets you have acquired from space a PDS file with name foo.L0.pds (.pds extension is significant). Select it in the Windows Explorer, open a context menu and click on "IMAPP Unpack" item. A command prompt window will pop-up and the unpack.exe program will start processing your file to level-1A. Relax... Processing a full Terra overpass data (acquired during 10-15 min) takes about 5-10 min. depending on the speed your box can run. After the console window closes look into the L1a directory you have specified during installation and you will find level-1A file with name MOD01.AYYMMDDHHMM.hdf , where YYMMDDHHMM denotes beginning time of the processed data.
In the same manner you can process L1A files further. First select "IMAPP geolocate" from the Explorer's context menu for MOD01.AYYMMDDHHMM.hdf and after a certain time you will get a geolocation product file MOD03.AYYMMDDHHMM.hdf in the same directory. More further use "IMAPP calibrate" menu item for MOD01.AYYMMDDHHMM.hdf to start calibration process. At this step you have enough time to have a dinner... Full passage granule requires about half to one hour to be calibrated.
If you have got a set of files with names like MOD02*.AYYMMDDHHMM.hdf (this is a standard level-1B product) - you are lucky. Otherwise read this manual further and try to resolve the problem.
IMAPP for Windows includes four console 32-bit applications. This means that you have to use a command prompt window to use them. To abort execution during processing use Ctrl+C keyboard sequence. The primary way to specify input and output file names and other processing parameters is to set them as environment variables. This is by original design and are quite common for Unix gurus. For those who have not used command prompt for a long time we provide following example of sequence of commands
SET L0File=c:\eos\l0 SET L1aFile=c:\eos\l1a unpack.exe |
Windows port of the IMAPP introduces several improvements simplifying its usage. The first one is usage of imapp.ini file to store rarely changing (lets call them "static") parameters. This is a common parameter=value file that can be edited as text. After initial installation records in this file should be valid and ready to use. This feature eliminates a need to employ global system environment or definition of many parameters during each session. If a parameter have been defined both in the INI file and environment - the last one have a precedence.
IMAPPW programs are also able to accept a name of "primary" input file as a command line argument. Possibility to run IMAPP programs from Explorer's menu uses this feature. This feature significantly relies on the scheme of default name generation because all other parameters are substituted by default values or are read from the imapp.ini (or global system environment). In short this scheme works as follows
Meaning of the most useful parameters are shortly described further in this manual. Each parameter definition is accompanied with a abbreviated list of possible methods to define this parameter. A method in the list have precedence over the subsequent ones. The following notation convention is used inside such lists:
| E (Environment) | the parameter can be defined via environment |
| I (INI) | the parameter can be defined via imapp.ini file |
| C (Command line) | the parameter can be specified in the command line |
| +, - | the parameter is mandatory (or not) |
Please, pay attention to the following note: all values representing directories must have a slash "\" at the end (e.g. "c:\imapp\log\").
IMAPP programs use several types of ancillary data. With the installation package ( or separately) you can obtain most the data you need to run the processing. But it should be emphasized that some data can change between version or be time dependant and require periodic renewals. Current information on usage and availability of the ancillary data can be obtained from SSEC's or Scanex's IMAPP sites. Details on usage of some ancillary files are provided further side-by-side with the particular program description.
Note: ancillary file in ASCI text must be preserver in the original format (do not require Unix <-> DOS convertion) - so use binary mode while transferring them between platforms
Several run parameters are common to the most IMAPPW executables. They are summarized here:.
Program unpack.exe reformats science and telemetry information from MODIS level-0 files into the level-1a files in a HDF format. Level-0 MODIS file contains a sequence of time-ordered CCSDS packets reconstructed from the satellite downlink stream. At EOSDIS such a file (strictly speaking a set of files) is called Production Data Set (PDS). We will use this term as well. At Dundee Satellite Receiving Station's site there is a page http://www.sat.dundee.ac.uk/modisformat.html that contains short introduction into the MODIS level-0 formats and links for further reading.
MODIS level-1A file is a standard EOSDIS science product identified as MOD01. This product is implemented in a HDF format and contains raw sensor data, on-board telemetry and created during processing quality information formatted as HDF objects (SDSs and attributes). This allows anybody to use general purpose programs to explore the data and get an MODIS earth view image on a screen.
Note for Aqua data processing: Orbital data from Aqua satellite come in separate data stream with APID 957( GBAD data stream). Receiving software by ScanEx stores these packets in separate files with extension .gbd. If the unpack.exe finds during run a *.gbd file with the same name as the input PDS file has it tries to run GBAD processor on this file too. Output attitude and ephemeris files are produces for each output MOD01 granule with corresponding names (*.att and *.eph). This feature simplifies future processing with geolocate.exe.
The following table lists ancillary data files used by unpack program along with according parameters specifying their location.
| Ancillary file | Parameter | Notes |
|---|---|---|
| ENG_DATA_LIST | EngDataList (E,I +) | List of telemetry data fields with their position in the input and output streams |
| leapsec.dat | LeapSec (E,I +) | Leap seconds file. Required for accurate time conversions |
Please, note that file leapsec.dat must be kept up-to-date. SDP Toolkit's support page (http://newsroom.gsfc.nasa.gov/sdptoolkit/leaputc.html) contains a link to the recent version and additional comments concerning this file. Another source of the data is maintained at ftp://g0dps01u.ecs.nasa.gov/OTHR/
Besides ancillary data location parameters unpack program uses following parameters
This program performs a geolocation processing of MODIS data and outputs an HDF file containing arrays of latitude/longitude and imaging angles for each 1km pixel. Output file has EOS product identifier MOD03.
IMAPP can calculate geolocation fields using satellite ephemeris and attitude information contained in the Direct Broadcast downlink stream. This feature lets to perform processing just after the data reception. IMAPP can also use definitive ephemeris/attitude ancillary files as the DAAC's operational software does to achieve more precise geolocation. The main disadvantage of the definitive data is that it may be available only after a certain time (1-2 days) after imaging.
Definitive data are provided now for public by SSEC at ftp://terra.ssec.wisc.edu/pub/terra/ephemeris/ . Attitude and ephemeris data files are provided separately and are organized by time. In order to use definitive data for geolocation you must have both attitude and ephemeris files for the appropriate time range. The files have names like "AM1ATTNF#001030720012000000000000". In order to decode it pay attention to the fields in bold. They mean: AM1 - Terra platform, ATT (or EPH) - attitude or ephemeris data, 03 -month, 07 -day, 2001- year, 20 - hour of start of the data. Most files covers time range of 2 hours. Please, note that authors of this manual are not responsible for mentioned definitive data source and information you have just read may be out-of-date.
| Ancillary file | Parameter | Notes |
|---|---|---|
| leapsec.dat | LeapSec (E,I +) | Leap seconds file (used by unpack.exe too) |
| GEO_parameters.dat | GeoParms (E,I +) | MODIS sensor geometry calibration parameters (Terra) |
| aqua/GEO_parameters.dat | AquaGeoParms (I) | MODIS sensor geometry calibration parameters (Aqua) |
| utcpole.dat | UTCpole (E,I +) | Earth motion file |
| de2000.eos | PlanetEphem (E,I+) | Space bodies ephemeris are in use for Sun and Moon position calculation. This is compiled to binary form from de200.dat |
| lwm1_n*x* | LandSea (E,I -) | Land-sea mask data. This parameter is a list of all the files separated with ";". Full file path must be specified at least for the first item in this list. |
| dem30ARC_*_imapp.hdf | DEMfiles (E,I -) | 1km DEM files. This parameter is a list of all the files separated with ";". Full file path must be specified at least for the first item in this list. |
Utcpole.dat is a Earth Motion file containing a record of the Earth's variable rotation with respect to Coordinated Universal Time (UTC). IMAPP requires this information for accurate transformations between inertial and Earth-fixed coordinates. For accurate geolocation the file requires periodic (about once per moth) updates. Recent version and comments regarding this file are available from http://newsroom.gsfc.nasa.gov/sdptoolkit/leaputc.html (or ftp://g0dps01u.ecs.nasa.gov/OTHR or ftp://oceans.gsfc.nasa.gov/COMMON/).
Land-sea mask (files lwm**) are required because standard EOS geolocation product (MOD03) should contain a piece of this data inside. The whole worldwide coverage is divided into six tiles stored in separate files. IMAPPW distribution does not include these files. They may be downloaded separately from http://www.scanex.ru/eosdb/imapp.htm or found inside geolocate.tar.gz file from SSEC's original source distribution. So you should fetch and decompress them (each will take about 150 MB) in the ancillary data directory before starting work. To make the geolocate.exe use these files and process Land-Sea mask you have to uncomment (remove starting ";" sign) or add the LandSea parameter in the imapp.ini and point it to the files you have installed. IMAPPW can also run without this files if you do not require land-sea mask in the product files.
Digital elevation model (DEM) files are required for terrain correction and writing valid height fields in the product files. Total DEM files volume is about 1.8GB. These files are also optional, may be downloaded from ftp://origin.ssec.wisc.edu/pub/IMAPP/MODIS/Level-1/ and should be pointed by DEMfiles parameter to be used during geolocation process.
Geolocation program utilizes only two additional parameters which specify names for input and output files.
MODIS data stream from the Aqua satellite does not contain ephemeris and attitude information. This means that geolocate.exe may not be used in usual mode utilizing MOD01 input as the only data source. Three possible ways out are to...
The last of three main IMAPP programs performs systematic calibration of MODIS earth view data from raw device counts to absolute radiances. As a result standard MOD02 product is generated. Calibration algorithms are described in the book "MODIS Level 1B Algorithm Theoretical Basis Document (ATDB MOD01)". The output product consists of 4 separate files:
| MOD02QKM | Earth view calibrated data at 250m resolution, contains the 250m bands only |
| MOD02HKM | Earth view calibrated data at 500m resolution, contains the 500m bands (3-7). It also contains the 250m bands resampled to 500m. |
| MOD021KM | Earth view at 1km resolution, contains the 1km bands split into two categories: Reflective (8 -19, 26) and Emissive (20-36 except 26). It also contains the 250m and 500m bands resampled to 1km. |
| MOD02OBC | On-board calibrator data extracted from level-1A |
As an input the calibration process utilizes both level-1A and geolocation files generated at the previous processing steps.
Calibration process requires a set of look-up tables (LUT) that are generated by MODIS Characterization Support Team (MCST) from pre-flight and operation time measurements. All required LUTs are supplied with the IMAPP installation package. Parameters referring following calibration Lookup tables are also repeated for Aqua version (with Aqua... prefix, corresponding files are stored in ./ancillary/Aqua subfolder)
| Ancillary file | Parameter | Notes |
|---|---|---|
| Reflective_Lookup_tables_file | ReflTable (E,I+) | Calibration LUTs for reflective bands (1-19,26) |
| Emissive_Lookup_Tables_file | EmisTable (E,I +) | Calibration LUTs for emissive bands (20-25,27-36) |
| QA_Lookup_TAbles_file | QATable (E,I +) |
Following parameters specifies location of output files. If they do not fully defines all files placement common default naming scheme will be used. That is output files will have names with "MOD02**" in place of "MOD01" in input L1a filename.
L1B_EV_250M (E -) - full filename for MOD02QKM product
L1B_EV_500M (E -) - full filename for MOD02QKM product
L1B_EV_1000M (E -) - full filename for MOD02QKM product
L1B_OBC_FILE (E -) - full filename for MOD02OBC product
a template of the output filenames that may help to specify all previous names at once. All four names are generated from this template with addition of product suffixes. If you set this parameter you have not to set L1B_* ones. For example if L1bFile was set to "c:\mod01\12345" then you will get "c:\mod01\12345.MOD02QKM.hdf" etc.
EOS QuickView (EOSQV) program was developed as an add-on to the basic IMAPP programs in order to simplify handling of level-0 data sets. This program incorporates a lot of level-0 and level-1A code and generally provides the same functionality as the unpack.exe does. But it adds a graphical user interface which helps to do the most tasks in point-and-click manner. EOS QuickView also makes it possible to...
EOS QuickView is installed together with other IMAPPW stuff and shares with the unpack.exe parameters defined in the imapp.ini file.
While beeing using EOSQV for the first time you should check validity of working parameters. This can be done with dialogs callable though the Options menu item.
This dialog box lets to define generic processing parameters. Many of them are the same as for the unpack.exe and are shared between the programs. This parameters are stored in the imapp.ini file. Common parameters are stored under the [IMAPP] section and parameters used by the EOSQV only are defined in the [EOSQV] section. So take into account that modification of the common parameters via EOS QuickView GUI may influence the behavior of the unpack.exe.
Short descriptions of the fields available in the General options dialog follow. If the parameter is common to EOSQV and unpack.exe its name is given in parenthesis. Parameters that have a value of file or directory name may be specified either with directly typing the value in the edit box or selecting them with help of a standard dialog boxes that callable via "tree point" buttons.
This dialog is callable via Options|Quicklook... menu item and provides possibility to customize quicklook generation parameters.
EOSQV always generates a quicklook from one MODIS band and draws it using a gray-scale palette. The band that will be used for quicklooks may be selected from the Band combo box. Note that the reflective (1- 19) MODIS bands are not available during operation in the night mode. If a reflective band is in use this will lead to the night mode part of the quicklook picture to be filled with white. So if you are often have a night mode data you'd better to use one of the emissive bands for preview. Otherwise you may use any band that you will find to be the best looking. For example bands 17 and 18 of Terra's MODIS look quite nice.
You can also define an integer scale at which quicklooks will be created. There is a Scale field for this purpose. Current EOSQV version does not support scrolling in the quicklook window. So you should find a scale value that will give quicklook size appropriate for your monitor settings. For screen resolution 1024x786 the scale value of 8 was found to be suitable in the most cases.
Due to the MODIS scanner performs 12-bit quantization a some sort of pixel value transformation must be supplied during conversion to a viewable 8-bit gray image. EOSQV utilizes the simplest method - linear stretch of 4096 levels into 256. Because the real earth view data usually occupy only a part of the available dynamic range this method often gives unacceptable pictures. The EOS QuickView offers two simple ways to solve this problem. The first one is to postcorrect an quicklook picture to enhance it (see Image correction dialog description later in this text). The second option is to adjust parameters of the level reduction conversion itself. The last method can be switched on/off with the Histogram stretch checkbox. If the histogram stretch is used the conversion procedure stretches to 256 levels only a part of input image dynamic range that contains 98% of the histogram. This provides automatic conversion parameters adjustment to the current image. But this method does not always provides good results due to the following reasons. Currently a one-pass procedure is used for the quicklook generation and in order to gather the histogram it utilizes only the first data scan which has not less then 1/3 valid image pixels. Sometimes this leads to poor estimation of the full-image histogram in case when the image has significantly varying brightness (e.g. night at the north). So this feature should be considered as experimental and may be changed in future releases.
This dialog gives possibility to adjust quicklook viewing parameters. The dialog provides two sliders that control imaging parameters which behave like common brightness and contrast. You can play with this parameters in order to get an acceptable picture. Dragging the sliders up increases brightness and contrast of the quicklook and vise versa. The Reset button returns parameters to the "central" values which correspond to 1-to-1 image levels conversion (i.e. no level adjustment). All parameter changes are applied to the currently visible quicklook immediately and will be stored for future use after the OK button click. This dialog may be used even during quicklook generation process.
There are two ways to open an input level-0 file (PDS) in the EOS QuickView. The first (quick) way may be useful if you know the PDS's filename and are going to create and view the quicklook for the whole data set. In this case the menu command File | Open PDS... may be used. After selection of the input PDS file via a standard file open dialog process of a quicklook generation will be started. At this time the application window's size will be adjusted to the expected size of the quicklook and a time rule will be drawn at the right side of the window. The time rule may be quite useful while dealing with level-0 because the most of tasks with such data are performed in terms of start and end times. The rule contains a mark and absolute value for the first full minute time and marks with time offsets relative to this point.
Progress of the processing is indicated in a status bar at the bottom of the program window. The quicklook is being drawn to the window during operation. This also gives a visual hint on the process completeness. Quicklook generation can be interrupted at any time by pressing the ESC button. Regardless of full quicklook was generated or not the PDS file will be treated as open and will become a current opened file.
The second way to select a PDS file is to use a Select PDS dialog available trough File|List PDSs.... menu item. This dialog contains a list of PDS files contained in the specified directory. Besides filenames this list contains also file sizes and start and end times. The directory to be searched for the PDS files may be defined in the edit box at the top of the dialog. After selecting a file in the list the are three operation that can be initiated on this file:
The main function of the EOSQV is an export of a selected range of MODIS level-0 data into a level-1A. The time range of data to be exported can be defined interactively by selecting a region on the quicklook image with mouse. Click and hold the left mouse button and drag the rectangle selection box over the image. Refer to the time rule at the right while selecting the range.
Export operation can be started directly from the Select PDS dialog or with the File|Export... menu command after the PDS file has been opened. As the first step the Export parameters dialog appears in order to ask the user for export parameters. At the top of the dialog the name of the current PDS file is shown. Then a button follows which defines currently selected time range in form "Start Time - End Time = Number Of Scans". This button also lets to overwrite the time region to process. The dialog contains also two controls groups for parameters related to BMP and level-1A exports accordingly. Set the checkbox near the appropriate group caption to select this type of export to be performed. Both export types can be defined simultaneously and done during single processing pass.
For the BMP export operation the following parameters can be defined.
For the level-1A export there is only one main parameter that must be defined - output file name. By default export operation is performed by the EOSQV itself. As an option you can set the Run as separate process checkbox and execute unpack.exe with specified parameters in a separate console window .
IMAPP contains an additional tool L0_scan.exe that helps to check quality of input PDS files and get general information on its content. This program takes two command line arguments: name of a PDS file to gather information off and name of output text file that will contain the program output. The second parameter is optional and if it is omitted all the output will be directed to the console window. The tool reports following information on the PDS file:
The following is an example of output of the L0_scan.exe for PDS file containing one occurrence of the packet stream time inversion near the beginning :
input filename: f:\eos\l0\10090757.pds
------------------- Messages ----------------
1%
Warning: out-of-time-order packet, utc: 2000-10-09T08:01:00.797330Z,
time: 245232068.074309, last_time: 245234165.226363
100%
------------------- Summary ------------------
n_packets: 1335423
n_day_pkts: 1333655
n_night_pkts: 0
n_scans: 445
start_time: 2000-10-09T08:00:48.980222Z
stop_time: 2000-10-09T08:11:44.830111Z
Several interesting sites not mentioned before