• In Versions 5 and 6 you can instead use a graphical interface, the "IDL Development Environment." To enter that, type idlde. IDLDE offers a number of convenience features for the experienced user.

• In Version 7, a more elaborate graphical interface called the "IDL Workbench" has replaced the Development Environment. Again, to invoke the Workbench, enter idlde. Here is a screenshot of an IDL Workbench session. If you want to explore the Workbench approach, try the exercises suggested in Chapter 2 of the version 7 Getting Started manual.

• Another alternative is to use IDLWAVE This is an integrated editor/IDL execution facility based on GNU Emacs. The many powerful convenience features in IDLWAVE are useful mainly to experienced IDL programmers. Here is a screenshot of an IDLWAVE session.

• The descriptions below are for the command-line environment invoked by the idl command and employ "direct graphics" commands (as distinct from the "object-oriented graphics" used to create GUI displays). However, the basic commands entered during an interactive session are the same for all environments (e.g. in the Workbench you would enter these in the Command Line window opposite the IDL> label).

• If you get messages complaining about the "license server" or stating that you have been shifted into the "7 minute demo mode," then there is a problem with the licensing software which authorizes individual users to access the IDL executables. This may be because too many users (at UVa, more than 50) are trying to access IDL (in which case, you just have to wait). Alternatively, there may have been a failure in the license daemon which authorizes you to link to the IDL source code. You will need systems-level help to address the latter difficulties. At UVa, IDL source code is maintained by ITC in directories under /common/rsi.

• If you cannot access certain routines, check to be sure you have defined all the necessary shell variables and aliases to point properly to the various IDL directories. See the "Setup" section at the end of this writeup. An incorrect path is one of the most common sources of problems in IDL sessions.
  If you log onto the UVa Astronomy LINUX network and do not specifically define any IDL environment variables, you will be automatically set up to run the latest version of IDL with proper links to all current software directories, including the Astronomy User's Library, MOUSSE, and ATV.

• If the windows environment does not respond properly, check to be sure you are running X-windows and that the windows parameters have been properly set for the kind of visual display environment you intend to use (e.g. 8-bit pseudocolor, 24-bit truecolor, etc). The LINUX terminal command xdpyinfo will list the supported modes for your display. The IDL command help,/dev will list the current properties of your display assumed by IDL. For more details, see the Image Display section below.

• If you have trouble with the colors on your terminal changing or "flashing" when you move the cursor during an IDL session, see the Image Display section below.

• Your personal "idl directory", if you have one, will not be accessible unless it is the current directory OR it is included in the IDL_PATH shell environment variable. At UVa, this will be automatic if the directory is in your root directory and is named, or aliased to, "idl". (See the environment setup file example below.)

IDL will always execute the special "Startup" file defined in the $IDL_STARTUP environment variable. A standard version of this file must be executed before the MOUSSE routines will run properly. This is done by default for UVa Astronomy users. See the "Setup" section below. You can modify the startup file as you like (e.g. to open special plotting windows or to establish main-level common blocks).
You can also execute personal startup files to further customize your session. These are usually batch files (see "Program Execution" below), executed by typing @[filename].

If you are in cursor mode, you may have to move the cursor to the active window and press mouse buttons to complete the interrupt. On some commands (e.g. array calculations or I/O) an interrupt may require some time to take effect.

During an interactive session, your command line entries are stored in a command recall buffer (as in LINUX tcsh). You can use the "up arrow" ("down arrow") keys to move backwards (forwards) through the command buffer to recall or edit commands. Use EMACS-like editing commands. The default for the command buffer length is 20 commands. (It is useful to set this to a higher number by defining the system variable !EDIT_INPUT=100 in the startup file. This is done by default in the MOUSSE startup files.) Recall/edit is an exceptionally useful means of iteration during an IDL session.
In editing, be careful not to give the ^d command on an empty line, since this will terminate your session!

x = a+b & y = sqrt(x)
A set of &-linked commands on a single line constitute a "micro-program" which can be executed, then edited and re-executed with a couple of keystrokes.

2. HELP

[Up to Contents]

IDL is thoroughly documented in electronic manuals. ITTVIS issues a full set of manuals in PDF format with each license. If the PDF versions have been installed on your computer system, they can be accessed through the LINUX command idlman. The most important manuals are Using IDL, The IDL HandiGuide/Quick Reference, Building IDL Applications/Application Programming, and The IDL Reference Guide. These are accessible on-line from within an IDL session through a Hyperhelp facility. (Note: the titles of some manuals have changed with version number.)
The current introductory IDL guide is called Getting Started with IDL---click for a PDF version. The printed versions of the older IDL Basics manual for V3 and V4 take you through a number of interesting sample applications and provide helpful hints.
The best way to remind yourself quickly of the operation of intrinsic IDL routines is to refer to the IDL Quick Reference. Unfortunately, this "quick" guide has become rather unwieldy, and, at least for a reference on basic IDL functionality, it may be easier to use one of the older versions of the IDL HandiGuide.
You will probably find it helpful to make a hardcopy of the Quick Reference or HandiGuide sections titled "Functional List," "Syntax," "Statements," "Executive Commands,", "Special Characters," "Subscripts," 'Operators," "System Variables," and "Graphics Information." That will total only about 30 pages.

Browser Access to Documentation:
• ITTVIS IDL Documentation Distribution Site

• To access documentation for IDL_5.1 in HTML using a Web browser, click here. and bookmark the page.

• IDL Astronomy User's Library Copy of IDL 6.4 Documentation

• My IDL Resources page

• The IDL Astronomy User's Library Resources page

• The comp.lang.idl-pvwave newsgroup

• David Fanning's IDL Tips & Tricks

• Ray Sterner's Quick Guide to IDL Syntax

• At the LINUX prompt, type idl; the version number is displayed on your terminal at the beginning of each IDL session.

• Check the directory to which the LINUX shell variable $IDL_DIR points.

• Within an IDL session, type print,!version or help,!version,/str to display the information on the version and assumed operating system stored in the system variable !version.

One of the best ways to learn how to write and use IDL programs is simply to inspect existing IDL programs in the AstUseLib directories. You can more them in LINUX, or use the AstUseLib getpro routine to copy them to your local directory. To view public routines during an IDL session, type .run -t [routine name].

• idlhelp or idlman: given from the LINUX prompt, these commands open the PDF versions of the standard IDL manuals (if they are loaded on your system).

• ? : given from within IDL, this invokes the IDL HyperHelp facility. Click on the "Index" tab for a list of all entries (lists general discussions as well as individual routines, the latter listed in all-caps). (Warning: several versions of IDL, e.g. 6.0, employed a more awkward PDF-based help facility. If you are saddled with this, it's better to upgrade.)

• ?[command] : provides HyperHelp information on the given intrinsic IDL command. Does not work for user-supplied routines. Note: no space or quote mark between ? and name.

• [command without arguments] : Most AstUseLib and other user-supplied procedures are coded such that if you enter the name of the routine without arguments, a list of expected arguments is printed on your screen. Does not work for intrinsic IDL routines (but see help,/rou below). Does not work on most functions (as opposed to procedures).
  This is a useful convention to code into your own programs. Use the n_params function to sense the absence of input parameters, as in the following:

    if (n_params(0) eq 0) then begin
           print,'CALLING SEQUENCE: program_name,var1,var2,var3'
           return
         endif
    

• man: The man command without arguments produces a graphics menu display similar to that generated by ? but which provides information on AstUseLib, MOUSSE, and other user-written routines. man is a program distributed by Wayne Landsman from: http://idlastro.gsfc.nasa.gov/ftp/contrib/landsman/help/.
  Note: the topics included are only those "pro" files which have been parsed by a special documentation extraction routine (mk_library_help) and placed in a special help directory. This is usually done by the person who maintains your IDL system.

• man,[command-name-string]: This version of the man command, in which you enter a string containing the name of the command you want information on, prints the information header of the routine in your IDL command window. It works on any properly formatted IDL routine in the IDL_PATH. For instance, if you want information on the AstUseLib FITS_READ command, you would type:

  man,'fits_read

  Note that the argument of man must be an IDL string---i.e. it must be quoted (but the trailing quote mark can be omitted). (Do not enter the .pro suffix for the filename.) If you already know the name of the command of interest, this is faster and more convenient than using the graphics version.
  The man function operates by finding the first file with the name "[command].pro" in the IDL_PATH and listing all those lines in the file between the two lines starting with ";+" and ";-" in the first two columns. It is strongly recommended that you place headers of this type in your own *.pro routines. There is a standard header format adopted by most IDL coders, though you do not have to follow this in your own routines.
  Unfortunately, intrinsic IDL routines cannot be accessed by man.
  The IDL Users Library routine doc_library,[command-name-string] has similar functionality; it also lists the directory path to the file.

• $more [directory/command.pro] : lists any local program. You can define LINUX shell variables named $ATV_DIR, $ASTUSE_LIB, etc, to point to the appropriate directories to avoid having to remember which these are.

• getpro,[command] : writes a copy of any non-proprietary procedure file to the current directory. Does not work on intrinsic IDL software. You can display or edit the procedure to modify its function as desired. If you do not alter the name, and place it in the IDL path ahead of its original location, then your version will be compiled and executed instead of the nominal one. It may be safer to change the name (both the file name and the procedure name within the file) to prevent inadvertent errors.

  Note: one deficiency of current user-supplied procedure documentation is that the dependencies (routines called by a given procedure) are not listed. If you use getpro to obtain and edit a supplied routine to behave differently, this may have unintended consequences for the functioning of other procedures. One method of looking for such interactions would be to grep the ASCII files in the library directories for other references to the routine you intend to change. A quick way to identify what subroutines are called by a given program is to .run the program as the first step in an IDL session. Each subroutine is listed on the screen as it is compiled.

• .run -t [command] : an alternative way of obtaining a listing of a routine. This will compile [command.pro] and simultaneously send a listing to the screen with line numbers appended (same numbers as printed in IDL error statements, so is useful for debugging). To save the listing in a file, use the form: .run -l (filename) [command]. (Warning: if you mistakenly omit the filename here, you may start overwriting the [command] file if it is in your current directory.)

• print,[variable]: display the value of any active variable on your terminal. Works for any variable type, but beware in the case of large arrays(!)
  Formatting: row vectors are printed across the screen, column vectors are printed down the screen.
  [Note: The printf command is used to send output to a selected file or other external unit.]

• help : lists all active variables, their characteristics, compiled programs, information on storage areas, etc.; various options. But does not explain commands. There are many optional keywords for use with help. Examples are given below. To get a full list, type ?help.
  • help,/rou : prints argument list for all compiled routines (other than intrinsic IDL routines). Lists procedures and functions separately.

  • help,/sy : prints current values of all "system variables", which are special variables known to all routines. Names of these begin with a "!", e.g. !dir, !path, etc.

  • help,/rec : prints contents of command recall buffer in reverse order

  • help,/dev : prints parameter settings for current graphics device

  • help,/mem : lists current memory usage

• print,!d or print,!p : print the system variables that control the device display and the plotting environment, respectively. See the IDL manuals for details.

• print,!path : print the current path list of directories searched for *.pro routines. Since many instances of failed or missing software occur because your path is incorrectly set, it is useful to remember to check !path if you run into trouble. If the !path string is too long to print, use the strmid routine to make and print extractions from it.

• $printenv : will list all default shell parameters, including the IDL directory defaults, if defined.

• journal,[filename]: Starts the journal utility, which records in a file all entries you make and most IDL responses. Certain responses (e.g. to help) are not recorded to prevent cluttering up the file.
  Any text you enter on a given line preceded by a semicolon will be ignored by the compiler and included in the journal file as a comment---so you can annotate your session on-line to your heart's content.
  Your journal file will be written and closed when you type journal again or exit your IDL session.
  It is hard to exaggerate the value of keeping IDL journal files for serious work. It is good practice always to use a journal file so that you can check or reproduce what you have done, recover errors, and so forth. If you edit out the blunders and undesirable clutter of your journal files, you will have a handy running record of your work.
  Journal files permit rapid re-creation of an IDL session, in whole or part, or (if the activity was worth saving and repeating) can be edited into the form of a main program, subroutine, or script.
  Warning: the journal routine will overwrite an existing file with the given [filename] without warning. Use unique names for each session. Also, the journal file may be lost in the case of an IDL crash or a system failure (e.g. power outage). For long sessions, you may want to plan for more than one journal file.

3. PROGRAM EXECUTION

[Up to Contents]

NOTE: all IDL main programs, subroutines ("procedures"), and functions are assumed to be in files with the explicit extension '.pro'.

Creating:
To create or modify variables, simply use them in an assignment statement. Memory for each is automatically set aside and expands indefinitely. No pre-declaration of variables is necessary. For details on types of variables in IDL, see "Part II: Components of the IDL Language" in the Application Programming manual. E.g.:
z = 1.0e-8 creates a floating point scalar
c = 3.0d10 creates a double-precision floating point scalar
a = [1,2,3] & a = [a,4,5] creates and then expands an integer vector
testdata = fltarr(512,256) creates a 512x256 2-D floating point array with zero entries.
This array will be 512 elements wide (# of columns) and 256 elements high (# of rows). Note that this column/row assignment convention is reversed from FORTRAN and normal matrix notation but that it corresponds to normal f(x,y) graphical notation. IDL arrays are stored with the first index changing fastest. This allows faster display of images because the elements on each horizontal scan line are stored contiguously.
Note also that (again to conform to graphical standards) the first element of any IDL vector or array is always at index 0 not 1. That is, in the array z=[20,30,40,50], z(0)=20 and z(3)=50. z(4) is undefined.

testdata = findgen(512,256) creates a 512x256 2-D floating point array with each element set to the value of its 1-D subscript (starting at 0)
name = "Maxwell" creates a string
If you need to create a long string (e.g. in format statements), you can define pieces of the string on separate lines and then concatenate them. (E.g. stringtot = string1+string2).

"Structures" in IDL are sets of other variables (an arbitrary mixture) that can be referred to by a single name. They can be very useful in data analysis problems.

The help command gives you a listing of active variables and their properties.

Deleting:
To delete variables (e.g. when you have too much memory in use), use delvar. You can overwrite the contents of any variable simply by using an assignment statement. A fast way to flush arrays is simply to set them equal to a scalar (e.g. a=0). If you use a lot of files, it is also good to prevent file quota errors by occasionally giving a close,/all command.

System variables:
These are special variables, with names prefaced by ! (e.g. !help_path), that are universally known to all IDL routines. About 30 of these are pre-defined and assist with various aspects of program operation, mainly graphics output. You can create your own system variables using the defsysv command. You can change the value of system variables (except "read-only" variables) using assignment statements. To see the current values of all system variables, type help,/sy.

IDL supports a full set of arithmetic, relational, logical, min/max, and matrix operators. Among others:
  + - * / ^ ++ -- MOD > NOT AND EQ GE LT # ..., etc
See Building IDL Applications/Application Programming for a complete description.

IDL provides over 750 intrinsic mathematical and other functions, not to mention the hundreds of others present in the IDL User's Library. For example:
SQRT EXP ALOG ALOG10 SIN ATAN ASINH GAMMA ROTATE RANDOMN
GAUSSFIT INT_3D INTERPOLATE SORT STRMATCH ..., and so forth.
See the IDL Reference Guide for details.

Many of the operators and mathematical functions take vector or array arguments as well as scalars.

.run [name] : compiles the IDL procedure(s) or function(s) in the file [name].pro. If this is a main program, also executes it.
IDL will locate the first file with this name in the IDL path. Beware of using names for your own procedures which are the same as those of intrinsic or supplied library routines (unless you specifically intend to replace those).
Program files can contain more than one IDL program. Simply concatenate programs together. All will be compiled by the .run command and recognized separately in future calls. However, it is not recommended that you combine programs this way. Among other things, you lose the capability of reading internal headers with the man command (see above).

You do not have to give a separate .run command in order to execute a subroutine which is in your path and has a file name identical with its procedure name except for the added '.pro' extension. Simply type the command's name, with all required parameters, and IDL will automatically search the path for such a file and execute it. If the file name does not match the procedure name it contains, then you must .run the file separately.

You do have to use a .run [name] command to execute a main program the first time. To re-execute, you can use .go.
The .run [name] command will compile but not execute a procedure or function file.
You must re-run any command whose procedure file you have edited during your session or the revisions will not take effect.

.con : Continue a program interrupted by a pre-programmed stop command.
.go : Restart a previously compiled main program from beginning.
@[name] : Execute the "batch" file [name].pro containing a list of IDL commands in the same form as they would be entered from the keyboard. Can be used to run a large background job or a user-specific initialization file at the beginning of a session. Perhaps most common is to execute a batch file that duplicates a set of commands derived from an earlier IDL session (e.g. for graphics output). Such a set is called a "script".
Each line is executed separately so that multiple line commands involving do-loops etc. cannot be included. (But subroutines which are called from the batch file can, of course, include such features.)
You can set up common blocks, compile procedures, and so forth in this mode. Text after a semicolon (;) on a given line is ignored by the compiler.

An efficient way to update or execute parts of a script---e.g. a command sequence captured using the journal utility---is to run an editor on the script in one window while executing IDL in another and cutting-and-pasting snatches of code between them. This method is especially useful when you have a large number of options (e.g. doing statistics, histograms, and plots on a data file containing many different variables).

• Main programs: a main program is simply a file of IDL commands which can be compiled and executed by a .run command. Main programs behave just as though you were typing the same commands from the keyboard except that you have full access to multiple line entries, blocks, loops, etc, which can't be used from the terminal. A main program will recognize any variables or routines already active at the main level of your interactive session, and these remain active when the program completes.
  Main program files do not have a special header. Simply start with the first executable statement. A main program file must close with an end statement.
  Main programs cannot be executed by a procedure or function.

• Procedures: IDL "procedures" perform like subroutines in FORTRAN. All inputs and outputs must be specified on the command line that calls the routine (or passed through common blocks or IDL system variables). Procedures can be called from the main level or by any other procedure. Procedure calls look like this (note the absence of parentheses):

  PROCEDURE_NAME,input_parm_1,input_parm_2...output_parm_1,$
  output_parm_2...keyword_1=value1,keyword_2=value2....   

  • Parameters for input/output are separated by commas. If the parameter sequence is too long for a single line, break into several lines using the dollar sign continuation convention (as here). Information can also be passed by common blocks or system variables.

  • Procedures and functions cannot recognize variables active in your interactive session unless they are passed as parameters, in common blocks, or as system variables.

  • Similarly, the main level of IDL cannot recognize variables that are defined in procedures or functions but are not passed through calling parameters, common blocks, or system variables back to the main program. Internal variables vanish on a normal exit from a procedure.
    In the process of debugging procedures, you will want to be running under on_error,0 (which is the default). If an error occurs in a procedure, this will stop execution at the procedure level and allow you to inspect all of its local variables.

  • As in FORTRAN, ordinary parameters must be entered in the specified order. Trailing parameters can be omitted, if there are defaults for them coded into the subroutine.

  • Keywords are parameters that pass information to subroutines, but unlike the standard parameters such as those listed in the preceding example (input_parm_1,...output_parm_2..), they are optional. They do not have to appear in the subroutine call, and if present they can appear in any order. They therefore represent a powerful extension to the rigid subroutine call conventions of other high level languages and are especially useful in the case of complex data processing tasks. Values of keywords are usually determined by assignment statements:

    PROCEDURE_NAME,parm_1,parm_2.....KEYWORD_1=100.,KEYWORD_2='dumbo',...

    Alternatively, in the case of switches, keywords can be set to a value of 1 by using the following syntax:

    PROCEDURE_NAME,parm_1,parm_2...../KEYWORD_1,.....

    The keyword name in a calling statement can be truncated to as short a string as allows the keyword to be uniquely identified.
    The keyword_set(name) function can be used within a subroutine to sense whether a particular keyword was set by the user in the current call to the subroutine.
    IDL supports "keyword inheritance" such that keywords which may not actually be defined in the procedure declaration but which are included in the command that called the procedure can be passed through to other procedures that do recognize them. This adds considerable flexibility to writing interactive software. See the IDL manuals for details.

  • Procedure files must start with a special header line, as follows:

    PRO Name,parm_1,parm_2,...keyword_1=keyword_1...

    and must close with a return and an end statement.

• Functions: IDL "functions" perform like functions in FORTRAN. They return output to a single variable which appears on the left hand side of an assignment statement:

  result = FUNCTION_NAME(parm_1,parm_2...keyword_1=keyword_1...)

  • The parentheses are required, and parameters within them must be separated by commas. As for procedures, only the parameters given in the calling sequence or passed through common blocks or system variables will be known to the function.

  • Don't be misled by the format of the function call. Functions can return a large amount of information. The variable on the left hand side can be an array or structure. The parameters in parentheses can also be used to return output (as in the count parameter of the where function). Information can also be passed by common blocks or system variables.

  • Note the potential confusion which can arise because the format of a function call is similar to an assignment statement involving an array on the right hand side. In IDL V5 or later, the default format for specifying indices in arrays now uses brackets, rather than parentheses (e.g. array[i,j,k]), although V5 still permits the use of parentheses. In older versions of IDL, the way to avoid confusion between arrays and functions is to pre-compile (non-intrinsic) functions using .run before you compile any procedures which use them or attempt to use them interactively. You will get an "Undefined Variable" error message if you neglect to do this.

  • Function files must start with a special header line, as follows:

    FUNCTION Name,parm1,parm2,...

    and must close with a return,[output] and an end statement.

• Add comments to program files and scripts by using a semicolon (;). Any text on a given line following a semicolon is ignored by the compiler. Executable statements can precede the semicolon.

• retall : important utility that returns you to the MAIN program if an error occurs in a subroutine. The default in this case is to remain under control of the subroutine (so that you can debug the routine by examining the current values of its variables, for instance).
  The danger if you do not return to MAIN is that you will perform other tasks (e.g. reading in new data) under control of the subroutine but later exit the routine, leading to the disposal of all such data. Type retall to escape the subroutine.
  An alternative is to set on_error,1 at the beginning of a session, which automatically returns to the MAIN level after an error. on_error,0 sets the default mode (stay in the subroutine).
  You can check where you are by typing help. The first line of the printed information block lists your current location (procedure name and line).

• If using the cursor, remember to place it on the active window (defined by the wset or chan routine) before clicking; errors result otherwise. Return the cursor to the command window when finished with cursor function.

You can do this before or during an interactive IDL session (though you cannot change your IDL path during a session). You must, of course, be sure you have all the subsidiary programs needed by the new software also in your path.

4. IMAGE RETRIEVAL

[Up to Contents]

During an active session, IDL maintains relevant data in random-access memory stored in variables with arbitrary names chosen by the user. The first step in IDL image analysis is normally therefore to read image data files from disk storage into IDL variables in RAM. These variables can be manipulated at will using arithmetic, extraction, compression, expansion, renaming, conversion, and a multitude of other built-in and user-supplied functions. They can be written back to disk as files in various formats.
Note that this is in contrast to IRAF and most other standard astronomy software packages, where there is no intermediate data storage, all manipulation begins and ends with data stored as files, and one must refer to images by their file names at all stages in the analysis process.
(In certain situations, it may be preferable not to have IDL read an entire file into RAM. Here, you can instead use the ASSOC function to transfer only part of a file as it is needed; the data will not be held in RAM.)

IDL can read/write files up to 2.2 GB in size (and longer on some platforms). However, your host computer may have limitations that prevent access to files this large. See the "Files and I/O" chapter of the Building IDL Applications/Application Programming manual for information on handling large files.
Note that any filename must be given as a string in the commands described below. That is, it must be quoted (unless you use a pre-defined string variable). Example: fits_info,'m87_nucleus.fits (the trailing quote mark can be omitted if it is the last entry on a line). Wildcard notation can be used with routines that can take multiple file input (e.g. fits_info,'m87*.fits).

• sd,[directory] or cd,[directory] : change directory to the image directory, like cd in LINUX. The directory name must be a string. sd, without an argument, is equivalent to $pwd.
  Note: a $cd command to the shell will not change your default IDL directory.

• dir: list the contents of the current directory.

• FITS file information utilities. The standard data format for most astronomical data sets today is the Flexible Image Transport System (FITS) format. The IDL Astronomy Users Library contains a large package of routines to manipulate FITS files. Intrinsic IDL supports a variety of other common scientific data formats (see the manuals for details) with similar functionality.
  fitsdir: list (selectable) keywords from the headers of FITS files in the current directory.
  fits_info and fits_help list information concerning the structure of a FITS file (number of extensions, sizes of headers and arrays, etc) without reading the entire file. This can be important to confirm the type of file (image, table, etc.) with which you are dealing. fits_info will operate on a list of files.
  headfits (a function) will read the full header of a FITS file on disk into an IDL variable (without reading the data). You can then display the contents of the header with either the print or hprint commands (the latter intended for FITS headers). headfits will work on compressed files (*.Z or *.gz).
  E.g.: hprint,headfits('m87_nucleus.fits.gz')

• FITS file reading routines. There are several different sets of utilities for accessing FITS files. For a comparison of their advantages and disadvantages, see http://idlastro.gsfc.nasa.gov/fitsio.html. The two most widely used are probably fits_read and mrdfits.
  • fits_read,[filename],[image variable name],[header variable name],... : read a FITS disk file into IDL image and header variables.

    Example: fits_read,'m87_nucleus.fits',m87,hdm87,/noscale

    This command reads the image data in the FITS image file m87_nucleus.fits in the current directory into active variable m87 and the corresponding header section of the file into active string array variable hdm87. m87 will have the characteristics (byte, integer, float, etc.) of the FITS file data.
    The optional noscale keyword used here prevents the (default) automatic scaling that converts the image values to fluxes using calibration parameters in the header. For many purposes, the scaling is a nuisance.

  • fits_read can handle extensions and groups in FITS files. Among the alternatives, fits_read is the best current choice for HST data sets, for instance.

  • The corresponding routine to write a FITS file is fits_write (see below).

• FITS table utilities. There is an entire suite of Astronomy User's Library routines to read, write, or manipulate FITS ASCII or binary tables. For more information, see http://idlastro.gsfc.nasa.gov/ftp/pro/fits_table/aaareadme.txt.
  For instance, to read a FITS file containing a binary table of sources and extract coordinate information:

   
         fits_read,'sources_file.fits',table,hdtable
  
         tbhelp,hdtable     ; List contents of each field 
                            ;   from the header
  
                 ; Extract RA and DEC vectors, based on
                 ;   tbhelp listing:
  
         rad    =  tbget(hdtable,table,1)
         decd   =  tbget(hdtable,table,2)
            

  Alternatively, you could perform the same extraction as follows:

  
          ftab_help,'sources_file.fits'   ; List contents  
  
          ftab_ext,'sources_file.fits','RA','DEC',rad,decd  
       

• FITS header keyword extraction/manipulation: There are many commands, mostly starting with the prefix sx, that permit extraction and manipulation of FITS keywords.
  For instance, to extract and print the name of a target from a FITS image header variable:
  targ = sxpar(hdm87,'TARGNAME') & print,targ.

  To add a history comment to a FITS header variable:
  sxaddhist,'22 AUG 2004: Subtracted mean sky background',hdm87

  To add a new parameter in a FITS header variable or modify an existing one, use sxaddpar.

• SDAS and IRAF files. These file formats, which store data and header segments separately, are now deprecated, but the AstUseLib has read/write routines for them. See strd and irafrd, respectively.

• Other formats: IDL users have produced IDL routines to read most of the other standard image storage formats. See: read_gif, read_jpeg, read_srf, read_tiff, etc. There are corresponding write routines for all of these.
  ITTVIS also supplies an intrinsic file reader that will read most commercially important image file types. Use the command checkim=query_image([filename],info) to determine whether a particular file is in a suitable format. If so, the returned value of checkim will be 1 (0 if not). For details on the file structure, type print,info,/str. Use the read_image function to read the file and its associated color tables, if any (see the help files and the next section for information on color tables).
  Of course, intrinsic IDL also includes utilities to read ASCII and binary files (formatted or not) in non-image formats (see readf, readu, etc.).

• Endian-Conversions: "endian" refers to the storage convention adopted for multiple-byte quantities by your computer's memory management system. If the most significant byte is stored first, this is a "big-endian" convention. If the least significant byte is stored first, this is a "little-endian" convention. As you might expect, manufacturers have not been able to agree on a universal standard for byte order. Therefore, you will likely be faced occasionally with having to convert byte order for data read from files generated on other computers. This is not an issue in reading/writing FITS files. But it could arise in reading other types of binary files written on a different computer architecture. A symptom of an "endian-ness mismatch" is unusually large or small numbers in data where these are not expected.
  IDL includes the swap_endian function, which allows you to change the storage convention for data in your session. It also allows you to identify the convention adopted by your computer, as in the following:

  
  a=3.0e5  ; Define a large number
  b=swap_endian(a,/swap_if_big_endian) & print,b
       ; If b is different from a, your machine is big endian
       ; If b is the same as a, your machine is little endian
  

  If your machine does not match the computer that generated the relevant data file, then use swap_endian to swap the stored data.

5. IMAGE DISPLAY

[Up to Contents]

Hardware considerations impinge more on making image displays than on other aspects of interactive computing. Color monitors utilize three color injectors---red, green, and blue (RGB)---each of which can be adjusted to 256 different intensity levels at a given screen location. In principle, they can therefore display 256^3 = 16.8 million different colors at any point. Until the past eight years or so, however, most computer monitors were capable of displaying only one byte (8 bits, or 256 different levels) of information at a given location. These are known as 8-bit monitors. Users were forced to choose only 256 out of the 16.8 million possible color values for their displays. This kind of limited display environment is called a pseudo-color or indexed-color display. Most IDL (and other scientific image processing) software written up to 2001 assumes pseudo-color displays.
Modern computer monitors, probably including the one you are using, feature 24-bit displays. That means they are capable of utilizing all possible 3-color combinations. IDL image displays using this maximum color palette are known as true-color displays.
Naively, you would expect that anything labeled "true" has got to be better than something labeled "pseudo." And you would be right---if you were interested in processing commercial digital camera images or making glamorous press-release versions of astronomical images. But that's not the main concern of most astronomers. Instead, you will discover that working with displays in pseudo-color (native to an 8-bit monitor or emulated on a 24-bit monitor) is most appropriate and convenient.
The reasons are, first, that the human eye usually cannot distinguish even 256 levels of intensity or color so that the additional color resolution possible in 24-bit displays is rarely of scientific value. (Aesthetics are another matter.) Second, most images of interest to you will be intrinsically monochromatic. In fact, most basic image analysis work in astronomy is based on gray-scale displays. Finally, the use of true-color displays requires the creation of 3-dimensional arrays (three elements are needed at each x,y position to feed the three color injectors), whereas the scientific images you will start from are typically 2-dimensional. Conversion back and forth between 2-D and 3-D adds needless complications.

Accordingly, most of the subsequent discussion is aimed at indexed-color/pseudo-color displays, whether on 8-bit or 24-bit monitors.

The array that is sent to your monitor hardware for display must be a byte array, which means that its element values are always in the range 0-255.
• 24-bit Monitors: On a 24-bit monitor, the displayed array is always a 3-D IDL byte array, one of whose dimensions is always 3. Each index in this dimension codes for one of the three color injectors. The array structure can be im(3,width,height), im(width,3,height), or im(width,height,3). In the last of these forms, the red, green, and blue image planes are stored sequentially. In the first of these forms, the "color-triples" at each pixel are stored together: that is, the first six elements in storage will be r_00,g_00,b_00,r_10,g_10,b_10, where r_00 is the red intensity in the (0,0) pixel, and so forth. The entered values are converted directly into brightnesses of the monitor's 3 color injectors.
  In the case of a color photograph, for instance, a 24 bit monitor can display the full palette of colors that are embedded in the image, and hence 24-bit displays are called "true-color" displays. However, the colors do not necessarily have to correspond to perceived ones. The most widely used true-color image file format today is the JPEG format (though this is not normally employed for scientific data).

• 8-bit Monitors: On an 8-bit monitor, the displayed array is always a 2-D IDL byte array, im(x,y)=n, where n is called the index of the display at pixel (x,y). The entered index value must be converted into brightnesses for the 3 color injectors by using an intermediary known as a color lookup table.
  Color Tables: A color table consists of three byte vectors R,G,B, each 256 elements long, containing values between 0 and 255. If a given pixel contains the index value n, the red, green, and blue injectors at that pixel are set to the values [R(n),G(n),B(n)]. The resulting appearance on your computer screen therefore depends both on the array values and the color table.
  For a "gray-scale" color table, R(n)=G(n)=B(n). If the three vectors contain different entries, a colored display will appear. These colors have no necessary relation to the physical appearance or properties of the object in the image; hence such displays are called "pseudo-color" displays; more generally, they are called "indexed-color" displays. However, there is a direct correspondence between the index values in the image array and the colors which appear on the screen. Although it does not add fundamental information, pseudo-color display of a 1-byte array can be very useful in exploring subtle structures in a complex image, for instance. Pseudo-color displays need not appear "unnatural"; they can closely approximate a true-color display but are inherently limited to only 256 levels of color/intensity. The GIF format stores images in 2-D indexed form with an accompanying color table.
  On a 24-bit monitor, a color table is used in the "emulated" index-color mode set with the device, decomposed=0 command (see next entry). But color tables have no effect in the standard true-color mode described above.

• Indexed-Color Displays on a 24-bit Monitor: Because indexed/pseudo-color displays are more useful in many scientific applications than is true-color, IDL allows you to emulate indexed-color displays on a 24-bit monitor. In this mode, you again must supply a 2-D byte image array and a color lookup table (as for an 8-bit monitor). The display commands will then use the color table to create a 3-D true-color image which will appear on the monitor just as the pseudo-color image would have appeared on an 8-bit monitor. In order to use this mode, you must declare device, decomposed=0 (more details in next section).

1. The first step is to determine which "visual display classes" your hardware can support. At the LINUX prompt, type xdpyinfo. (You must be in X-windows to use this command.) This will print a list of the display configurations possible on your system. The entry "class: Pseudocolor, depth: 8 planes" indicates that your system supports 8-bit pseudo-color displays. The entry "class: TrueColor, depth: 24 planes" indicates your system supports 24-bit true-color displays. Some systems will support both of these.
   On Apple MAC OS-X systems, the standard X-11 package can be set to different visual classes. Start X-windows. Click on X11 ===> Preferences ===> Output. The "Colors" menu gives you a choice of the numbers of colors possible. "256 colors" is pseudo-color mode. "Millions of colors" is true-color mode. (Most X-11 releases support both.) Make your selection. You must quit the X-11 program and restart it for the changes to take effect. You can check your selection by giving the xdpyinfo command within an X-window.

2. Define a visual class for your session. Start IDL. The first command you give that references a window display function or the device utility determines the visual display class that will be used throughout your session. You cannot change classes after that first command. Therefore, it's usually best to put explicit configuration commands using a device call in a startup file that is executed before your interactive session begins. The three relevant forms of the device call are the following:
   • For 8-bit pseudo-color: device,retain=2,pseudo_color=8

   • For 24-bit true-color: device,retain=2,true_color=24,decomposed=1

   • For 24-bit emulated pseudo-color: device,retain=2,true_color=24,decomposed=0

   The retain=2 keyword requests IDL to handle the "backing store" for your display (which holds copies of parts of your display that have been overwritten); this avoids malfunctions that can occur if instead your windows system does the backup.
   These commands must be given before any window is created during your session.
   In the absence of a device command, the default assumed on a 24-bit monitor is true_color=24,decomposed=1.
   To check the current status of your graphics device, type help,/dev. (Note: the emulated pseudo-color mode is indicated by the entry "Graphics pixels: Combined".)
   Although you cannot change the visual class after its first implementation during a given session, in 24-bit mode you can toggle between device,decomposed=1 and device,decomposed=0 as desired.

3. Reserved Colors on 8-bit Monitors: With an 8-bit monitor, the maximum number of colors available for all applications is 256. When IDL is invoked, it will normally be able to obtain only a fraction of these, say 210, for its use from your windows manager because other applications (Firefox, Acrobat, etc) will have already reserved some colors. The number will vary depending on the number of competing applications. The standard image transfer and display routines like tvscl and tvlct take this into account by using the smaller number of color levels but adjusting them to the full dynamic range possible on your terminal display. For instance, the color vectors may be only 210 elements long, but these will range in index value from 0 to 255. The number of colors available is contained in the system variable !d.n_colors or can be displayed with help,/dev.
   "Color Flashing": If the rest of your 8-bit terminal display blinks out or changes color when you move the cursor into an IDL window, then IDL is using a "private color table" which gets applied to your whole screen when the cursor activates it. This probably means IDL is attempting to use more colors than were free in the shared color table. To ameliorate the problem, try this:

                  Exit IDL
                  Restart IDL
                  As the very first two commands, type:
   	           device,retain=2,pseudo_color=8
                      window,0,col=k
   

   ...where k is the minimum number of color levels you think will be acceptable in your displays.
   Color flashing will not occur on a 24-bit monitor even in emulated pseudo-color mode.

The hardest part of displaying images is selecting the range of image values you want to display and then making them distinct on the screen.
The array you send to your display must be a 2-D byte array (pseudo-color) or a 3-D byte array (true-color). The descriptions below are for intrinsic or emulated pseudo-color based on 2-D images and involve the use of color tables. There are various utilities for converting your original array (often floating-point) to the proper format and automatically displaying it with pre-defined color tables.
1. First, you must decide on the size and shape of the display. Depending on the application and the size (in pixels) of your monitor, you may want to compress the original image or extract subarrays from it. In order to inspect the pixel structure of the image, you can expand images so that one original pixel occupies, say, a 10x10 pixel area on your screen. Basic utilities for extraction or enlargement of images are discussed under "Image Inspection & Manipulation" below.

2. Next, you must determine the range of original image values, imin to imax, you wish to display. The human eye cannot normally distinguish 256 levels of either gray scale or color, and astronomical images often contain much more than a 256:1 intensity range. Only rarely would you want to display the whole range of values present in an image. Normally there is a significant sky background or bias/dark current pedestal which needs to be clipped out. Often, the highest values in images are caused by cosmic rays or other artifacts and lie far above scientifically interesting values.
   To determine the appropriate min/max values you could proceed by trial and error, iterating the image display. More objective methods include using the image value histogram; determining the mode of the sky background and its standard deviation; and making plots of sample slices across the image. Or, you could rely on any of several programs in the user libraries.

3. Next, you must decide on the transformation, or rescaling, between image values and display values. Most common is a linear transform function in which the zero point is determined by imin and the slope is determined by imax-imin. The original image value range is divided into 256 equal bins for display. Contrast in the display increases with the slope, which is inversely proportional to imax-imin. Most image display routines (such as tvscl) employ a linear transform.
   Depending on the distribution of interesting image values within the chosen range, a non-linear transform may be more useful. For instance, to increase contrast at lower image values and decrease it at higher values, you might use transforms such as alog10(f), f^0.3, or asinh(f). Segmented linear or step-transforms can also be useful. Whatever special transform you apply to the image, you can then use the standard (linear) routines to display the transformed values.
   Alternatively, in pseudo-color you can use a linear transform of image values but manipulate the color table to achieve non-linear discrimination in the displayed image. In some situations, this may be preferable, but it is usually more complicated than simply changing the mathematical transform.

4. Finally, you must select a color table for the display and apply it. The appearance of images can change dramatically depending on the color table settings. In pseudo-color mode, the color table is stored separately by the hardware and is always ready to be applied to a displayed image. The default color table is a linear gray-scale running from 0 to 255. IDL supplies a total of 41 pre-defined color tables, and there are various tools for modifying these or creating new ones.

Note: ctv and ctvscl are intended for 8-bit pseudo-color or 24-bit emulated pseudo-color displays only. They do not support true-color displays. If you want to display intrinsically 3-D images, you must use tv and tvscl.

• chan,N (similar to intrinsic wset): select window N for display . This becomes the "active" window, i.e. available for I/O. The cursor will only work on the active window. Creates and displays the window if it did not exist before (i.e. substitutes for the window and wshow functions). N can be between 0 and 31. [Although a display window will be automatically opened by any display procedure the first time it is called, you should use the chan procedure first if you want to take advantage of Mousse functionality. It is also a good idea to keep plotting and image display windows separate.]

• cdel,N (similar to wdelete): Delete window n. You can also quit or iconify windows during an IDL session by using the regular window manager functions.

• ctv,image (similar to tv): display image in current window (or window 0 if no other window has been opened) without scaling by transferring image pixel values directly to the window buffer. The image buffer will contain byte(image)---i.e. values between 0 and 255 only---and will "wrap" for values of 256 and higher. Only !d.n_colors different values can be displayed (determined by your monitor). The window size is adjusted to match the original image (in pixels). Among other things, ctv can be useful for locating interesting low contrast features in an image with a large dynamic range.

• ctvscl,image (similar to tvscl): as for ctv except that the buffer values are scaled linearly between the sky value (set to display value 0) and a high point determined by the variance of the fluxes in the image. There is no wrapping. The maximum number of color levels displayed on the screen is !d.n_colors. ctvscl is generally much more useful than ctv; it has a number of optional features you may want to explore.
  If you don't like the defaults, you can specify the maximum and minimum data values by using the optional keywords max and min. E.g.

  ctvscl,m87,min=30,max=8000

  It is easy to iterate displays by recalling the command line and editing the min,max values.
  From this example you can see the advantages of rescaling images to convenient values in the range 0 to a few thousand rather than using actual flux values. To rescale, use commands like m87 = 1.0e15*m87. In fact, if the quality of the display is your only concern, there is no point in retaining actual flux values in your working image set. Just remember not to use the scaled image for computations where the units matter.

  Note: tvscl does not accept min/max keywords and always scales between 0 and the maximum data value in the image. If that is not appropriate, you must independently clip the image before input to tvscl. Because of this, the display following the command ctvscl,image will differ from that of tvscl,image.

The command to change ("load") the stored color table is tvlct,R,G,B, where you must have defined the three vectors beforehand.
To capture the current color table to vectors (e.g. to inspect them or as a basis for revising them), use tvlct,rr,gg,bb,/get.

A set of 41 predefined color tables is supplied with IDL; these can be loaded with the loadct,N command, where N runs from 0 to 40. The default gray-scale is loaded by the command loadct,0.
You can sample and load the supplied tables in a GUI using the intrinsic routine xloadct.
You can examine the correspondence between color and index using David Fanning's cindex routine.

On most 8-bit monitors, the display will automatically update whenever the color table is changed (at a loadct,N command, for instance). However, on 24-bit monitors you must reload the image in order for the change in color table to take effect (even in emulated pseudo-color mode).
A variety of tools is available for adjusting the supplied color tables or creating new ones. These include xloadct, xpalette, stretch, and David Fanning's xcolors.
A quick way to reverse the standard white-on-black color table 0 to the black-on-white sense preferred by astronomers, which is more sensitive to faint features, is the following. This would be useful to code as a procedure or script.

   
              loadct,0
              tvlct,rr,gg,bb,/get
              rr=reverse(rr)
              tvlct,rr,rr,rr    

• To display logarithmically stretched images: ctvscl,alog10(image). Obviously, you need to insure that image does not contain zero or negative numbers. A quick way to truncate low values is to use the syntax alog10(image > k). where k is a scalar. The notation (x > y) means "the maximum of the pair x,y."

• Use a pre-defined lookup table to transform image values, e.g. to their logarithms. Faster than actually converting values using the alog function. E.g.: for integer image arrays with values in the range 0 to 1000, try this:

    
              t=findgen(1000)  & t(0) = 1
              quiklog=alog10(t)
  
              ctvscl,quiklog(image)   

  Note: In IDL, the form array1(array2) returns the values of array1 in the elements specified by the values in array2. array2 must be an integer or longword integer array.

• cimage=tvrd([arguments]): Copy the contents of the active window image buffer into the variable cimage. Be sure the desired window is active (e.g. by using chan,N) and that the window is fully visible on your terminal screen before calling tvrd.
  tvrd is useful in preserving the appearance of an display after you have manipulated the input image or color table or annotated the display; it is a basic tool in routines that make hardcopies of graphics displays. However, this is a case where behavior is determined by the visual class of your graphics display.
  8-bit pseudo-color: The window buffer contains a 2-D byte image, which is copied by the command cimage=tvrd(). Note the empty parentheses (if you wish to copy the whole window). In this case, cimage is the 2-D byte-scaled image output by the last call to the tv or tvscl routines. If you also save the current color table with tvlct,rr,gg,bb,/get, you can later reproduce the appearance of the display with the commands

              tv,cimage
              tvlct,rr,gg,bb  

  You could make a permanent copy of the display in a GIF file with the command write_gif,'copy.gif',cimage,rr,gg,bb. With a gray-scale color table, the array c2image=rr(cimage) will reproduce what you saw on the screen.
  24-bit emulated pseudo-color or true-color: The window buffer contains a 3-D image, which is copied by the command cimage=tvrd(true=N). Here, the true keyword (values 1-3) determines over which dimension in the resulting array color will be interleaved. The appearance of the display can be reproduced with the command tv,cimage,true=N. You could make a permanent copy of the display in a JPEG file with the command write_jpeg,'copy.jpg',cimage,true=N. Because JPEG is a "lossy" format, the copy will not be exact; if you need an exact version (that could be read back into a later IDL session), you could store in TIFF format.
  In 24-bit decomposed=0 mode, the color tables were used to create the 3-D display image, but you cannot "back them out" of cimage in order to recover the 2-D byte version that was created by the original tvscl command. You can, however, create a 2-D image that approximates the original version as follows: newver=color_quan(cimage,N,rr,gg,bb). Here, rr,gg,bb represent the output color table needed for a proper display of the 2-D newver array. You can redisplay the new image or save it and the color table in GIF format.
  David Fanning's tvread is a more elaborate, device-independent version of the tvrd function for 8- and 24-bit monitors and optionally writes various types of output files.

• blink,windowlist : blink between windows. Windowlist is a vector containing numbers of windows to blink (in order). E.g. blink,[0,2,1]. To compare different exposures of the same field (e.g. two different filters), you will want to adjust the scaling of the two independently with ctvscl before calling blink. To terminate, you must type in the command window, which may require you to use the window manager to bring it up. Blink rate is adjustable.

• tvbox...; tvcircle...; tvellipse...; etc : draw various shapes in active window, e.g. to locate point sources identified by other routines. To remove these, you must re-load the image in the window. Adjust the color or darkness of the overlays by using the color keyword.

• xyouts... : write text to active window. A variety of fonts is available. To remove previously-written text, you must re-load the window.

• curs,type : select graphics cursor type

• To read the cursor position on an active window: use the cursor routine.

• set_plot,[device-name] : allows you to change the current output device. Normally, you will be using X-windows graphical output, for which the command is set_plot,'x. (This will be set by default when you begin your IDL session.) The only other output device type you would commonly deal with is a PostScript file. You can switch output to a PostScript file by typing set_plot,'ps . Return to X-windows by typing set_plot,'x . (The chan command automatically returns you to the X-windows default.) You can determine the currently assumed device parameters by typing help,/dev. set_plot supports a number of other devices as well (see the IDL manuals).

• ATV is an IDL GUI-based image display program written by Aaron Barth which combines much of the functionality just described with many of the image inspection capabilities described in the next section. It is intended for astronomical imaging, and its appearance and behavior is similar to SAOimage (example here).

• ATV is a quick and easy way of inspecting images (brightnesses, x or y profiles, morphology, etc.), doing rough analysis like getting FWHM's and aperture photometry of sources, and extracting RA,DEC positional information from headers. It is oriented toward optical/IR images but will work on any kind of 2-D image data.

• ATV is the best place to start if you want to explore GUI-based IDL displays for astronomy. The code is publicly available, and you can readily adjust default behavior if you so wish.

• At UVa, ATV software is kept in /astro/idl/atv and is included by default in your IDL path.

• To display an existing image array in ATV, simply type atv,image

• To display an image directly from a FITS file on disk, type atv,[filename.fits] or use the "File" pull-down menu.

• ATV allows zooming and roaming around images. (It automatically compresses or extracts from images too large for the screen.) You can adjust the scaling of the image, e.g. with the "Min/Max" entry boxes, and select between linear, log, or asinh scaling. You can interactively adjust the zeropoint and contrast in the color tables (set MouseMode to "Color" and drag the mouse over the image holding down the left-hand mouse button).

• Using single keystrokes, you can display a row plot (r); column plot (c); surface plot (s); contour plot (t); a histogram (h); or local image statistics (i). You can also quickly make an extraction of a spectrum from a spectral image (x).

• The p command does aperture photometry (DAOPHOT-style, with background subtraction) on a selected compact source, with optional display of the radial profile of the source. Photometry, including FWHM's, can be saved to a file.

• Image displays in ATV can be readily stored in PostScript, FITS, JPEG, TIFF, and PNG output file formats. You can label the display or add overplots before output.

• ATV Links (UCI)
  General info
  Instructions on how to use ATV

Eventually, you may want to read more than is here about image display techniques, though I recommend that new users just plunge ahead and experience success and failure on their own first. A good introduction to color displays in IDL is David Fanning's "Working with Color" writeup. Also see his "Color Tips" page. The basic ITTVIS introduction is Chapter 5 "Graphic Display Essentials" in the Using IDL manual. More details on image displays and use of color can be found in the on-line help system under "IDL Tutorials/IDL Display Concepts"; the examples mostly involve full-color image formats (e.g. JPEG, TIFF, PNG). The v7.0 Image Processing manual covers image transformation techniques of interest more in geoscience and medicine than astronomy. For general background on specifically astronomical image processing, see T. A. Rector et al., "Image-Processing Techniques for the Creation of Presentation-Quality Astronomical Images," AJ, 133, 598, 2007; and R. Lupton et al., "Preparing Red-Green-Blue Images from CCD Data," PASP, 116, 133, 2004.

6. IMAGE INSPECTION & MANIPULATION

[Up to Contents]

• print,max(a); print,min(a); print,minmax(a); print,mean(a); print,variance(a); print,median(a); print,moment(a): print various statistical properties of image array. (moment prints first four moments.)

• curval,image : read off image values in each pixel interactively with cursor in current window. To convert to fluxes, RA, DEC using information from the image header, use curval,hdimage,image, where hdimage is the associated header variable. The intrinsic IDL routine for reading off image values is rdpix.

• tvlist,image : prints a matrix of image values in vicinity of cursor location in current window. Options to write to file. imlist is similar, but coordinates are typed in (does not use the window/cursor).

• zoom : displays an enlarged segment of the image in the current window centered on the cursor. Middle mouse button changes zoom factor. Right button exits. Works by copying the currently displayed window buffer, not original image values, and the current version works only for pseudo-color displays. Zoomed window vanishes on exit. Keyword /continuous can be used to follow the motion of the cursor, but you may want to restrict the region displayed, depending on the speed of your computer.

• To extract and examine a subarray, you can use standard IDL subscript notation. E.g.

       part=image(100:200,150:250)

  extracts a 101x101 subarray from the image. You can interactively determine the edges of an area of interest using curval or cursor.

• To enlarge (or compress) an array: new=rebin(image,N,M). N and M must be integer multiples of the current dimensions of image. Compression is often needed with images which are too large to fit on your monitor (typically limited to about 1000x900). Packages like ATV and SAOIMAGE do this compression automatically.
  For magnification, add the keyword /sample if you want to preserve the original pixel structure. This substitutes nearest neighbor sampling for the bilinear interpolation which is the default.
  To regrid an image to non-integer multiples of its original format, use frebin or congrid If you intend to make flux measures from the compressed or expanded images, be sure to scale the result so that flux is conserved in a given region of the original image. For example, if you use rebin to compress an MxM image to an NxN image using sample=0, each pixel in the resulting image will contain the average of the corresponding pixels in the original, so that the total flux is a factor of k² smaller than in the original, where k=M/N.

• Image smoothing is often useful in dealing with low contrast or noisy data. Various options are available in IDL; most widely used are smooth and median, which yield the boxcar average and median values, respectively.
  To display an image smoothed by a 5x5 boxcar in one line: ctvscl,smooth(image,5).

• To plot an entire column: plot,image(N,*); to plot values between rows r1 and r2 in a column: plot,image(N,r1:r2).

• To plot a row: plot,image(*,N).

• To interactively select & plot rows or columns with the cursor: try profiles.

• To plot the mean of 5 adjacent columns: plot,avg(image(c0:c4,*),0). For 5 adjacent rows: plot,avg(image(*,r0:r4),1).

• To extract an image slice with arbitrary start and end points (selected by the cursor from the active window and marked with a "rubber band"): slice=profile(image). You can then plot the extraction.

• To estimate the background level, use sky.
  sky works by estimating the mode of the image values. It therefore assumes there are many more pixels near the sky background level than there are source pixels. It will not work well in the case of very low sky backgrounds where the sky histogram is not smoothly continuous.

• surface,image: produces surface plot (projected 3-D) of image. See the IDL manuals for many enhancements available in surface plotting (e.g. shade_surface).

• plothist,image: evaluate histogram of pixel values in the image and plot the result. The default histogram runs from the minimum to the maximum of the array with a bin size of 1.0 unit. Beware when the array contains erroneous values which are very large or very small. By default it is assumed that the array contains integer values. Optional parameters adjust cutoffs and binning. The histogram function evokes strong emotions; to delve more deeply, see "Histogram: The Breathless Horror and Disgust", a tutorial by J. D. Smith.

• contour,image: plot a contour diagram. To specify the contour levels, use the levels keyword. E.g. to plot 10 contour levels at values of 25, 50, 100, 200...:

  
                     nup = findgen(10)
                     clev = 25.*2^nup
                     contour,image,levels=clev 

  For an example of placing a contour plot over an image display, see the header of Liam Gumley's imdisp routine.

• Flagging values in given range: use the where function. E.g.: to highlight pixels containing a -666 flag:

  
  
         tx=image          ; create temporary copy (unless you don't 
                           ;   mind corrupting the original array)
         finder=where(tx eq -666) ; create a list of relevant pixels
         tx(finder)=10000  ; highlight them (we assume normal tx values
                           ;   are << 10000)
         ctvscl,tx,min=0,max=9000  ; flagged pixels will now stand out
     

• To convert coordinates between decimal and sexigesimal form, use sixty [decimal to sexigesimal], ten [sexigesimal to decimal], or radec [RA and DEC from degrees to sexigesimal]. For instance:
  radegrees=15.0*ten(22,30,17.5)

• ATV includes the abilities to zoom an image and to roam around parts of an image too large for direct display. It quickly produces a variety of informational plots (see description above).

• You can operate on image arrays using any IDL routine which accepts 2-D inputs, including the full range of standard arithmetic and other mathematical functions. Most such routines are array-oriented and do not require you to worry about do-loops or element structure in arrays.

• Most IDL mathematical functions behave sensibly and give you the results you intuitively expect. E.g. if a and b are arrays with the same dimensions, then c = a*b produces an array with the same dimensions in which each element is the product of the corresponding elements in a and b. (There are separate operators--#,##---for other types of matrix multiplication.)

• Many kinds of image manipulation can be executed with one-line commands in IDL. E.g.:
  Edge detection: ctvscl, a-shift(a,1,1)
  Unsharp masking: ctvscl, a-smooth(a,k) displays the difference between the original image and a boxcar-smoothed version with a smoothing length of k pixels. This is fast, but the smoothed image includes the effects of any sharp structures (like stars). Better, though slower to execute, is ctvscl, a-median(a,k).

• Changes are, of course, made to the temporary data sets stored in RAM. The original files from which the data were transferred into your IDL session are not affected (unless you call special file manipulation routines). However, this also means that permanent versions of the modified images are not kept at the end of your IDL session unless you deliberately save the dataset or write new output files (see next section).

• If you want to maintain the applicability of the image header to the manipulated image, then you must update the header after each change. You can add comments to the existing header by using the sxaddhist routine. Changes which don't affect the keywords included in the header (e.g. sky subtraction, excision of foreground stars) can be simply documented this way. However, changes in the flux scale or, especially, the image format (through extraction, rotation, rebinning, etc) usually require that the keywords be changed in order that later routines will function properly. A special set of AstUseLib routines, including hextract, hrot, hastrom, hrebin, etc. will do standard manipulations on images and image headers simultaneously. Astrometric information, for instance, will be preserved.

You can instantly create test images with commands like test=findgen(512,512).
Experiment with display/cursor routines using test images containing "hot pixels," e.g. test=fltarr(512,512)+1000. & test(150,250)=20000., steps, gradients, and so forth.
To create a 16-level "test pattern" of stepped, uniform 100x100 subimages with pixel values running from 0 to 240:

patt=bytarr(400,400)
q=bytarr(100,100)+16
for i=0,3 do for j=0,3 do patt(i*100,j*100)=(i+4*j)*q
ctv,patt

Synthetic point sources can be created by placing "hot pixels" where desired and then convol-ving the result with a point spread function.
IDL's random number generators can be very useful in such applications. For instance, to add Gaussian photon noise to an array containing predicted photon counts:

b=size(array)
noise=sqrt(array)*randomn(seed,b(1),b(2))
noisyarray=array+noise

(Array values per pixel are assumed to be larger than ~30 counts. Otherwise use the poisson keyword to simulate photon noise.)

7. IMAGE STORAGE

[Up to Contents]

• FITS files: To write a FITS file from IDL variables, use fits_write.
  fits_write,[filename],[image variable],[header variable]: Write images to disk in FITS format. The resulting file will have the name [filename].fits.
  If the images have been manipulated and you did not employ the AstUseLib header-tracking commands such as hextract, hrot, hastrom, hrebin, and so forth, then you must create a new header (use mkhdr) or update the existing one to properly match the stored image. fits_write will optionally produce a simple header, or you can use the sx* commands, such as sxaddhist or sxaddpar to do this. It is particularly important to be sure that the parameters BITPIX, NAXIS*, and DATATYPE are properly entered. fits_write writes to the current directory.

• SDAS or IRAF files: Use stwrt and irafwrt, respectively. These routines each produce two output files (*.hhh and *.hhd for SDAS; *.imh and *.pix for IRAF). File transfer between FITS, SDAS, and IRAF formats can be troublesome, since not all modes have been carefully tested. This is true in both IDL and IRAF/STSDAS. Always check results to be sure.

• Other formats: As mentioned above, other popular, but not specifically astronomical, file formats are supported by IDL. See write_gif, write_jpeg, write_png, write_tiff, etc. An alternative is write_image, an intrinsic IDL procedure that also writes output files in such formats. IDL also supports writing MPEG animations.
  See the description above on how to use tvrd to copy the contents of a displayed image window to an IDL variable in preparation for writing to a file.
  GIF format is a good way to store line graphics or processed images (in 2-D format) with special color tables embedded. JPEG, which stores images in 3-D format, is a natural means to save true-color images, though the compression algorithm used to reduce the file size may compromise quality (this can be adjusted). GIF and JPEG are the most widely used default formats for Internet browsers. Unfortunately, neither of these, nor the other popular commercial formats, provide a way to save header information.

• save... : The intrinsic IDL save command will save variables and programs from the current session in a specially formatted binary file. All this material can be restored in a later IDL session with a single command. Delete all redundant or otherwise uninteresting variables (especially large arrays) before calling save. Review the save keywords before use. On a LINUX system, the output file will be in a universal XDR format which can be ported to non-LINUX computers. Use the restore command to read a save file.
  save can be very useful for storage of intermediate results but is not recommended for permanent records. Why? For one thing, it's too easy to forget what all the variables mean; if you take the time to write FITS or ASCII files, you are more likely to document the work. For another, save encourages soaking up mass storage with redundant copies of arrays which may be only slightly changed from their original values. It is more efficient to create a script which makes relatively simple changes to images rather than to store the intermediate versions. Finally, the restoration of save files depends on the availability of IDL---not guaranteed if you move somewhere else. Standard IDL FITS or ASCII file-writing routines are preferable for permanent data.
  The default name of the output file is idl.dat, and this file name will be overwritten on the next save. Best to change to something informative like ngc1068.sav.

8. IMAGE PHOTOMETRY

[Up to Contents]

• Native IDL lends itself to rectangular aperture photometry through simple commands such as mean=total(array)/n_elements(array), where array is a piece of a larger image (defined, for example, using the simple index notation: array=bigarray(x1:x2,y1:y2)). Refined versions of this simplest approach may be found in avg.

• Point Source Photometry: The AstUseLib contains an IDL version of the DAOPHOT 1987 FORTRAN release, described at http://idlastro.gsfc.nasa.gov/contents.html#C2. This performs aperture photometry and PSF-fitting photometry on point sources. It is functionally very similar to DAOPHOT-87 but offers the added versatility of the IDL interactive environment. The basic routines find and aper are especially useful. The IDL source code is available. These routines do not include the improved features of the DAOPHOT 1991 or later releases. To do PSF-fitting deconvolution of blended images, you can try the IDL package StarFinder or use the more recent stand-alone DAOPHOT or DOPHOT releases. The basic IDL routines remain very useful in preliminary assessment of data frames, prior to setting loose the mechanical scroungers, and in analyzing the results. (The output files of standard photometry programs can easily be read back into IDL.) IDL is also an excellent way to create synthetic data sets with known properties on which to verify photometry package operation.

• Surface Photometry: A number of basic routines such as dist_circle and dist_ellipse are available to support surface photometry, but there are no "official" AstUseLib surface photometry programs. Several individual users, including RWO, have their own routines which others may use on an "at your own risk" basis.

• The ATV image viewer contains an interactive circular aperture photometry utility, derived from DAOPHOT, which is very useful for exploring fluxes, FWHM's, and backgrounds of compact sources.

• A number of other user-written photometry packages, mostly for point sources, are available through the AstUseLib site.

9. ASCII FILES

[Up to Contents]

The best way to save small data sets (e.g. photometry output) is in the form of ASCII files, since these are easily edited and transported. Intrinsic IDL supports reading and writing ASCII files; see the IDL manuals. A sample script to write a file containing target names, coordinates, and brightnesses might look like the following:

       get_lun,unit
       openw,unit,'OutputFile'
       form='(a15,3x,f9.5,3x,f9.5,3x,f6.2)'
       for i=0,numtarg-1 do printf,unit,format=form,$
               targid(i),radeg(i),decdeg(i),vmag(i)
       close,unit
 

Note: you can check the output formatting by doing a test print to your terminal, using statements like:

       form='(a15,3x,f9.5,3x,f9.5,3x,f6.2)'
       for i=0,numtarg-1 do print,format=form,$
               targid(i),radeg(i),decdeg(i),vmag(i)
    

readf is the corresponding intrinsic IDL routine to read formatted ASCII files.

10. DATABASE ACCESS

[Up to Contents]

   dbopen,'[data base name]
             dbhelp,1
             dbprint,[10,100,1000],'*

11. PLOTS

[Up to Contents]

The basic IDL commands for making plots are plot, for creating a new plot, and oplot, for overplotting on an existing plot. Plots can be made to a terminal graphics window or to a variety of external devices.

This, however, is only the tip of an immense iceberg. IDL contains many options for making plots---so many, in fact, that the hardest part of the job can be keeping track of the multiplicity of optional parameters. Options in the form of keywords can be specified in the calls to the plotting functions or they can be invoked in the form of system variables, such as !p.title, which will apply to all later plot calls until changed.

• Standard plotting and graphics keywords are explained in Appendix B of the v7.0 IDL Reference Guide or on-line by typing ?plot and then selecting "Graphics Keywords."

• The most frequently used plotting keywords and system variables follow. All are optional. Examples of use are given below.
  psym or !p.psym: determines symbol type. A value of 0 (the default) produces a solid line with no discrete point symbols; values 1 through 10 select other types of (unconnected) symbols. 8 indicates that the user has defined a special symbol using the usersym or plotsym procedures.
  xrange and yrange: 2-element vectors giving the minimum and maximum for each axis. If not defined, autoscaling will occur. Corresponding system variables are !x.range and !y.range.
  linestyle: selects style of line drawn (solid, dotted, dash-dot, etc) if psym=0.
  xstyle and ystyle: set axis options (e.g. exact range rather than rounded off)
  !p.title,!x.title, and !y.title: strings for the plot title (centered over upper x-axis), x-axis title, y-axis title.

The IDL defaults are not as "nice" as those in SUPERMONGO, for example. However, you can quickly customize to obtain as sophisticated a plotting style as you like. All the functionality of SUPERMONGO and other astronomical graphics packages is inherent in IDL. Many of the 2-D and 3-D graphics routines are illustrated in the IDL Demos which come with the system.

Color tables for plots:

The plot commands accept the keywords background, which sets the background color of a plot, and color, which determines the color used for lines and symbols. By default on a pseudo-color X-windows display, background is set to index 0 and color is set to index 255. These therefore display the "bottom" and "top" of your current color table, respectively. [Color tables are explained under Image Display.] The actual appearance of your plot will then depend on what color table you have loaded (e.g. with loadct,N). The default table (N=0) produces a black background and white symbols. Your display will change if you use a different color table or plot/background indices. You can create a special set of color tables to produce a defined set of, say, 10 standard colors for your plots.
Note: it's easiest to use pseudo-color or emulated pseudo-color (device,decomposed=0) for plots. In 24-bit true-color (device,decomposed=1, you must specify colors as longword integers.

Sample plotting scripts for displaying plots on your terminal follow. These involve a mixture of intrinsic IDL and AstUseLib routines:

1. Plot galaxy surface brightness data (in magnitudes) with error bars versus radius to the one-quarter power. Use discrete symbols (not a connected line). On this plot, a galaxy with a standard "de Vaucouleurs" brightness profile will produce a straight line.
   Assume that the vectors sb, rads, and sberr already exist, with sb and sberr in units of ergs/s/cm^2/Angstrom/arcsec^2 and rads in units of arcsec. Assume that because of bad data points you must truncate the plot to eliminate the first 3 entries and those after the 20th.

    
   
   mags=-2.5*alog10(sb(3:19)) -21.1   ; Convert SB to magnitudes per arcsec^2
                                      ; in the STMAG system, ignoring bad data.
                                      ; Assumes all SB entries are > 0. 
                                    
   
   r25=rads(3:19)^0.25                ; Compute fourth root of radius vector
   
   magerr=1.086*sberr(3:19)/sb(3:19)   ; Convert uncertainties 
                                       ;   in SB to magnitudes
   
   !y.range=[25,18]               ; Set a non-default y-axis range, in
                                  ; this case the magnitude scale has 
                                  ; smaller (brighter) values higher
                                  ; on the y-axis
   
                                  ; Make the titles
   
   !p.title='Sample Surface Brightness Profile"
   !x.title='Radius(arcsec)^0.25'
   !y.title='Surface Bright (mags/arcsec^2)'
   
   plotsym,4,1.5,/fill            ; Choose filled triangles for plotting symbol,
                                  ;   50% larger than the default
   
   ploterror,r25,mags,magerr,psym=8
                                 ; Make plot on terminal screen with error bars; 
                                 ; The keyword psym selects the plotting symbol;
                                 ; Psym=8 specifies a user-created symbol, which
                                 ;    in this case was defined by plotsym.
                                 ;    No connecting lines between points.
                                 ; If psym were omitted or set to 0, no
                                 ;    symbols would be plotted and the points
                                 ;    would be connected by straight lines.
                                 ; The plot will appear in the active window,  
                                 ;    or in Window 0 if no others are open.
   
    

2. Make a contour plot of a smoothed image.

    
   chan,3                              ; Open plotting window
   
   !x.title='X'                        ; Make titles
   !y.title='Y'
   !p.title=' Contours for Image'
   
   square                              ; Set aspect ratio to make a square plot
                                       ; (Note that you must also give this command
                                       ; after the PostScript device is called
                                       ; when making hardcopies.)
   
   smooth_one=smooth(image,5)          ; Smooth the image by a 5 pixel boxcar
   
   clev=[10,20,40,80,160]              ; Define trial contour levels--assume
                                       ;     these values span range of interest
   
   contour,smooth_one,levels=clev       ; Do fast test of contour plot.  Check &
                                        ;   iterate clev for best appearance
    
   contour,smooth_one,levels=clev,/follow  ; Do more accurate (slow) plot,
                                           ;  with labels
    

3. Read, sort, and plot data from an ASCII file; make & display trial polynomial fits

   Assume x is a vector containing values in the range 0 to 5 and that y is the corresponding dependent variable. Assume that these are to be read from an ASCII file named xy.dat, which contains x and y in separate columns. xy.dat can contain an initial explanatory section and other separator headers, as long as none of these contain only one or two floating point numbers (since readcol will mistake those for data lines). By default, the readcol routine will read in the numerical x,y data ignoring any line containing alphabetic characters. No labels are put on the plot in this example.

    
   readcol,'xy.dat',xx,yy         ; Read data from the file.  Note that format 
                                  ;    statements are not required
   
   index=sort(xx)                 ; Sort the arrays in order of increasing x 
   x=xx(index)
   y=yy(index)
   
   chan,1                         ; Open Window 1 for plotting
   
   plot,x,y,psym=5                ; Plot the data with open triangles
   
   quadcoeff=poly_fit(x,y,2)      ; Derive coefficients for best quadratic
                                  ;    polynomial fit
   print,quadcoeff                ; Print these out (optional) [Note: 
                                  ;    quadcoeff is a vector]
   
   tx=findgen(101)/20.            ; Create independent variable vector for fitted
                                  ;    values (uniform interval of 0.05 x units)
   
   quadtesty=poly(tx,quadcoeff)   ; Create the fitted quadratic values
   
   oplot,tx,quadtesty,psym=1      ; Overplot the quadratic fit with plus signs
   
   cubcoeff=poly_fit(x,y,3)       ; Derive coefficients for cubic fit
   cubtesty=poly(tx,cubcoeff)     ; Create the fitted cubic values
   
   oplot,tx,cubtesty,psym=3       ; Overplot the cubic values with small dots
   
                                  ; [Assume the quadratic fit was adequate]
   
   delta=y-poly(x,quadcoeff)      ; Compute the difference between y data and
                                  ;    the best quadratic fit 
   
   nix=where(abs(delta) gt 2.)     ; Locate those y values which are more
                                   ;    than 2 units from the best fit
   
   weight=fltarr(n_elements(x))+1. ; Create a weight vector corresponding to
                                   ;     x with unit entries
   
   weight(nix)=0.0                 ; Give deviant points zero weight
   
   newquadcoeff=polyfitw(x,y,weight,2) ; Derive improved quadratic coeff.
   
   final=poly(tx,newquadcoeff)         ; Create improved fit values
   
   chan,2                         ; Open Window 2 for final, clean plot.
                                  ; (Window 1 is retained for comparison.)
   
   plot,x,y,psym=5,xrange=[1,3.5] ; Plot the data with open triangles in
                                  ;    Window 2.  
                                  ; Assume interest is limited to only 
   		 	       ;    part of data x-range.
   
   oplot,tx,final,psym=0          ; Overplot final fit with solid line
    

   4. Distinguish two samples in a histogram plot
   Suppose you have measurements of a quantity val for two samples of objects with a similar range of values and you want to compare the histograms of the two in a single plot. A simple way to do this in IDL is to create the union of the two datasets for a first plot and then overplot a shaded histogram of the second set. If the data consist of two vectors, val1 and val2,

   
        allval=[val1,val2]
        plothist,allval,bin=0.2,xrange=[0,8]
        plothist,val2,bin=0.2,/over,/fill,fcolor=200
   

   In the resulting plot, the val1 entries will be unshaded, while the val2 entries will be shaded with the fill color value 200 (the on-screen appearance will be determined by the pre-loaded color table).
   Similarly, in the case of a subset of data values that are to be distinguished by some second parameter:

   
        plothist,val1,bin=0.2,xrange=[0,8]
        good=where(param ge 0) 
        plothist,val1(good),bin=0.2,/over,/fill,fcolor=200
   

   Here, the vector param must have the same length as val1, and we assume the subset of interest is defined by having a non-negative value of param.

12. GRAPHICS HARDCOPIES

[Up to Contents]

The most common method of obtaining hardcopies or permanent storage of graphics output (plots or images) is to use PostScript files, since these can be printed on most laser printers. PostScript files can be later edited and reformatted, though special (non-IDL) programs are needed.

You should always experiment on the terminal screen with your plot format before dumping to an output file. It is easy to do this by working out the set of commands you want by plotting to the screen, then typing set_plot,'ps (in the case of PostScript output) and repeating the commands using the command recall buffer.

 
        set_plot,'x  :  Send output to X Windows (default) 
        set_plot,'ps :  Send output to the PostScript file "idl.ps"

The subsequent commands for sending data to the PS file are (mostly) the same as for putting data on your monitor screen, since monitors and PS files are interchangeable output devices for IDL.

You can always check the properties of the current graphics output device by typing help,/dev. You can change these defaults by using the device command. The hardware/software interfaces are sometimes non-trivial, and you will want to plan for a significant learning curve in doing things which are not "vanilla." Before sending large jobs to printers, vendors, etc., be sure to check the files using LINUX ghostview, xv, or other screen display programs.

Here are some graphics output methods for common situations:

• To make a PostScript hardcopy of a plot.
  1. Type set_plot,'ps to send output to "idl.ps"

  2. Optionally, make adjustments to the plot size, aspect ratio, orientation, lettering fonts, etc. using the device command

  3. Give the set of plotting commands here---just as for a screen plot; all system variables will still be in force

  4. Type device,/close to close the file idl.ps. (Note: the PostScript file is not actually written to disk until the close command is given.)

  5. Type $lp idl.ps to print the file on the default printer.

  6. The next plotting command will overwrite idl.ps. If you want to save it, you must change its name. E.g. $mv idl.ps bossplot.ps.

  7. Example (based on continuing Sample Script #1 under Plots above.)

     
     set_plot,'ps                  ; Plot will be sent to PostScript file "idl.ps", 
                                   ;     not to the screen
     !p.font=0
     
     device,/helvetica,font_size=12  ; Use Helvetica PostScript font
     
     ploterror,r25,mags,magerr,psym=8
                                   ; Make plot (in PostScript file) with error bars; 
                                   ; The keyword psym selects the plotting symbol;
                                   ; Psym=8 specifies a user-created symbol, which
                                   ;    in this case was defined by plotsym
                                   ; System variables for labels are still in force
                                  
     
     device,/close                 ; Close file 
     
     $lp idl.ps                    ; Send file to printer
     
     $mv idl.ps surbriteplot.ps    ; Rename PS file to save it.
     
     set_plot,'x                   ; Send further output to the terminal
     
     cleanplot                     ; Reset the plotting system variables to defaults
     

  8. The output file will have the expected "black-on-white" sense, unless you make use of the keywords described next.

  9. You can make color PostScript files by setting device,/color and using the background and color keywords (see the Plots section). However, the treatment of color tables differs from the X-windows case. Consult the IDL manuals.

• Making a PostScript hardcopy of a displayed image: use tvlaser. This dumps a bitmap of the current window to a PostScript file and prints it on the default PS printer. Various options to add comments, information from the header, change the format, and so forth. If you want to save the PostScript file, answer no to the query about "removing" it. For color output, use the colorps keyword. For true-color images, use the truecolor=N keyword.

• Other output formats. For image output in GIF, JPEG, TIFF, PNG, and other popular output file formats, you do not need to use the set_plot command. Simply send the image to an output file using the appropriate write command. See the description above of capturing image windows using the tvrd command. David Fanning's tvread program can dump an image and make an output file in one step through use of keywords like /jpeg.
  Tip: when producing PostScript output files of images for publication, it's useful to also make a GIF version, since this can be read back into IDL later if touch-ups are needed. IDL cannot read PS files (though LINUX xv can convert PS's to GIF's).

• Large Images: image output need not be confined to those sizes you can display on your screen, as in the above examples. Larger image arrays can be written directly to files using the write_gif, fits_write, and similar commands.

PostScript files are the best way to get high quality line-drawing plots and are generally well treated by publishers. Grayscale or color images are another matter, however, and should be approached in an iterative way. A published PS image may look very different from the one you printed locally. GIF or TIFF files may yield better results than PS.

There can also be difficulties with the finite resolution of screen displays copied using the tvrd routine. This is true for the quality of both the image and any lettering which may have been added to it. On a typical computer screen, you will not get more than about 900x900 resolution. However, a PostScript file at 300 dpi can yield much higher resolution (1800x1800 in a 6" image).

As an alternative to the screen-capture tvlaser method described above, you can write images and labeling directly to a PostScript file using the standard tv, tvscl, xyouts and other routines. The main difficulty is understanding the coordinate system used within the PS file and placing the various elements of an output page in the right positions. People wind up using trial and error (easy by writing the PS file, then giving a $ghostview [filename] command from within IDL).

If you store duplicates of your graphics output in GIF, TIFF, JPG, etc files, you can use non-IDL utilities like xv, Gimp, Photoshop, etc. to manipulate them further: in size, rotation, contrast, color table, and by applying various image enhancements. xv can convert GIF to PostScript and vice versa. Especially useful for compressing image sizes for use on the Internet, the arXiv Preprint Server, and so forth.

For more tips on making good PS graphics, see Making Figures with IDL by Kelle Cruz and David Fanning's http://www.dfanning.com/documents/tips.html#PostScript

13. IDL HINTS AND ANNOYANCES

[Up to Contents]

Not many people have experienced fully interactive computing before they start to use IDL. There are tremendous advantages but also many pitfalls for the unwary. The pitfalls will, of course, mostly seem obvious and trivial in retrospect---i.e. after you have learned to avoid them. A number of tips & warnings for IDL beginners are discussed in this section.

• If a procedure name is unrecognized or gives unexpected error messages, check to be sure your directory path (!path inside IDL) is set properly. The initial value of !path is taken from the shell variable $IDL_PATH. Check the directories in the path for the procedure.

• Unexpected behavior can also occur if you pick up the wrong version of a program for which there are multiple copies in your !path. Two ways to check out possible mistakes of this kind: print,file_which([program_name.pro]); or .run -t [program name] and inspect the listing of the program for version information. The file_which routine operates like LINUX which. If your !path does not explicitly include the current directory (which is recommended, see below) then you should add the /include_current_dir keyword.

• The notation :+/directoryname in the $IDL_PATH definition expands the path to include all subdirectories of the named directory which contain *.pro files. Use sparingly to avoid picking up "old" versions of routines, which are often stored in subdirectories of active versions.

• Errors will also occur if you try to call a function as a procedure or vice-versa. A procedure requires a [name],parm,... call while a function requires a [var]=[name](parm,...) call. Use help,/rou to keep functions and procedures straight. If you try to execute a main program by typing its name rather than by typing .run [name], you will receive an error message.

• Remember that data output will be written to the directory which is current when the output command was given. If you use sd frequently to move through directories, you may lose track of where the output went (though a journal file can tell you). If you are currently in someone else's directory on which you do not have write permission, the output commands will not function. Some commands (e.g. tvlaser) can produce large scratch files. "Unrecognized file" errors during input commands often mean that you are in the wrong default directory. Check with sd.

• After modifying a [name].pro procedure file, be sure to recompile it with .run [name] before attempting to use it. If the routine had been compiled before you modified it, simply entering [name] will execute the OLD version.

• In programming, note the great usefulness of interactive aids in intrinsic IDL. such as size, n_params, n_elements, keyword_set, and others. These allow you to assess the state of the variables in your program at any time.

• print allows you to print the current value of any variable to the screen at any time during an interactive session (beware, however, in the case of large arrays). A format statement is optional. Default formatting: row vectors are printed across the screen; column vectors, down. printf is the corresponding command to print to an output file. Both commands can be used within programs.

• readf is the standard interactive command to read data from an ASCII file. A format statement is optional. The same command can be used within programs. To read input from the keyboard from within a program, use the read command.

• readcol is a handy AstUseLib utility for reading in ASCII files of columnar data. A format statement is optional (depending on the content of the files); lines containing unexpected characters are automatically skipped. For fixed-format ASCII I/O, see readfmt and forprint.

• IDL array-oriented mathematics often eliminates the need for do-loops or index-specific programming, and you should use index-free notation wherever possible. (IDL is significantly slower when data elements are referenced by indices.)

• There are about 20 intrinsic IDL functions for creating arrays of different types and filling them with initial values, including intarr, fltarr, indgen, findgen, replicate and make_array.

• There is a large set of other array-oriented arithmetic functions, such as expand, mean, rebin, reverse, smooth, sort, total, transpose, and so forth.

• You will find many uses for the where function. E.g. to find all non-positive values in an array in order to prevent errors when taking logarithms, type find=where(a le 0). find will be a vector containing the indices of the non-positive entries. The subarray a(find) then will contain all those points.
  For example: to replace all values less than 1 in an array with 1, type
  a(where(a lt 1))=1.
  If, instead, you simply want a count of the number of entries with values less than 1, type:
  find=where(a lt 1, count) & print count

• Another quick way to place a floor on elements in an array: a = (a>1) replaces all elements with values less than 1 in array a with 1. (The notation a>1 means "an array containing the maximum of 1 and the original elements of a".)

• The recall/edit and journal utilities are indispensable means to iterating and recording IDL code.

• Remember that the first element of vectors and arrays is always at index 0. (This is to preserve the conventional axis notation for plots.) If b contains ten elements, element indices run from 0 to 9, not 1 to 10. A reference to b(10) is out of the vector's bounds. This will kill a subroutine, though it only generates an error message if it occurs on the command line.

• If some previously defined variables seem to have "vanished" or other peculiar behavior occurs, check to see that you are in the main program and not a subroutine. See retall above. To return automatically to the main program whenever an error occurs in a subroutine, enter on_error,1 at the beginning of your IDL session (strongly recommended except when debugging).

• To create test arrays/vectors in order to experiment with IDL procedures, use indgen, findgen, etc. For instance:
  a=findgen(100,100) creates a 100x100 floating point array with unique element values running between 0 and 9999.0.
  a=fltarr(100,100)+10. creates a 100x100 array containing 10.0 everywhere.
  Create vectors with simple definition statements like z=[1,3,9,27]; statements like z=[z,81] enlarge vectors.
  You can extract subarrays using index notation, as in parta=a(100:156,200:296).

• The maximum value of an IDL floating point constant is about 3e38. To deal with larger numbers, you must use double-precision. E.g. in computations involving the speed of light, good practice is to define c = 3.0d10 rather than c = 3.0e10. The largest double-precision constant allowed is about 4d88. (Minimum values are the inverses of these.) Computations which produce out-of-range values will lead to "overflow" or "underflow" error messages.

• Beware of implicit redefinition of variables.
  IDL dynamically defines variable type, and this is a classic source of IDL novice errors.
  For instance, x=5/2 returns the value 2 (integer) while x=5./2 returns 2.5 (floating point). x has different values and attributes in consequence. Watch out!

• Implicit redefinition of array variables can lead to big difficulties as in the following:

  
         A=fltarr(200,200)    ; Define floating point array with all elements
                              ;     = 0.0
         A(*,*)=1. OR A=A+1.  ; Proper way to fill array A with 1.0
         A=1.0                ; Wrong way to fill array A with 1.0.  
                              ; You have just changed A to the scalar 1.0!
   

• Another potential problem area lies in extracting vectors from arrays. In IDL a 1-D vector is assumed to be a ROW vector.
  Thus, if a is a 100x100 array, b=a(*,40) will create a row-vector of dimension (100). But b=a(40,*) will create a 2-D ARRAY with dimension (1,100) (= a column vector).
  To eliminate the orphan dimension here and obtain a standard row vector, use the reform function: b=reform(b). You could also use the transpose function to convert the column vector into a row vector.
  You can check the current structure for a variable using the help command.

• Here is an example of another problem you can encounter with IDL's implicit definition of variable type involving a dependent variable array from which you want to extract a particular value. If time is the independent variable and value is the dependent variable, you might want to locate a particular entry in value as follows and renormalize the value array:

   
             find=where(time eq 1000.)
             norm0=value(find)
             normalval=value/norm0  

  Naively, you expect norm0 to be a scalar, as it would be if you typed norm0=value(151), for instance. However, if you try this, you will find that here it is not a scalar, but instead has type array(1). This is because the where routine returns a vector, which definition propagates through to the definition of norm0. The resulting quantity, normalval, which divides two vectors of different lengths, is therefore a scalar, not a vector! The reform function is no help here. To get the intended result, you must define the normalizing constant as: norm0=value(find(0)).

  If you are writing code for general application, you would also want to confirm that find returns only a single non-negative element. The number of elements returned by the where function can be determined, for instance, as follows:

   
             find=where(time eq 1000.,count) 
             if (count eq 1 ) then begin...   

• When working with data arrays (e.g. of CCD or HST data) where the data has limited dynamic range and does not require more than 2-byte precision, it may be faster and more convenient to convert them to integer form after multiplying by a scale factor than to leave them in floating-point form. This saves storage, increases execution speed, and often reduces typing. You can save manipulated versions by back-converting, but you have to be careful to check that you haven't lost any bits off the end.

• Procedures that require vector arguments include max and min. If a,b,c,d are scalars, you must enter print,max([a,b,c,d]) rather than print,max(a,b,c,d).
  Miswriting statements like x = max(a,b) can actually result in a changed value for b. Beware.

• Functions such as smooth, rebin, and other resampling routines usually preserve the variable type. When applied to integer arrays, numerical errors can be introduced by truncation. If you are interested in preserving data values with such routines, you should apply them only to floating point arrays. To convert an integer array to floating point, enter a = float(a).

• Remember that the main level of IDL will not recognize variables which are defined in procedures (subroutines) but are not passed through calling parameters back to the main program. (Same as in FORTRAN.) Internal variables vanish on a normal exit from a procedure. To determine which variables are active, use help. You can, however, use common blocks (as in FORTRAN) to establish links between main level and subroutine level variables which are not in the calling sequence. Main programs have access to all variables which have been defined at the "main" level.

• You can also define your own system variables, which are known to all programs, using defsysv.

• IDL structures provide a compact and convenient way of moving large numbers of related variables of various types (scalars, arrays, strings) between procedures and main programs.

• A frequent source of minor trouble for users of other high level languages are the small differences in IDL syntax for if statements, then statements, do loops,etc. The latter in IDL are for loops. Also note the use of the case and goto statements. It is worth spending some time reviewing these areas before writing IDL code.

• IDL's element subscripting convention for 2-D arrays is reversed from FORTRAN. In FORTRAN, A(i,j) corresponds to the element in the ith ROW and jth COLUMN. But in IDL, A(i,j) corresponds to the element in the ith COLUMN and jth ROW. A(*,j) is a row vector. A(i,*) is a column vector.
  This convention was adopted for IDL because it produces the standard sense for (x,y) displays. The default display commands will show an array A(x,y) starting with column 0 at the left of the screen and row 0 at the bottom. When displayed, A(4,1) will be four units to the right and one unit up from the origin. This change in convention obviously requires care if one is trying to operate on a given dataset with both IDL and FORTRAN.

• For a quick reminder of IDL syntax, see Ray Sterner's IDL Statement Syntax page.

• IDL is not case-sensitive, though LINUX is. Commands which will involve the operating system (e.g. reading/writing files) therefore must observe case conventions.

• The period in executive commands (e.g. .run) must appear as the first character of the command line (no preceding blanks).

• The "single-quote" mark for strings, filenames, etc., is the apostrophe ('), not the "leading-single-quote-mark" (`). Syntax errors occur if you switch these. Unfortunately, on many terminal screens, these are hard to distinguish. On some keyboards, they are also on adjacent keys, which promotes typing errors.

• Strings must always appear in quotes; file names are treated as strings in most routines involving files.
  If a string is the last entry on a line (no further parameters or punctuation), then the trailing quote mark need not be entered. But in this case, be sure you don't enter an extra blank space on the terminal before pressing "return", or the string may be incorrectly read.

• A common typo which can lead to a syntax error is the substitution of a period for a comma in a procedure argument list.

• Recall that a vector is identified by enclosing brackets, not parentheses: x=[47,29,135], not x=(47,29,135).

• Beware of inadvertently typing mathematical operators where you should use relational operators. E.g.: if (x = 5) then ... instead of the correct if (x eq 5) then .... This can produce serious bugs that don't necessarily raise error flags.

• Recall that the > and < signs invoke the "maximum" and "minimum" operators, respectively. E.g. 10 > 15 is the scalar 15.

• It is a good idea to make liberal use of the sxaddhist routine to update the comments on your data set as you read, process, and save FITS images.

• Since interactive IDL sessions often produce a large number of active variables, it is useful to be systematic in naming each new variable. E.g. in reading arrays, use conventions like: fits_read,'filename',pic1,hdpic1 to keep the image and header arrays straight.

• To save typing and looking up long file paths, it's useful to define special string variables which contain the fully qualified names of your frequently used directories. Then you can just sd,[stringname] to change directories.

• If you manipulate the color tables, you will probably want to reload a standard table (e.g. loadct,0) before displaying a new image.

• Some IDL users like to write a specific, single-purpose main program to perform a particular task and then change it, or proliferate multiple versions, to deal with similar but slightly different applications (e.g. a new data set). However, since main program input/output is not made explicit when you .run a main program, the journal file does not succinctly capture it. Furthermore, unless one deliberately renames and saves the main program every time it is changed, it may not be possible to reconstruct what actually happened.

• Other users prefer to write more generalized subroutines, which take defined sets of input parameters but which can perform the function on a variety of different inputs. They then journal the sessions in which the routine is actually applied. In this method, the specific input to and output from each subroutine are clearly and concisely documented.

• The latter approach produces software which is usually more reliable and stable over long periods (not to mention usable by others), and it is easier to reconstruct what was done at a particular time. Other advantages, which can be crucial, are that in writing a more generalized subroutine, you are more likely to spend the time to review the logic, to adequately document what the routine does, and to catch typing errors (which can sink you in the main program approach). I have found this to be much more reliable than the "single-shot" main program approach.

• Whichever method you use, you should journal everything if you are doing serious work.

• Although it is possible to concatenate many IDL procedures into a single *.pro file and to compile them all with one command, this prevents you from using the man help procedure, which lists headers based on assumed file names. It will also prevent file_which from locating embedded programs. Unless you are dealing with large packages with numerous subroutines that are better kept together, it's cleaner and less confusing in the long run to keep all procedures in separate files. You can always write batch compilation files which catch all procedures needed for a given task. Or you can simply let the automatic compilation system find and compile programs when they are first referenced. This is less trouble and just as fast.

III. APPENDICES

APPENDIX A: IDL SETUP

[Up to Contents]

Before executing IDL, you must define a special set of IDL environment variables, so that LINUX and IDL know where to look for the various IDL executable and program packages.

If you use the UVa Astronomy Department system defaults, you will be automatically set up to use the latest version of IDL including the MOUSSE package as installed on our local servers. If you want to use the system defaults, you can ignore the rest of this section, except for morbid curiosity.

To define the IDL environment variables you must source a special idl_env file.

You cannot change the contents of the idl_env file or the defined environment variables after IDL begins execution.
The file that configures a default session on the Astronomy Department network is named /astro/idl/idl_env.csh. (For a BASH session, use idl_env.sh.)
Alternatively, you could define the various required IDL environment variables in your .login or .cshrc files. However, it is cleaner to keep all such definitions together in the idl_env file.

Each set of executables is authorized by ITTVIS to run on only certain hosts, so you must point $IDL_DIR to the appropriate directory for your current host to run IDL other than in a 7 minute, stripped-down, "demo" mode. The executables must be compatible with the current version of the operating system running on your host.

You must also specify the search path for all *.pro routines you wish to use (other than the intrinsic ITTVIS routines).

The path is defined in the LINUX shell variable $IDL_PATH, which is converted to the IDL system variable !path.
Once $IDL_PATH has been defined, it cannot be changed during an IDL session.
IDL's default in the absence of a specified path is to use the current directory only.
When IDL looks for a named routine, the path is searched in order from the first entry to the last. The first file with the expected name is executed.
The search path should include the IDL User's Library (distributed with native IDL), the IDL Astronomical User's Library, the ATV directory, and the MOUSSE directory.
The path should also include your own idl directory and any other directories containing customized routines which you wish to invoke without making them the current directory.
Beware of multiple versions of the same program (e.g. an AstUseLib version and a Users Library version)! You must arrange your path so that the preferred version occurs first in the path.
At UVa, the AstUseLib versions are preferred, and should be placed first in your path except for your own idl directory.
Recommended path structure (see examples below for syntax):

  [current directory], [your idl directory], [other special local
  *.pro libraries, if any], [Astronomy Users Library], [MOUSSE], 
  [ATV], [IDL Users Library].  

Separately, if you wish to use the databasing software, you must specify the location of the IDL databases (the zdbase directory)..

When you give the command idl under the following setup, this is what happens:

1. The executable idl located in $IDL_DIR is retrieved

2. The ITTVIS/IDL license is checked

3. If your host computer is authorized to use IDL, the normal program is executed

4. If you are not licensed for IDL, the program runs in a 7-minute "Demo" mode

5. The shell variables starting with IDL are converted to corresponding internal IDL system variables

6. The file $IDL_STARTUP is executed, if defined.

APPENDIX B: ENVIRONMENT SETUP EXAMPLE

[Up to Contents]

### This is the UVa Astronomy LINUX system default env file 
### as installed on the department server, March 2009.
### Location is:  /astro/idl/idl_env.csh


# C shell commands to define IDL environment variables and aliases.
#
# Script name: idl_env.csh
#
# This script is used by C shell users to define 
# the environment variables and aliases required by IDL
# related commands.
#
# To execute this script, give the following command; or,
# for automatic execution, place it in your .cshrc file.
#
#    source /astro/idl/idl_env.csh
#
# To overwrite the defaults set in this file, you can source other
# setup files from your individual .cshrc file.
#

##### 
# System choice - deprecated 03-07-11
#####
# set sys=`uname -a | awk '{print $3}'`
# if ( $sys == 5.6 ) then
#          setenv IDL_DIR          /net/astsun.astrosw/idl/
#       else
#          setenv IDL_DIR          /net/jeeves.common/rsi/idl/
# endif

# Note: the March 2009 configuration substitutes a departmental version
#       of IDL v7.0 (/astro/itt/idl) for the UVa v6.4 version
#       (/common/rsi/idl). 

#setenv IDL_DIR          /common/rsi/idl/
setenv IDL_DIR          /astro/itt/idl/
setenv IDL_HOME         ${IDL_DIR}
#setenv IDL_HELP                /common/rsi/idl/help
setenv IDL_HELP         /astro/itt/idl/help

# License file

#setenv LM_LICENSE_FILE /common/rsi/license/license.dat
setenv LM_LICENSE_FILE /astro/itt/license/license.dat

# Local packages

setenv ASTROLIB_DIR     /astro/idl/Astrolib
setenv MOUSSE_DIR       /astro/idl/Mousse
setenv MOUSSE98_DIR     /astro/idl/Mousse.98
setenv PIA_LOC_GEN      /astro/idl/PIA/
setenv ATV_DIR          /astro/idl/atv
setenv FUSE_DIR         /astro/idl/FUSE

setenv IDL_HELP_PATH    "+${MOUSSE_DIR}:+${ASTROLIB_DIR}:+${IDL_HELP}"

# ZDBASE is the location of the IDL-formatted databases 

setenv ZDBASE           /astro/idl/zdbase

# Notes on IDL_PATH
# 
# Symbol "+" means that the path will be expanded to include all
#   subdirectories within the given directory.  
# The user's own IDL programs are assumed to be in a directory named,
#   or linked to, "idl" under the user's home directory
# Directory $IDL_HOME/lib contains the IDL User's Library of
#   standard user-written utility routines.  

setenv IDL_PATH         ".:+~/idl:/astro/idl/UVAlocal:+${ASTROLIB_DIR}:${ATV_DIR}:+${MO
USSE_DIR}:+${MOUSSE98_DIR}:+${IDL_HOME}/lib:+${IDL_HOME}/examples:+${PIA_LOC_GEN}:+${FU
SE_DIR}"

setenv UIT_DATA         ${IDL_HOME}/data

setenv IDLUSR           ${HOME}
setenv IDLUSER          ${HOME}

alias idl $IDL_DIR/bin/idl
alias idlde $IDL_DIR/bin/idlde
alias idldeclient $IDL_DIR/bin/idldeclient
alias idlhelp $IDL_DIR/bin/idlhelp
alias idlrpc $IDL_DIR/bin/idlrpc
alias insight $IDL_DIR/bin/insight
alias idldemo $IDL_DIR/bin/idldemo

alias pia $PIA_LOC_GEN/PIA

# The following IDL startup file will be executed before
#    each IDL session.  User can specify a different
#    startup in his/her .cshrc file.

setenv IDL_STARTUP /astro/idl/startup/mousse_startup.pro

APPENDIX C: MOUSSE STARTUP FILE EXAMPLE

[Up to Contents]

  ; STARTUP FILE mousse_startup.pro
  ; Last Modified: 15 May 1992, 30 Sep 93, 27 Dec 93, 22 Aug 97

print , "Running mousse_startup.pro..."
print , "For help on AstUseLib and Mousse routines, use the Mousse MAN procedure."

setplot, 'X'
DEFSYSV,'!DEBUG',0
DEFSYSV,'!TEXTUNIT',0
DEFSYSV,'!TEXTOUT',1
DEFSYSV,'!PRIV',0
defsysv,'!psprinter','astro-hp'     ;  UVA default printer
cinit              ; CINIT sets up MOUSSE common blocks 
!PROMPT = 'IDL>'
!EDIT_INPUT = 100
on_error,1         ; Return to main program in case of error
print,' *** on_error,1 is default ***'
print,' *** If terminal is vt100/tek, type SETPLOT,0'

APPENDIX D: X-WINDOWS NOTES

[Up to Contents]

This guide assumes you are executing IDL from within an X-WINDOWS environment. So: first, you must start your windows system and open an appropriate window to enter IDL commands. The location of display windows opened by various IDL applications can be controlled in many instances. Consult the description of xpos under the window command. It is useful to place display windows away from your command window, wherever possible. You can move existing windows using your mouse.

Normally, you run IDL on your local machine with display on your local terminal. To run IDL on a remote host which supports X-windows with display on your local terminal, do the following:

1. Edit the .xinitrc file on your local machine to include the statement xhost [remote host], where [remote host] = the fully qualified network name of the IDL host you wish to use. E.g.:
   xhost parfait.gsfc.nasa.gov

2. Start X-windows on your local machine.

3. ssh or telnet to the remote IDL host and login

4. To obtain X-windows output from the IDL host on your local machine, type
   setenv DISPLAY [local]:0.0

   where [local] is the fully qualified network name of your local machine, e.g.

   setenv DISPLAY bonkers.astro.virginia.edu:0.0

5. Start IDL on the remote machine

6. New windows created by IDL should now appear on your local terminal. Of course, data display in this mode will be slower than if you were running on the local machine.