New information for the November 13 telecon has been added at the top of this page. The original posting (from November 6) follows below.
From Rae (Nov 12): I have produced a duplicate resolved (both with itself and with the all sky catalog) table to study the sources that were cut from the all sky catalog because psfchi was > 10.0. The figure (psfchi_cut_mag_dist.jpg) shows that between J magnitude 11 and 13 more than 1% of the sources detected by the processing software are absent from the all sky catalog. A typical missing source is shown in (psfchi_restored_3_band_source.jpg). This source and no match anywhere near it is the all sky catalog. The figure (psfchi_cut_all_sky_prox_dist.jpg) shows the proximity of sources in the all sky catalog to the (internally dup resolved) sources that were rejected by psfchi cuts. All sources with as_prox < 2" would not be restored to the all sky catalog if we decide to do this. The curious peak in 3-band sources with all sky prox of 4.5 arc seconds represents double stars that were resolved by the processing software but where the psfchi cut rejected one of the pair from the all sky catalog. A sample source is shown in (psfchi_restored_dumbell.jpg). All of the sources propsed for restoration have ph_qual of E (or U). We should revise the text of the explanation of ph_qual to say that the photometry of sources with ph_qual can not be trusted and that users should be referred to the uncompressed images for photometric information. The number of sources proposed for restoration is about 1.3 million.
|
|
|
|
| Figure 1 | Figure 2 | Figure 3 | Figure 4 |
The figures below characterize the nature/quality of the three band (EEE) detections in this catalog.
The figure below shows all 702,642 "EEE" sources. The fact that many/most of the sources arise from low-galactic-latitude confusion is evident in the significant reddening locus.
The figure below shows 35,194 sources with abs(glat)>20. The main sequence locus is evident and distinct, as is a haze of objects with extragalactic colors (the rightmost "finger" in the color-magnitude diagram). 6024 objects fall in this extragalactic finger with J-Ks between 1.0 and 1.3. 80% of the objects with J-Ks between 1.0 and 1.3 have ext_key_1 id's in the wsdb -- they are galaxies omitted from the PSC because of high chisqr. It will be interesting to investigate what fraction of the 20% are bright galaxies which did not make it into the XSC (probably because of bright star blanking).
In the figure below the psf_restore database has been joined with the v3_pwsdb and aperture photometry has been extracted for all of the EEE sources.
Below is the text and figures relevant to the November 6 telecon
From Rae: I have attached a figure that compares j_psfchi for sources in the v2 processing within 0.1" of sources in the v3 processing. (The previous figures were made with a 4" match condition and I was concerned that I might have been matching different objects.) There are two main features of this plot. The first is that v2 processing generated lower psf_chi values on average than v3 processing with the exception of the vertical tail of sources with v3 psfchi between .6 and 1.1 and v2 psfchi > 3.0. This tail is curious. It is made up of resolved double stars. I don't know why v3 processing assigned such a low psfchi for these sources in comparison with v2 processing. A sample of these things is given below: cntr | rd_flg | v2_rd_flg | cc_flg | j_m | v2_j_m | j_psfchi | v2_j_psfchi ----------+--------+-----------+--------+--------+--------+----------+------------- 575879 | 222 | 222 | ccc | 15.038 | 15.057 | 0.94 | 4.33 752760 | 226 | 222 | cc0 | 15.535 | 15.525 | 0.8 | 3.62 1046727 | 222 | 222 | ccc | 13.687 | 13.65 | 0.6 | 5.33 1144702 | 222 | 222 | ccc | 13.963 | 13.949 | 1.03 | 23.04 1307506 | 262 | 222 | c0c | 13.731 | 13.606 | 0.93 | 26.58
Psfchi is large for source pairs where the sources are too close together for the processing software to resolve into two sources but are far apart enough to create an image that is not circular. Figure 5 illustrates another feature of the algorithm used to select sources. 1,524,402 sources with j_psfchi > 10.0 are in the all sky catalog because the alogrithm used for making the preliminary selection had a cross band leak. All rows with a 136789 in the rd_flg in at least one band were selected with regard for j_psfchi even if the j band detection was a psf fit. Furthermore a j_psfchi < 10.0 in any band allowed a psfchi > 10.0 to be accepted in another band. A similar problem exists in H and K band. We need to quantify what is missing from the all sky catalog nction of separation and magnitude difference. We also should issue a warning about large psfchi's in the all sky catalog. Rae
From Roc: The chi-squared values from the profile fits will differ between IDR2 and the all-sky PSC because the PSFs and variance maps used in preliminary and final processing were different. Those used for final processing are better matched to the data because more sets were made, using more input data to better match the systemmatic differences in PSFs with time during the survey, and because some model errors were present in the preliminary PSFs that were corrected for construction of the final errors. Although we know that the final error model still contained problems that resulted in an overestimate of the uncertainty, the final processing psfs are in most regards better than those used in preliminary processing. One result of the better PSFs and variance maps in the final processing is apparently that the chi-squared values for true multiple, high SNR sources are larger than they were in the preliminary processing. That is, the confidence with which we distinguished bright multiple sources from singles was better than in the preliminary processing. The psf rejection threshold effect must primarily be due to a signal-to-noise. At high SNR, real differences between the observed source profile and the theoretical point response function will manifest themselves with elevated chi-squared since the differences are weighted by the variance map. As the bright of a source decreases, the deviations in the profile from the single-source psf become less distinguishable relative to variations due simply to noise. This was one of the reasons why when active deblending was being discussed for final processing, it was to be implemented only for relatively high signal-to-noise sources (Ks<13, or so) that exceeded some threshold in chi-square. For faint sources in which the variations in the source profile is dominated by photon noise, the chi-squared value tend to unity (if the variance map is okay). Recall for the preliminary processing, we actually had sub-optimal variance maps that were producing chi-squared values significantly different from unity for faint sources - this was one of the improvements in the psf-generation for final processing. This tendency towards unity for the chi-squared values of fainter sources is one of the reasons that chi-squared is such a good indicator of anomalous sources (e.g. H- and Ks-band only sources from single frame events such as hot pixels or cosmic-rays - see the plots I circulated Friday night). In retrospect, we probably needed a brightness-dependent chi-squared threshold in selecting sources for the release. At high SNR, the chi-squared limit should have been higher since doubles and multiples stand out from single sources with much greater contrast in chi-squared space. At the faint end, we could have cut in chi-squared space at a much lower level since reliable sources had much better behaved chi-squared values. Alternately, perhaps we should have removed the chi-squared threshold altogether, and instead built in a magnitude-dependent chi-squared term in the photometric quality indicator. Well, hindsight is 20-20 after all.... For all of these reasons, I feel that the chi-sqaured goodness-of-fit parameter for the R2 sources is an important quality indicator that must be provided the user. Roc On Sat, 2 Nov 2002, Rae F. Stiening wrote: > I have also looked at missing things and refer you to my Figure 4 which > shows the missing sources (only looking at J band) as a function of > default magnitude. There are distinct features in the red curve. For > missing sources with j_m < 14.0 and psc_idr2_prox between 8.0 and 28. I > suspect that most are doubles. If this is so a lower limit on the number > of missing doubles is 150,000. (I correlated 1/8th of the idr2 which > is 1/16th of the all sky rows.) > > For some reason the psf_chi problem (calculation difference between ier2 > and all sky) that is killing the brighter doubles starts turning off at > j_m of 13. As Roc indicates, the fainter missing sources may be there > because of changes in (db)mapcor. I found one of these in a sample of j_m > = 14.0 sources but most were still doubles. > > Does anybody have an idea why the psf_chi calculation difference between > the idr2 and the all sky processing is only there when j_m < 14? > > >