JWST Noise Floor. I. Random Error Sources in JWST NIRCam Time Series

The NIRCam detectors' pre-amplifiers in the SIDECAR ASIC have a reference voltage that will drift with time, which will affect the pedestal level of all pixels referenced to that voltage. This was introduced as error source 1 in Section 2. To address the reference voltage drift, the pre-amplifiers can be reset to the reference voltage periodically, which will introduce a random uncertainty into the pedestal level. The SIDECAR ASICs support resetting to the reference voltage either at frame boundaries (between frames) or at integration boundaries (between integrations). In early ASICs the pre-amplifier drift between resets was seen to be high enough to allow the pedestal to drift off the range of the ADC during longer integrations; hence, the NIRCam team elected to reset the pre-amplifiers on frame boundaries rather then integration boundaries. Subsequent delivery of flight ASICs showed much lower drift rates. To characterize the impact of reset per frame (RPF) versus reset per integration (RPI), a set of darks was collected during NIRCam cryogenic vacuum (CV) pre-delivery testing at Lockheed-Martin in both RPI and RPF mode. In these tests, the pedestal level of all pixels could be seen to drift over 19.3 minute dark integrations by 5–100 DN in RPI mode. The drift was usually linear over these 19 minute exposures, but in one case was observed to drift downward from 100 DN to −50 DN and back to 100 DN (Robberto 2014) while RPF introduces a random uncertainty of the order of 30 DN rms at each frame boundary in the sample up the ramp. Reference pixel correction effectively removes both the random RPF jumps and the systematic RPI drift; analysis (Robberto 2014) showed that either approach was roughly equivalent. Operationally, NIRCam elected to continue using RPF since there was no benefit to switching to RPI and significant legacy characterization data had been obtained in the RPF mode. Reference pixel correction or careful background subtraction can efficiently remove the signature of the pre-amplifier reset or drifts. The NIRCam grism subarrays and dispersed source location are placed at the edge of the NIRCam LW detector to ensure that the reference pixels are nearby and saved with active pixels.

Figure 2 shows an example time series of the reference pixels for a dark exposure. The pedestal/bias level of the reference pixels undergoes a sharp jump between each frame (10.7 s in duration). Fortunately, the pixels within an amplifier move together, so reference pixel subtraction or background subtraction will efficiently remove the offsets.

Zoom In Zoom Out Reset image size

NIRCam's STRIPE mode and full-frame mode both employ four amplifiers to simultaneously read out or reset 4 pixels at once. This simultaneous use of four amplifiers reduces the frame time (to scan through all pixels in a 2048-pixel-wide subarray or full frame) by a factor of about 4. Thus, the STRIPE and full-frame modes increase the brightness of objects that can be detected before saturation as well as increase the number of reads for a detector pixel before saturation to average down readout noise. Users have the option to select one or four amplifiers with the number of output channels in the Astronomer's Proposal Tool (APT; STScI 2020b). On the flip side, the four amplifiers will have different behaviors such as separate gains and offsets in the bias level. The differential biases between the amplifiers will show up as step function changes in the slope of the image at the amplifier boundaries (512, 1024, and 1536), as visible in Figure 3 (top panel). If a spectrum of a source is extracted without any vertical background extraction or reference pixel correction, the amplifier offsets cause sharp discontinuities in the summed spectrum. The image shown here is from a grism exposure during a cryogenic vacuum testing 3 (CV3) test at NASA Goddard. The locations of amplifier boundaries and reference pixels are visible in Figure 1.

Zoom In Zoom Out Reset image size

Fortunately, the amplifier offsets are efficiently removed by reference pixels and background subtraction. For example, subtracting the average reference pixel in each amplifier will reduce the offsets between amplifiers, shown in Figure 3 (second from the top panel). One could subtract the mean reference pixel in each column, shown in Figure 3 (third from the top panel), but given that there are only four reference pixels at the bottom of each column, this introduces extra noise in the subtraction. Instead, one can use the background region for each column (∼105 pixels used here) to improve the removal of gradients and the amplifier offsets. In Figure 3 (bottom panel), we determine the median of all pixels from Y = 0 to 30 and Y = 180 to 255 in each column and subtract this value from the whole column. This is repeated for all columns. This background-subtraction method can efficiently remove the pre-amplifier offsets to the level where amplifier discontinuities become smaller than the combined read and photon noise in the image. This background-subtraction method can be applied to the STRIPE subarrays that are 64, 128, 256, or the full 2048 pixel tall. We note that some amount of self-subtraction occurs from the extended wings of the point-spread function (PSF) for a 64 pixel tall subarray. A 20 pixel aperture contains ∼6% less flux than a 128 pixel aperture at 3.5 μm, but will not produce a significant time dependence to the flux. We discuss aperture losses more in Paper II.

It is instructive to look at dark frames to isolate the contributions of the readout electronics, including the SIDECAR ASIC, and how much they contribute to the error budget. The dark frames do not have issues with the photometric stability of illumination lamps, which is a persistent challenge with ground-based tests. They also do not contain photon noise in the background, which can make electronic noise sources harder to see. We study here the contributions from read noise, including the 1/f noise from the electronics. This experiment with dark frames does not test subtle changes in behavior of the electronic read noise under illumination, such as crosstalk where pixels that have no direct flux from a source in an image correlate with other regions of the detector with high flux rates.

A standard long dark exposure used for detector characterization is a single integration that is 108 groups of one frame each (RAPID mode). The detectors are read in the full-frame mode so that each frame is 10.73677 s in duration with the four output amplifiers in use, thus the total time for the exposure is 19.3 minutes. We use one of these long dark integrations to create simulated time series as if it were observing many consecutive integrations in the full-frame mode. The long darks are broken into 54 sequential pairs of images that are subtracted from each other (group 1 minus group 0, 3 minus 2, 5 minus 4, etc. in a zero-based counting scheme). This method of breaking up the sequence precludes the ability to study the effects of detector resets, which only occur at the beginning of an integration. We find that the detector reset does affect the dark current in an integration, but we do not expect detector resets to significantly impact the time series. This is discussed further in Appendix D. We also relabel the times to include the reset time that would normally occur after each integration, so the duration of the exposure from start to finish is 29.0 minutes. The relabeling of the times makes the simulated exposure similar to a real time-series observation but does not impact the analysis of the time series, which focuses on the scatter in the flux between read pairs.

Figure 5 shows an example dark read pair from frame 55 minus frame 54. In the raw data, there are four prominent vertical stripes that are 512 pixels wide and 2048 pixels tall due to offsets in the readout amplifiers. These amplifier offsets are caused by drifts in the reference voltage that are reset once per frame and vary from amplifier to amplifier. Additionally, there are obvious correlations along the fast-read (X) direction due to 1/f noise that appear as horizontal stripes in the image.

Zoom In Zoom Out Reset image size

The ramps-to-slope pipeline ncdhas or the STScI JWST pipeline will apply reference pixel correction, bad pixel masks, and nonlinearity correction before fitting the groups up the ramp to a line. In this work, we use ncdhas. The reference pixels are comprised of a 4 pixel boundary along the edges of the detector, which are not connected to detector photodiodes and can therefore be used to measure common-mode electronic noise without being affected by photon counting noise. As seen in Figure 5, reference pixels also share the amplifier offsets, so they can be used to reduce the large offsets between amplifiers.

The reference pixels are most useful for subtracting the offsets between amplifiers because there are eight rows of 512 pixels or a total of 4096 reference pixels to average for each amplifier at the bottom and top of the arrays, as shown in Figure 1. There are an additional four columns on the left and right side boundaries (8176 pixels) that can be used to further correct the left and right amplifier offsets. However, the 1/f noise that correlates along the fast-read (X) direction is more difficult to remove with reference pixels. Only 4 pixels on the left and 4 pixels on the right side may be used to subtract the 1/f noise in a given row and the four amplifiers can exhibit different 1/f noise. Therefore, significant residuals remain along the fast-read (X) direction in the reference-corrected read pair shown in Figure 5. These side reference pixels also cannot correct for read noise correlations that happen on shorter length scales than 512 pixels (5 ms in time).

Exoplanet transit measurements are dependent on the photometric stability within an aperture or spectrum. We therefore perform aperture photometry on the dark frame as if it were a weak lens image with a 70 pixel radius circular source aperture and a 75 to 100 pixel background annulus illustrated in Figure 5. We place three different source apertures with the same radii on the detector for comparison of different regions. The background subtraction will largely remove the offsets affecting the whole 512 × 2048 vertical stripe read out by one amplifier. However, the correlated 1/f noise along the X direction will not be subtracted efficiently by an annulus. Instead, a row-by-row line fit to the background can be more effective.

For a read pair subtraction, the correlated double sample (CDS) read noise is 13.3 e⁻ or 7.2 DN for the A5 detector studied here. If we assume that each pixel's read noise is uncorrelated with its neighbors, the total noise within the source aperture is then $\sqrt{{N}_{\mathrm{px}}}\times \mathrm{RN},$ where N_px is the number of pixels and RN is the read noise of a single pixel. If we also assume that the background pixels are uncorrelated with each other and the source pixels, then the total noise in background-subtracted photometry is 1310 DN.

This theoretical read noise is smaller than the photon noise expected inside an aperture during photometric stability time-series applications. To calculate the photon noise as well as convert the noise into ppm, we need to assume a well filling fraction for the integrations. Here, we assume a peak pixel count of 30,000 DN or about 60% well depth. With this detector well filling fraction and a weak lens plus 8 waves (WLP8) PSF, the number of DN collected within a 70 pixel radius aperture is S_tot = 3.9 × 10⁷ DN during the integration. The Poisson shot noise for photons is ${\sigma }_{\mathrm{phot}}=\sqrt{{S}_{\mathrm{tot}}}/\sqrt{G}=4400$ DN, where G is the gain (G ≈ 1.8 for the LW detectors). In terms of fractional noise, the photon shot noise should be 113 ppm, and the read noise should be 33 ppm. Therefore, the read noise is expected to be about one-fourth the shot noise for a two-group RAPID read mode (i.e., CDS). When averaging together 111 full frames over 60 minutes (an order-of-magnitude transit duration for most planets), the photon noise would contribute 11 ppm and the read noise would contribute 3 ppm.

In our experimental time series, however, the standard deviation in the time series in these three aperture source locations is 22,000 to 25,000 DN (without any attempt to correct for amplifier offsets or 1/f noise using reference pixels or otherwise). If uncorrected, this extra read noise will contribute 560 to 630 ppm to the time series per read pair! This read noise would be ∼5× the ∼113 ppm photon noise. However, there are ways to significantly reduce the pre-amplifier offsets and 1/f noise.

These large correlations in read noise can be reduced with the reference pixels, which are designed to track electronic noise without the presence of background photon noise. We use the ncdhas pipeline to apply reference pixel correction, which dramatically reduces the amplifier offsets and also reduces 1/f noise correlations along the fast-read (X) direction. The standard deviation of the flux for the three source apertures considered drops down to levels from 10,000 DN to 11,500 DN (depending on the aperture), but still falls short of the expectation of 1310 DN if the read noise is uncorrelated.

The background annular aperture can affect the precision of a light curve, but there is a balance between photon noise, background noise, and 1/f noise. The aperture radius studied here (70 pixels) has an area (15,400 pixels²) that is larger than the area of background annulus (13,700 pixels). If background and photon noise were the only factor, the background annular aperture could be larger to increase the precision. However, a large background becomes decreasingly correlated with the source pixels and 1/f noise makes using a background annulus much larger than 100 pixels less helpful for foreground-limited exoplanet system observations.

4.2.1. Row-by-row Subtraction

4.2.2. Row-by-row Amplifier-by-amplifier Subtraction

4.2.3. More Sophisticated 1/f Noise Reduction Techniques

The current NIRCam grism time-series mode, which uses the GRISMR element in the pupil wheel, is particularly susceptible to 1/f noise because the dispersion direction is parallel to the detector's fast-read direction as shown in Figure 1. This is also visible in Figure 6 where the grism apertures (in red) are parallel to the 1/f noise correlations. The background subtraction along the spatial direction (vertically) will efficiently remove pre-amplifier offsets from frame to frame and amplifier to amplifier. However, there are only limited background regions along the horizontal direction available to subtract out 1/f noise. This will result in increased noise on the right three amplifiers for the F322W2 filter (2.4–4.0 μm over ∼1600 pixels) and the left two amplifiers for the F444W filter (3.9–5.0 μm over ∼1100 pixels), in the native raw detector pixel coordinate system used in this paper.

Zoom In Zoom Out Reset image size

We give some illustrative numbers for a 0.165 μm wide (R ∼ 20) bin of a grism spectrum centered on 3.34 μm as listed in Table 1. Without reference pixel subtraction, the grism apertures have standard deviations of 80,930 DN. Reference pixel subtraction drops this value to 9900 DN.

As in Section 4.2.1, we perform row-by-row subtraction to further reduce the noise to 3860 DN. We use the pixels from X = 5 to X = 301 for the row-by-row subtraction to ensure there is essentially no source flux being subtracted. Note this is only possible when there are no nearby sources at X = 5 to X = 301. For a well depth of 60%, there are 3.7 × 10⁶ DN of source counts in the 0.17 μm bin, which amounts to a photon error of 1,430 DN assuming a gain of 1.8. Therefore, 1/f noise can potentially dominate the error budget for broadband grism time series. Expressed in ppm, the read noise after processing with row-by-row subtraction is 1044 ppm as compared to a photon noise of 390 ppm for a CDS subtraction (two groups in ramp).

Given the high 1/f read noise, it is important to maximize the number of samples up the ramp. The 1/f read noise averages down like $\sqrt{N}$ statistics after background subtraction (i.e., each read can be assumed independent and identically distributed). Therefore, for the example bin size of 0.17 μm, there should be at least seven reads to average per integration to ensure that the read noise is less than the photon noise. For RAPID and BRIGHT1 modes, there should therefore be seven groups and for BRIGHT2, there should be at least four groups (of two frames each). When deciding between BRIGHT1 and BRIGHT2, we recommend BRIGHT2 because it coadds reads together rather than skip frames. Using >7 groups for RAPID and >4 groups for BRIGHT2 would make the photon noise about equal to the read noise for this aperture size but ideally the read noise should be a small fraction of the photon noise.

4.3.1. Using the GRISMC Element

One way to mitigate 1/f noise is to use a different grism element (GRISMC) in the pupil wheel, which disperses the spectra along the column (vertical) direction. The NIRISS and NIRSpec instruments' spectroscopic modes are designed this way so that the dispersion direction is approximately perpendicular to the fast-read direction. This allows spatial background subtraction along the fast-read direction to mitigate 1/f noise. GRISMC is being discussed as a science enhancement for a future JWST cycle, most likely beyond cycle 1.

The trade-off with the GRISMC pupil element is that vertical subarrays will only be read with one amplifier (WINDOW mode), and thus will be about 4 to 4.5 times longer than four amplifiers (STRIPE mode), depending on line overheads. The overheads come into play the most for small subarrays. A 2048 × 64 STRIPE subarray has a frame time equal to 0.34061 s whereas a 64 × 2048 WINDOW mode has a frame time equal to 1.558 s. Therefore, the maximum brightness observable without saturation with a GRISMC pupil element and 64 × 2048 WINDOW mode would be K ≈ 5.7 compared to the GRISMR 2048 × 64 STRIPE mode with K ≈ 4.0. For this reason, the GRISMC is not currently available for time-series observations. If enabled as a new mode, GRISMC observations could have substantially reduced 1/f noise for the majority of known transiting planets, which are fainter than K ≈ 5.7.

Figure 7 demonstrates the advantage of the GRISMC dispersing element in NIRCam's LW pupil wheel. Before background subtraction (left panel), prominent 1/f noise can be seen as horizontal lines and bands approximately perpendicular to the dispersion direction. We show an example row-by-row linear background subtraction using the pixels from X = 1537 to X = 1760 and X = 1774 to X = 2044. This efficiently reduces the 1/f noise along the rows, but leaves a small residual nearby the source. The residuals are difficult to see by eye in the image, but are apparent in the covariance matrix in Appendix C.

Zoom In Zoom Out Reset image size

Background subtraction can alternatively be performed on short pixel distances (≲70 px), where the source flux is low (≳10 spatial pixels from the source center) with the GRISMC element. As seen in Figure 4, the 1/f noise begins to appear for more than ∼70 clock cycles corresponding to distances more than about 70 pixels along a row and frequencies below 1400 Hz. Subtracting out 1/f noise at 70 clock cycle timescales is not possible with the GRISMR element, where the spectrum extends more than 1100 pixels along the fast-read direction for the F444W filter. Furthermore, the background subtraction with the GRISMC element can be performed within one amplifier rather than extrapolating one amplifier's 1/f noise behavior from other amplifiers as was done with the GRISMR element. We found that a background from 7 to 30 pixels from the source efficiently removed 1/f noise down to 683 DN (205 ppm) in a 0.165 -μm-wide wavelength bin as shown in Figure 7 (right). The GRISMC therefore allows a standard background subtraction and summation source extraction methods, which efficiently reduce 1/f noise to below the photon noise level.

4.3.2. Small Spatial Apertures

4.3.3. Optimal Covariance-weighted Flux

Here, we provide a modification to the conventional optimal extraction formula from Horne (1986), that uses the covariance matrix of correlated pixels. The goal is to find a scheme that estimates the total flux by summing the value in each pixel j. We start by calculating a profile function P_j in the spatial direction that describes the fraction of the total flux in pixel j. The profile is normalized so that

$$\begin{eqnarray*}&&\displaystyle \sum _{j=1}^{m}{P}_{j}=1.\end{eqnarray*}$$

The summation for NIRCam would be along the Y (spatial) direction for a fixed wavelength. There is a small tilt of the grism spectrum of 0.11 deg (i.e., 1:480), so summing along the Y-direction will slightly mix together different wavelengths at the subpixel level. The exoplanet science applications we focus on here are concentrated on measuring broad molecular features, where the maximum possible resolution of the instrument is not as critical. Therefore, we allow for a small amount of wavelength mixing by studying summations along the Y-direction instead of a more complicated interpolated model. The technique below can also be modified to include summation along the spatial and spectral directions.

Any pixel can be used to estimate the total flux by dividing by the profile

$$\begin{eqnarray*}&&{f}_{j}=\displaystyle \frac{{S}_{j}}{{P}_{j}},\end{eqnarray*}$$

where S_j is the background-subtracted flux. Therefore, the flux total is a weighted average of each pixel's estimate of the total flux f_j . The goal becomes a weighting scheme to determine which weights w_j minimize the variance in the total background estimate

$$\begin{eqnarray*}&&{f}_{\mathrm{opt}}=\displaystyle \frac{{\displaystyle \sum }_{j=1}^{m}{w}_{j}\tfrac{{S}_{j}}{{P}_{j}}}{{\displaystyle \sum }_{j=1}^{m}{w}_{j}}.\end{eqnarray*}$$

The weights w_j can take on any value while still giving an unbiased estimator of the flux (Horne 1986). The weights are not normalized like the profile, P_j .

If one assumes that all pixels are independent thereby having zero correlation, the optimal weights (ones that minimize the variance in f_opt) are

$$\begin{eqnarray}&&{w}_{j,\mathrm{var}}=\displaystyle \frac{1}{\mathrm{Var}\left(\tfrac{{S}_{j}}{{P}_{j}}\right)}=\displaystyle \frac{1}{\tfrac{1}{{P}_{j}^{2}}\mathrm{Var}({S}_{j})}=\displaystyle \frac{{P}_{j}^{2}}{{V}_{j}},\end{eqnarray} \tag{ 1 }$$

where Var() is the variance or the expectation of the squared deviation from the mean and V_j  = Var(S_j ) (Horne 1986).

Exoplanet observations, where high signal-to-noise is needed to separate the planet signal from the stellar signal, are generally in the foreground limit (where the variance approaches the number of photons), so V_j  = N_phot,j in this limit. The profile is also proportional to the number of photons from the source, so

$$\begin{eqnarray*}&&{P}_{j}={{kN}}_{\mathrm{phot},j},\end{eqnarray*}$$

where k is a proportionality constant

$$\begin{eqnarray*}&&k=\displaystyle \frac{1}{{\displaystyle \sum }_{i=1}^{m}{N}_{\mathrm{phot},i}}.\end{eqnarray*}$$

In this case, the weights become

$$\begin{eqnarray}&&{w}_{j,\mathrm{sum}}=\displaystyle \frac{{k}^{2}{N}_{\mathrm{phot},j}^{2}}{{N}_{\mathrm{phot},j}}={k}^{2}{N}_{\mathrm{phot},j}={{kP}}_{j}.\end{eqnarray} \tag{ 2 }$$

Therefore, using this traditional formula (Equation (1)) for bright sources will provide nearly the same extraction as straight summation,

$$\begin{eqnarray}&&{f}_{\mathrm{sum}}=\displaystyle \sum _{j=1}^{m}{S}_{j}.\end{eqnarray} \tag{ 3 }$$

In other words, in the limit where the read noise and background noise goes to zero, conventional variance weighting approaches straight summation.

However, in the presence of noise correlations, the formulae become modified. The optimal weights for correlated data are the sums of the inverse covariance matrix along one dimension (Schmelling 1995),

$$\begin{eqnarray}&&{w}_{j,\mathrm{cov}}=\displaystyle \sum _{i=0}^{m}{C}_{{ij}}^{-1},\end{eqnarray} \tag{ 4 }$$

where C_ij is the covariance (Cov()) of the variable S_i /_Pi with S_j /P_j ,

$$\begin{eqnarray}&&{C}_{{ij}}=\mathrm{Cov}\left(\displaystyle \frac{{S}_{i}}{{P}_{i}},\displaystyle \frac{{S}_{j}}{{P}_{j}}\right)=\displaystyle \frac{1}{{P}_{i}{P}_{j}}\mathrm{Cov}({S}_{i},{S}_{j}).\end{eqnarray} \tag{ 5 }$$

The variance in the total flux is

$$\begin{eqnarray}&&\mathrm{Var}({f}_{{opt}})=\displaystyle \frac{1}{{\displaystyle \sum }_{j=1}^{m}{w}_{j}}\end{eqnarray} \tag{ 6 }$$

for both conventional optimal extraction and covariance-weighted extraction. In Section 4.3.4, we compare the results of conventional variance weights (Equation (1)) and profile weights (Equation (2)) that result in straight summation and covariance weights (Equation (4)).

4.3.4. Empirical Covariance Matrix

The ideal weighting of the spectrum from Equation (4) depends on the covariance matrix C_ij . We use a long dark exposure from the JWST Optical Telescope Element and Integrated Science Instrument Module (OTIS) test performed at NASA's Johnson Space Center on 2017 August 23 to estimate the read noise in C_ij . We process the data by performing CDS read pairs of long dark data (108 frames) with reference pixel corrections, column-by-column median subtraction, and row-by-row median subtraction. These steps will be performed on grism time series before extraction on an observation of a bright source with two groups per integration in the RAPID read mode (one frame per group). The resulting images from the column-by-column and row-by-row subtraction are similar to Figure 6 (right). We consider a wavelength bin that is 100 pixels in the dispersion direction (0.1 μm) and 14 pixels in the spatial direction.

Figure 8 shows the covariance matrix of this wavelength bin, taken from the median covariance matrix of 20 regions of the detector, where each region is 100 pixels in the dispersion direction (X) and 14 pixels in the spatial direction (Y). The diagonal elements ∼150 (e⁻)² are smaller than published NIRCam detector performance tables (σ² ≈ 13² (e⁻)² =169 (e⁻)² for CDS mode). However, if we skip the row-by-row and column-by-column subtraction step, they are consistent with σ² ≈ 13² (e⁻)² = 169 (e⁻)². It is clear that 1/f noise causes covariance most prominently along the spectral direction (the fast-read direction), with a covariance near 20 (e⁻)². Surprisingly, there is also a covariance between spatial pixels (the slow read direction) of ∼10 (e⁻)². The average covariance profile in the dispersion direction shows jaggedness to it in Figure 8 where the covariance rises and falls every other pixel with a peak-to-peak amplitude of ∼4 (e⁻)². This is the even/odd column offset discussed as noise contribution 4. Note that the read-noise covariance matrix described here is for the CDS mode, equivalent to when there are two groups per integration. The covariance matrix elements will decrease by a factor of approximately $\sqrt{2/{N}_{G}}$ where N_G is the number of groups.

Zoom In Zoom Out Reset image size

The empirical covariance matrix from Figure 8 is used to estimate the optimal pixel weights using Equation (4). We reduce the noise and jaggedness of the empirical covariance matrix in Figure 8 by averaging the value along the diagonal, across all spectral pixels and across the spatial pixels with the same value of i − j. Here, the profile is the value from pynrc (Leisenring 2019), which uses webbpsf (Perrin et al. 2014). We assume that the photon and background noise is $\sqrt{{N}_{\mathrm{phot}}}$ and that they are completely uncorrelated from one pixel to the next, so the covariance matrix is the sum, in quadrature, of the empirical read-noise covariance matrix and a diagonal photon noise matrix. To calculate N_photon we assume a source is bright enough that the maximum well depth in the spectrum image is 60% of the well depth.

The resulting optimal weighting function w_j /P_j is plotted in Figure 9. This optimal weighting significantly increases the contributions from the inner pixels compared to variance weights. The optimal weighting scheme also negatively weights some pixels outside the core of the PSF to reduce correlated noise.

Zoom In Zoom Out Reset image size

Conventional optimal variance weighting slightly lowers the contribution of read noise to the overall error budget. We test this conclusion with a series of grism images comprised of a constant A0V star spectrum added to the CDS read pairs, but do not add photon noise to focus on the read noise contribution. We do background subtraction along the dispersion and spatial directions, as would be done in a spectroscopic extraction pipeline. We perform sum extraction over a 14 spatial pixels and then combine together 165 spectral pixels into a 0.165 μm wavelength bin. The standard deviation of sum extraction (black weights in Figure 9) is 3860 DN, which is 2.7 times the read noise, as shown in Table 1. The standard deviation of the extracted flux drops to 3060 DN if conventional variance weighting (purple weights in Figure 9) is used.

Optimal covariance weighting significantly lowers the contribution of read noise to the overall error budget. We use Equation (4) (red weights in Figure 9) to reduce the read noise. For the covariance weights, we assume constant off-diagonal matrix elements of 15 (e⁻)² and do not include the spectral covariance. The resulting standard deviation of 850 DN is now 60% of the photon noise, but still above ideal read noise limit of 350 DN, due to remaining pixel correlations. Thus, we recommend optimal covariance weighting when extracting grism time series to dramatically decrease the 1/f noise contribution. As the number of groups of the ramp is increased, the read noise should drop as 1/$\sqrt{{N}_{G}}$ so the covariance weighting is most important when N_G is small.

The methodology in calculating P_j and Cov(S_i , S_j ) can significantly affect the precision on the time series. Typically P_j and the photon noise contribution to Cov(S_i , S_j ) can be found by fitting smooth functions (polynomials) along the dispersion direction. If P_j is calculated from each integration individually, it will inherit the 1/f noise variations of each integration. Therefore, we hold P_j constant along the time series with a value saved from the middle image at all wavelengths. Alternatively, P_j could be calculated from a PSF library or theoretically from webbpsf (Perrin et al. 2014). As we discussed in Section 4.3.2, one can also use a very small spatial aperture to reduce the contributions from 1/f noise. However, the optimal covariance approach allows one to use a wider aperture that will be less sensitive to pointing jitter and wave front variations.

So far, we have discussed the magnitude of correlated read noise on an integration-by-integration basis, but here we explore how it will impact science targets. We calculate the expected noise in the transit depth or eclipse depth of a generic exoplanet. For this calculation, we assume a planet orbits a G-type 5800 K star with a 3 hour transit duration is observed using a 2048 × 256 STRIPE subarray and is read out in RAPID mode with one frame per group and four amplifiers (number of output channels = 4). We consider a range of K magnitudes from 6 to 11. We then calculate the maximum number of groups up the ramp that can be commanded while keeping the peak pixel count below 50% well depth to safely avoid severe nonlinearities. Higher percentages of well depth, such as 80% or 90%, are possible to fit more groups in and thus increase the signal-to-1/f-noise ratio. The trade-off is that detector nonlinearities may go uncorrected by the pipeline or there could be reciprocity failure. Here, we assume a conservative 50% value to also ensure that inaccuracies in the JWST NIRCam throughput or a source's absolute calibration does not risk saturating the brightest pixels. From there, we calculate the noise due to photon counting statistics (i.e., $\sqrt{{N}_{\mathrm{phot}}}$) and compare it to the read noise. As in Section 4.3, we focus on a 0.17 μm wide bin centered at 3.34 μm

Figure 10 shows the photon noise as a function of K Vega magnitude. The photon noise scales as $\sqrt{f}$, where f is the flux because it is governed by Poisson statistics. The read noise drops slowly with K magnitude because the time to reset between integrations takes a smaller and smaller fraction of the total exposure as the number of groups per integration grows. We show the read noise assuming sum extraction using Equation (3) (1044 ppm for CDS integration) and covariance-weighted extraction using Equation (4) (230 ppm for CDS integration). If a sum extraction is used, the read noise will dominate for K ≲ 8.7. However, covariance weights can ensure that read noise is smaller than photon shot noise for K > 6.0.

Zoom In Zoom Out Reset image size

Some of the most interesting targets will be nearby bright sources, where the photon noise is small and the feature size is small. For context, the feature size of atmospheric signatures on archetypical planets are expected to be ∼300 ppm for hot Jupiters and Neptunes but only 10 ppm for cool super-Earths like K2-3 b if they have high-metallicity atmospheres (Greene et al. 2016). Therefore, observations of transiting cool super-Earths will require careful treatment of the correlated read noise, such as the covariance-weighting scheme, to achieve the precision needed to probe their atmospheres.

We note that Figure 10 shows a somewhat counter-intuitive result: read noise dominates for bright sources and becomes less important for faint sources. The reason is that for exoplanet transit observations, the detector well depth is filled to approximately the same level (∼32,000 DN or 50% well depth per pixel) regardless if the system's magnitude is 6 or 11. For the bright source with a few groups up the ramp, the read noise will be a larger fraction of 32,000 DN than for a faint source with many groups up the ramp.

It is helpful to review the basic operation of a near-infrared detector (e.g., Rieke 2007) in order to understand the systematics affecting time-series data. Figure 12 shows a schematic of an NP semiconductor junction for a H2RG detector inspired by Smith et al. (2008). The crystalline HgCdTe structure has four valence electrons that form covalent bonds in a tetrahedral structure. By varying the relative amount of Hg, Cd, and Te (doping), it is possible to increase or decrease the number of electrons to create mobile charge carriers that move along the crystaline lattice. The N-type semiconductor is a negatively doped HgCdTe crystalline structure that has negative charge carriers (electrons). The P-type semiconductor is a positively doped HgCdTe crystalline structure that has positive charge carriers (vacant holes in the lattice structure than can move like electrons).

Zoom In Zoom Out Reset image size

Here, we discuss how the N-type and P-type semiconductors operate as a photodiode. The junction of these two HgCdTe materials enables electrons to flow from the N-type material to the P-type material where it can complete a valence shell in the semiconductor material. This produces a negative charge on the P-type side. Conversely, the positively charged holes from the P-type material flow to the N-type material to complete the valence shell and this produces a net positive charge on the N-type side. The resulting voltage difference is called the contact potential.

In the region where electrons have completed the valence shells in the P-type material and holes in the N-type material, the valence shells are complete. This region spanning across the two semiconductor materials is called the depletion region (pictured in Figure 12), because it is depleted of mobile charge carriers. The positive net charge on the N-type material and negative charge in the P-type material causes a voltage (potential) difference across the detector. It should be noted that the depletion region is depleted in mobile charge carriers, but is not depleted in net charge because it has a net positive charge in the N-type depletion region and negative charge in the P-type depletion region.

The NP junction in Figure 12 is reverse biased with a positive potential on the N-type material and negative potential on the P-type. The positive potential on the N-type material attracts the negative charge carriers (electrons) whereas the negative potential on the P-type material attracts the positive charge carriers (holes). The reverse bias thus reduces the number of mobile charge carriers and increases the size of the depletion region. The increased depletion region size also increases the well depth or capacity of electrons that the detector may collect before saturation. The reverse bias is applied at each reset at the beginning of a detector's integration. Figure 12 shows the initial reverse voltage that is applied during a pixel reset. The physical size of the N-type HgCdTe semiconductor is much larger than the P-type HgCdTe because the density of carriers is lower in the N-type semiconductor.

Note that Figure 12 shows the mobile charge carriers only. Textbook NP junction schematics often show the net charge and thus symbolize the regions with charge carriers as the empty and the depletion regions with "+" symbols on the N-type semiconductor and "−" symbols on the P-type (e.g., Halliday et al. 2004). The schematic in Figure 12 instead shows the mobile charge carriers as "+" and "−" because it is easier to visualize the charge trapping effect and motion of mobile charges this way (e.g., Smith et al. 2008).

When an incoming photon strikes the N-type depletion region, it will excite an electron from the valence to conduction band which creates an electron–hole pair (Rieke 2007). The depletion region has a net electric field pointed toward the P-type side, which controls the direction electrons and holes will migrate. The electron moves to the extremity of the depletion zone in the N-type material whereas the positive hole generated crosses the NP junction and adds a mobile charge to the P-type side. The incoming photon has the effect of decreasing the size of the depletion region. A second photon will excite another electron–hole pair so that a hole moves to the P-type side. This continues and decreases the size of the depletion region. The diffusion of charge carriers across the junction changes the voltage across the NP junction, which can be measured with a sensitive amplifier circuit.

As the size of the depletion region decreases, the detector becomes decreasingly sensitive to new photons. Thus, the NIRCam HgCdTe detectors are never strictly linear. Eventually, with enough photons, the depletion region is too small to create electron–hole pairs with incoming photons. This is where the detector approaches saturation and no longer changes voltage with more light.

The accumulated charge from absorbed photons in the detector is measured with a multiplexor (MUX) that forms a readout integrated circuit (ROIC; Rieke 2007). This readout circuitry enables the signal to be measured nondestructively up the ramp in so-called multiaccum mode.

A sensitive amplifier converts the measured voltage potential (as compared with the bias level after reset) into a signal. The electronic circuit that digitizes the voltages is tuned to ensure the 16 bits (65,536 possible DNs) measure the voltage across the NP junction from the value after reset (and before most photons are detected) to the saturation voltage where the maximum number of photons are collected and no more electron–hole pairs can be produced in the depletion region. The minimum voltage is several thousands of counts above zero to ensure that fluctuations in the bias signal do not go below zero. The 65,536 DN level is tuned to be just above the HgCdTe maximum well capacity to measure the full well capacity of a pixel.

We start by applying a smoothing kernel to each group (one read for RAPID mode) up the ramp of the raw data. The kernel is a uniform normalized kernel 1 pixel in height and 161 pixels along a row. This enables subtraction of 1/f noise that has length scales shorter than the line time (i.e., timescales shorter than 512 clock cycles or 5.12 ms), which the median subtraction cannot remove. Care must be taken not to apply the kernel through the source aperture, which would subtract and distort any signal from a star. The kernel should be wider than the source aperture diameter and interpolate through the source points.

We use the astropy.convolution.convolve() routine, which calculates a convolution of the kernel by the image. astropy.convolution.convolve() ignores all NaN values in the mask and renormalizes the kernel by the sum of the remaining kernel points. This efficiently interpolates over bad pixels and can also interpolate over the source aperture by creating a circle of NaN values around the source. This is similar to the row-by-row method where the pixels are selected outside the source aperture with no bright sources but it operates on timescales shorter than 5.12 ms. Our smoothing kernel is a flat boxcar normalized to 1.0. The resulting time series has a standard deviation of 2800 DN to 3700 ppm. This is slightly larger than the much simpler row-by-row amplifier-by-amplifier subtraction in Section 4.2.2.

We note that the four channels can sometimes exhibit the same 1/f correlation behavior, as shown in Figure 13 because they share a common reference voltage. Another option to remove 1/f noise is to subtract the three out of four output amplifiers that have the source on them by the one output amplifier without the target. We subtract the first amplifier's time series (pixels X = 1 through X = 512) from all amplifiers by shifting and flipping the 2048 × 256 image subsection to the corners of the four amplifiers at (X = 1, X = 513, X = 1025, and X = 1537). Of course, this means that this amplifier is perfectly subtracted and no noise or signal remains in this one-fourth of the image. The subtraction step should increase the read noise on an individual pixel (for the remaining three output amplifiers) by $\sim \sqrt{2}$ because the errors from the subtracted pixel should add in quadrature to the original pixels. This increase in read noise on a per pixel level might be acceptable if it decreases the correlated errors along the rows. On the other hand, if each amplifier exhibits its own individual correlated noise behavior, the reference amplifier will fail to subtract out all correlated noise and may introduce more noise.

Zoom In Zoom Out Reset image size

The net effect of this method is to increase the noise over a row-by-row median subtraction. The standard deviations in the time series range from 8500 DN to 10,800 DN for this reference amplifier method as opposed to the 5500 DN to 6600 DN for a row-by-row median. This means the correlations within an amplifier are not efficiently subtracted and may be propagated to the remaining amplifiers.

Another approach to treating the 1/f noise is dimensionality reduction with PCA. Here, we assume that the counts for each pixel along a column is a random variable that can be correlated with pixels in other columns. Each row represents a different noise instantiation. First, we apply reference pixel correction to remove the pre-amplifier resets. Next, we calculate the mean value of each pixel in the 108 group stack and subtract all groups by this mean image. This removes all bias levels and focuses on the time variability of the pixels. Here, we assume that the dark current is negligible. Next, we mask all bad pixels by setting all pixels with an absolute value larger than 200 DN and replace them with zero DN to remove bad pixels that can drive the principal components. The first 10 principal components are plotted in Figure 14 for two example groups (54 and 55).

Zoom In Zoom Out Reset image size

The first 10 principal components explain 14% of the total variance in each group up the ramp. The principal components from one group to another are different, but the ordering of these components generally follows a pattern of increasing frequency with higher-order principal components (smaller eigenvalues). This is expected for 1/f noise where the lowest frequencies have most of the power. Since the first principal components are different from one group to another, we calculated the principal components for each group independently. We then create a noise model image which is the matrix multiplication of the eigenvectors by the principal components for 10 components. This model is subtracted from each group to create a 1/f-corrected image.

The PCA subtraction is applied to each group up the ramp and then these groups are subtracted in pairs to create a time series. We use 10 principal components as a starting point for this method. The final time series has a measured standard deviation of 5300–5500 DN within the background-subtracted aperture. This is comparable to the row and column subtraction standard deviation of 5500–6500 DN. Adding 10 more principal components for a total of 20 eigenvectors only reduces the time-series noise to 5100–5400 for the three apertures.

In another approach, we treat each amplifier separately so that the principal components are calculated independently. This approach means that a noise signature in three amplifiers but not the fourth does not propagate to the fourth amplifier when subtracting the PCA-based model. This individual-amplifier model has four times the number of variables as the combined PCA approach, so we fit with only the first five principal components in each amplifier. The resulting eigenvectors are shown in Figure 15.

Zoom In Zoom Out Reset image size

If we treat each amplifier separately, the PCA model is more effective in reducing 1/f noise. We begin by keeping five principal components within each amplifier and subtracting these from each amplifier separately. The resulting standard deviation of the time series is 2500–3820 DN. This improvement is beyond what is possible on a real source because the principal components include pixels over the source aperture. To simulate a real photometry scenario, it is necessary to interpolate over the source aperture points as done in Appendix B.1 for the smoothing kernel. However, we stop the PCA at this point because it does not do as well as the simple row-by-row amplifier-by-amplifier mean subtraction performed in Section 4.2.2.