Overview of the Instrumentation for the Dark Energy Spectroscopic Instrument

The formal start of the project that became DESI occurred with DOE approval of Critical Decision Zero (CD-0) or Mission Need on 2012 September 18. DOE subsequently selected Lawrence Berkeley National Laboratory (LBNL) as the lead laboratory for DESI and appointed the LBNL Project Director. Extensive design studies began at this time and showed that the instrument would need an FOV of 2°–3° diameter, many thousands of fiber positioners, high throughput, and excellent PSF stability for sky subtraction. This period also coincided with significant research and development (R&D) and systems engineering activity. The R&D included the development of fiber positioners, study of alternative positioner technologies, work on spectrograph PSF stability that would be critical for sky subtraction, and further development of low-noise amplifiers for the CCDs. During this time period, we had many conversations with vendors about the corrector lenses, spectrograph optics, and VPH gratings. We also received generous support from the Gordon and Betty Moore Foundation and the Heising–Simons Foundation. This early financial support enabled us to begin procurements of critical, long-lead components, namely the glass for the corrector and the first spectrograph.

The systems engineering approach included an extensive model for all contributions to throughput and to system noise. Our throughput model included all of the relevant hardware components. These included the primary mirror and spectrograph collimator reflectance, transmission of optical elements, and antireflection (AR) coatings, as well as vignetting, blur due to optical misalignments, defocus between the fiber tips and the detectors used for guiding and wave front sensing, fiber system focal ratio degradation (FRD), lateral alignment errors, and the gratings and detectors. The noise included sky noise, scattered light in the telescope, corrector, and spectrograph, fiber crosstalk, and the detector noise. The project maintained a throughput and noise budget for each subsystem and a margin on all quantities. The allocations were initially based on estimates either derived from communications with vendors or based on calculations, and these estimates were replaced over time with measurements of the as-built components. Examples of measurements include the coatings and material absorption in the elements of the corrector, fiber system, and spectrographs, and the spectrograph detectors.

The project immediately began trade studies to support site selection. The two critical aspects of the instrument that especially influenced site selection were the total mass at prime focus and the focal ratio of the primary mirror. An instrument capable of observing tens of millions of galaxies in just 5 yr would be very massive, and only older-generation telescopes were expected to be able to support the expected mass of the instrumentation at prime focus. Numerous design studies showed that correctors with a 2°–3° diameter FOV were only feasible for telescopes with a primary mirror of f/2.5 or slower. These considerations narrowed the list of potential telescopes down to a short list in the 4 m class. Given this aperture size, the flux limit requirement, and survey duration, the instrument would clearly require extremely high throughput and at least a very substantial fraction of all of the dark time at the site. These technical factors and discussions about the programmatic availability of various candidates led to the selection of the 4 m Mayall Telescope in 2013. The Mayall Telescope is a Ritchey–Chrétien (RC) design that was built to support a prime-focus cage with an observer and a coudé focus, in addition to a hyperbolic secondary for the wide-field RC focus, and therefore could support substantial instrumentation at prime focus. The telescope is part of the Kitt Peak National Observatory and is located approximately 80 km WSW of Tucson, Arizona at an altitude of approximately 2021 m on the land of the Tohono O'odham Nation.

The next two milestones were CD-1 (Alternative Selection and Cost Range) on 2015 March 19, and CD-2 (Performance Baseline) on 2015 September 17. During these phases, we produced both a detailed design for the survey and the final instrument design, which were published as DESI Collaboration et al. (2016a) and DESI Collaboration et al. (2016b), respectively. We also developed detailed performance parameters including science and technical requirements, a project execution plan, risk registry, risk management plan, and other documents that are a standard part of DOE program and project management.

The total instrument cost was approximately $75M. At CD-2, the total project cost from DOE was set at $56.328M as spent dollars, and the date for project completion (CD-4) was set to the end of FY21. There were also $19M in foreign and private (including university group) contributions. This does not include the costs to operate the survey. U.S. Congressional approval for the start of DESI was included in the FY15 Energy and Water appropriations legislation. CD-2 approval also included authorization to begin procurements of long-lead items in advance of CD-3 (Start of Construction). Construction formally began with CD-3 approval on 2016 June 22. DESI achieved CD-4 (project completion) approval on 2020 May 8, 1 yr ahead of schedule and $2M under budget. This date marked the formal completion of the instrument development and construction phase and readiness to begin the survey. Table 2 summarizes key dates and milestones for the DESI Project. Table 3 lists the institutions that contributed to the instrumentation, along with their principle responsibilities.

There are four Level 1 science requirements that motivate the survey data set. These requirements set the survey area, the aggregate precision for how well DESI will measure the isotropic cosmic distance scale R(z) from BAO at 0 < z < 1.1, the precision for how well DESI will measure the Hubble parameter at 1.9 < z < 3.9, and the allowable size of systematic errors from instrumental and observational methods for the angular diameter distance and the Hubble parameter. The Level 1 requirements are listed in Table 4.

The four Level 1 requirements were used to develop Level 2 requirements on the survey data set. These are a series of requirements on the average density, on random, systematic, and catastrophic redshift errors, and on redshift completeness for the three dark time target classes. The Lyα QSO sample includes specific requirements on the S/N and on the contribution of calibration errors on the flux power spectrum. There are specific fiber completeness requirements on each target class that impact the density of positioners on the focal plane as well as the relative priorities of targets. The Level 2 requirements are also listed in Table 4. These were the requirements during the development of the instrumentation. Some modifications to the Level 2 requirements were made after the instrumentation was complete, most notably the addition of requirements for BGS. No requirements related to BGS influenced the instrument design.

The L2 requirements flowed down to 11 L3 requirements, which are listed in Table 5. These are divided into three categories: implementation requirements, programmatic requirements, and environmental requirements. The six implementation requirements directly flow to the Level 4 technical requirements on the subsystems. The two programmatic requirements relate to the survey area and the input catalog. The three environmental requirements include the image quality, range of zenith angle, and telescope guiding.

Table 5. Level 3 Imposed and Derived Technical Requirements

Number	Text
L3.1	Implementation
L3.1.1	The spectral range shall be 360–980 nm.
L3.1.2	Spectral Resolution shall be (a) >1500 at wavelengths 360 < λ < 555 nm, (b) >3000 at wavelengths 555 < λ < 656 nm,
	and (c) >4000 at wavelengths 656 < λ < 980 nm.
L3.1.3	The median S/N = 7 flux limit will be 10, 9, 9, 8, and 9 × 10−17 erg s−1 cm−2 in redshift bins of 0.6–0.8, 0.8–1.0,
	1.0–1.2, 1.2–1.4, and 1.4–1.6 for an [O ii] doublet emission line in an ELG with an exponential half-light radius of
	045 observed in 11 seeing with the sky spectrum under median, dark-sky, photometric conditions.
L3.1.4	The fiber density shall be less than 700 per square degree.
L3.1.5	The FOV shall be no less than 7.65 square degrees.
L3.1.6	The spectroscopic PSF shall be characterized for all fibers in each science exposure over the full wavelength range such
	that the PSF bias shall not exceed 1%.
L3.2	Programmatic
L3.2.1	The 9000 deg2 survey shall be completed in 4 yr including 6 months commissioning and validation. A goal is
	14,000 deg2 survey in the same period but not more than 5 yr plus commissioning and validation.
L3.2.2	A target galaxy and QSO catalog shall be assembled to a depth of r = 23.4 mag with astrometric errors not exceeding
	100 mas rms for each target class.
L3.3	Environmental
L3.3.1	Median seeing shall be assumed to be 11 FWHM, characterized by a galaxy Moffat profile with β = 3.5.
L3.3.2	DESI shall meet all of its requirements while observing with zenith angles between 0° and 60°.
L3.3.3	Telescope guiding accuracy shall be assumed to be 100 mas rms.

Note. Level 3 Imposed and Derived Technical Requirements, which are motivated by the top-level science requirements (Level 1) and the survey data set requirements (Level 2) that are listed in Table 4. The L3.1 Implementation requirements are the top-level requirements on the instrumentation, including the overall throughput, fiber density, and spectral range. The L3.2 Programmatic requirements constrain the survey design. The L3.3 Environmental requirements describe the expected performance of the instrument and telescope. We made some modifications to the Level 3 requirements after the instrumentation was complete.

Download table as:  ASCIITypeset image

DESI utilizes a spectral range of 360–980 nm (requirement L3.1.1) because of the Level 1 redshift range requirement. Specifically, the short wavelength cutoff of 360 nm aids in the identification of the Balmer break at 364.6 nm in very low redshift interlopers that could contaminate the target classes. This limit also enables the detection of the Lyα emission line at the observed wavelength of 377 nm at z = 2.1, the entire Lyα forest between Lyα and Lyβ at z > 2.5, and the use of the Cd 361.0 nm arc lamp line for wavelength calibration. The long-wavelength cutoff enables detection of the [O ii] doublet in ELGs up to z = 1.6, which corresponds to the observed wavelength of 970 nm. This long-wavelength cutoff also enables more secure redshift measurements for other targets; most notably the rest-frame 400 nm break feature in LRGs can be measured up to z ∼ 1.4.

The spectral resolution λ/Δλ requirement L3.1.2 flows from the requirements on redshift accuracy, precision, and on the fraction of catastrophic redshifts, especially for the ELG sample. The resolution is sufficient to identify the [O ii] doublet as a pair of lines for galaxies at z > 0.49. This is important as the [O ii] doublet will usually be the only detected spectral feature, and the resolution of the doublet is critical for an unambiguous redshift measurement. The spectral resolution may be less at lower redshifts (z < 0.49) because at those redshifts the Hα line is also visible to enable a secure redshift.

The S/N = 7 requirement on the [O ii] doublet (L3.1.3) at various fluxes and redshifts sets joint requirements on total instrument throughput, the resolution, read noise, and the environmental conditions. These limits are set in redshift bins because the [O ii] doublet is observed through a thicket of night sky emission features that produce a modulation of the flux limit as a function of redshift within each Δ z = 0.2 redshift bin. The fiber density requirement (L3.1.4) and the FOV requirement (L3.1.5) are motivated by the expected density of targets and the number of spectra required to meet the science requirements. They flow from three Level 2 requirements: the complete survey area A_survey, the minimum time required to observe any one field or tile T_tile, and the total survey duration T_survey. Since a circular FOV does not perfectly tile the sky, DESI initially adopted a correction factor of F_over = 1.21 for the amount of overlap with neighboring tiles, although subsequently changed to a Hardin et al. (2001) icosahedral tiling with multiple layers before the start of the survey. Accurate knowledge of the 2D spectroscopic PSF shape is required to extract the spectra and obtain unbiased measurements of the flux, noise, and resolution. We characterize uncertainty in the PSF measurement as the PSF bias, which can produce artifacts in spectra, such as distortions of absorption or emission features, or lead to mis-modeling of the sky subtraction derived from other spectra in the same exposure. Requirement L3.1.6 sets a limit on the PSF bias.

The four programmatic requirements set the area and duration of the bright and dark time surveys, the conditions under which the dark time program will be executed, the astrometric errors of the input catalog, and the uniformity of the data. Requirement L3.2.1 on area and duration is driven by total operation costs. This in turn sets requirements on throughput and the number of targets per exposure, and it was also used to set a requirement on the inter-exposure time of 2 minutes. The inter-exposure time is the time between closing the shutter on one spectroscopic exposure and opening the shutter to start the next. The inter-exposure time requirement impacts nearly every subsystem, as between exposures we read out the previous spectroscopic exposure, move the telescope and dome, adjust the atmospheric dispersion corrector (ADC), adjust the hexapod, adjust the fiber positioners for the next asterism, acquire the new field, and start guiding. Requirement L3.2.2 quantifies the allowable astrometric errors, which flow down from the target densities for each target class and the total fiber positioning error budget. This requirement is met by the Legacy Survey's data set (Dey et al. 2019).

The three environmental requirements are on the median seeing, zenith distance range, and telescope guiding. The median seeing (requirement L3.3.1) is motivated by the historical performance of the Mayall with the MOSAIC instrument (Dey & Valdes 2014), and effectively, it is a requirement that upgrades to the telescope do not negatively impact the seeing. This requirement impacts the optical design, throughput calculation, and exposure time per tile. Requirement L3.3.2 states that DESI shall meet all requirements while observing between zenith angles from 0° to 60°. This is motivated by the survey footprint and the flexibility needed to target fields at a range of airmass. This requirement impacts the optical design (most notably the ADC), throughput, and exposure time. Finally, L3.3.3 requires the telescope guiding accuracy to be 100 mas rms. This is also motivated by the historical performance of the Mayall Telescope and is a component of the fiber positioning error budget.

The management of the DESI project followed standard DOE guidelines for similar projects. The project was led by the LBNL Project Director, who was responsible and accountable for the successful execution of the construction. The Project Director also represented the project to the DOE, LBNL, and other participating institutions. The LBNL Project Manager was responsible for the management and safe execution of the DESI Project. This includes adherence to the technical, cost, and schedule baselines, and risk and integrated safety management.

DESI includes two co-Spokespersons. The Spokespersons represent the science collaboration and are responsible for encouraging the productivity of the collaboration, the performance of the science working groups, and advising the Project Director. The Project Director is also advised by an Executive Board, who are appointed by the Project Director from the senior membership of the project and the science collaboration. They provide advice on the scientific scope, mission objectives, operational priorities, and membership issues.

In addition to the Director and Project Manager, the DESI Project Office included two Project Scientists, an Instrument Scientist, Commissioning Scientist, Systems Engineer, and Safety Officer during the construction phase. ⁹⁷ The Project Scientists were responsible for maintaining the science requirements, evaluating the scientific impact of any changes to hardware during construction, preparation of software, development of the survey strategy, and the overall plan for commissioning and operations. The responsibilities of the Instrument Scientist were to ensure that the instrument met the technical requirements, evaluate the technical impact of changes or trade-offs during construction, and plan for commissioning and operations. We created the Commissioning Scientist role as we began detailed commissioning planning because we realized that an additional, high-level position would be helpful to lead this effort. The Commissioning Scientist was responsible for commissioning planning, and the organization of commissioning observations and analysis. The Systems Engineer was responsible for high-level systems engineering and value management oversight of all aspects of the project, such as interface control documents (ICDs), maintenance of budgets for system-level requirements such as throughput, and technical alternative studies to determine the best design solutions. The DESI Safety Officer was responsible for ensuring that Environment, Health, and Safety concerns are addressed, and that integrated safety management was implemented in all phases of DESI construction.

The DESI Project Manager leads the Technical Board, which, during construction, provided information on progress toward scientific and technical goals, along with budget and schedule information, and presently focuses on performance during operations, maintenance, and potential improvements. During construction, the Technical Board included the Project Director, Project Manager, Project Scientists, Instrument Scientist, Systems Engineer, WBS Level 2 Managers and Cognizant Scientists, and the Project Safety Officer. Each Level 2 Manager was paired with a Cognizant Scientist to efficiently address the scientific impact of technical issues. The Systems Engineer, along with the Project Director, Project Manager, and Project Scientists formed the Change Control Board, which evaluated changes to technical scope. The Level 2 Managers were responsible for the completion of the tasks in their subsystem. This included the identification of the resources necessary to complete the subsystem within the projected schedule and budget, meet the subsystem design requirements, and conform to interfaces with other subsystems.

The corrector optical design was guided by a series of Level 4 requirements, as well as a set of Level 5 specifications with implementation details. The key Level 4 requirements fall into several broad classes: wavelength range, image quality, throughput, and coupling with the fiber system. The Level 3 requirement on the bandpass of 360–980 nm applies to both the image quality and throughput. The image quality requirement on the FWHM at zenith is a field-weighted mean of <04 over the full bandpass, and a maximum of <06 from 360 to 450 nm and <05 from 450 to 980 nm. At 60° from zenith, the maximum FWHM is <075 and <06 over these two wavelength ranges. The image quality requirement was one of the most significant design challenges, as it required the minimization of lateral chromatic aberrations over this entire bandpass. In addition, atmospheric dispersion at 60° from zenith corresponds to several arcseconds between the wavelength extremes, which constrained the design of the ADC system. The throughput specification on the optical design only applied to the glass transmission, as there was a separate requirement on the performance of the optical coatings. The glass throughput requirement varied with wavelength, and the coating requirement had an average transmission of 98.5% per surface over the entire bandpass. There were also specifications on the homogeneity of the glasses, as inhomogeneity can degrade image quality.

Another significant design challenge was the requirement that the focal surface must be sufficiently large to contain the 5020 fiber positioners, as well as the Guide/Focus/Alignment (GFA) cameras and fiducials, and have an angular FOV of at least 3° (driven by requirement L3.1.5, as described above in Section 2). The chief ray deviation at the focal surface was also constrained to be <05 on average and <1° maximum in order to obtain good coupling into the fiber cables and avoid substantial tilts of the fiber positioners that would both complicate the focal surface mechanical design and impact the filling factor of the positioners on the focal surface. One compensation to the complexity of the optical design requirements was that the focal surface did not need to be flat. Instead the requirement was that the radius of curvature must be greater than 3000 mm (convex toward the sky). To limit the complexity of lens fabrication and testing, we imposed requirements to have no more than one aspheric surface per lens and to limit the maximum aspheric departure to no more than 30 mrad. The constraint on the maximum aspheric departure was recommended by potential vendors in order to allow interferometric testing with better than Nyquist fringe sampling. There were some early studies that attempted an all-spherical design, but those could not produce more than half the FOV of the final design.

The corrector has four large fused silica elements. C1 and C4 are the largest lenses, both being more than 1 m in diameter with two spherical surfaces. C1 has a diameter of 1.14 m and mass of 201 kg, and C4 has a diameter of 1.03 m and mass of 237 kg. C2 and C3 are both approximately 0.8 m in diameter, and the first surface of both lenses is an even asphere. The ADC lenses are ∼0.8 m diameter prisms figured from borosilicate, and each has two wedged, spherical surfaces (∼025) for first-order correction of atmospheric dispersion away from zenith. The total mass of glass in all six lenses is 864 kg. ⁹⁸ There are no bonded joints in the ADC in order to reduce the system complexity and risk. The dispersion magnitude is set by the rotation of the ADC lenses relative to one another, and the direction of the correction is set by their net orientation relative to the parallactic angle. The optical system is asymmetric due to the presence of the wedged, spherical ADC elements. We consequently characterized the design image quality across the FOV to verify performance, and not simply as a function of radius, and set separate requirements at the zenith and at 60° from the zenith.

The corrector causes a central obscuration that impacts the throughput, and the finite length of the corrector produces a field-dependent shadow. The design includes some allowance for vignetting of the larger C1 and C4 elements in order to reduce the mass and size of those elements. This vignetting is present starting at 145 off axis and is 5% by the edge of the FOV at 16.

The efficiency of coupling to the fiber cables is greatest when there is minimal deviation of the chief ray ⁹⁹ from the local fiber normal. The mounting holes for the fiber positioners, described in Section 5, were machined to orient the fiber tip at the optimal angle for their location on the aspheric focal surface, rather than simply normal to the surface. The deviations between the surface normal and the chief ray meet the requirement that they are less than 05 on average and less than 1° at all field positions. The chief ray deviation at 500 nm is shown in Figure 5.

Zoom In Zoom Out Reset image size

The design does not include a constraint on the distortion of the pupil, although the shape of the pupil is both useful to quantify the projected size of the fibers on the sky and as input to the optical design of the spectrograph collimator. The focal ratio is f/3.68 at the center of the field and increases toward the edge. There is some anamorphic distortion, such that there is a somewhat larger increase in the meridional plane direction than in the sagittal plane. The net effect of this distortion is that while the 107 μm diameter fibers are 1585 diameter circles on axis, they are 139 × 151 (meridional × sagittal) ellipses at 165 (420 mm) from the center of the field. The variation of the f/# and plate scale with radius is also shown in Figure 5.

The final optical design, called the "Echo 22" design by the project, meets all of the requirements. The two aspheric surfaces are well within the maximum aspheric departure requirement of 30 mrad, with peak values of <15 mrad and <11 mrad for C2 and C3, respectively, with the maxima at the edge of the lenses. The corresponding magnitudes of the aspheric departure were about 1.1 mm for C2, 0.3 mm for C3, and about 0.2 mm for the focal surface. The optical design of the corrector is presented in Miller et al. (2022, in preparation).

The lens blanks for C1 and C2 were produced by Corning, the lens blanks for C3, C4, and ADC1 were produced by Ohara Corporation, and the blank for ADC2 was produced by Schott North America, Inc. The lens blank orders were placed between 2014 July and 2015 January. All were delivered by the end of 2014 except for ADC1, which was delivered in 2015 August, and ADC2, which was delivered in 2015 July. Because poor glass homogeneity (variation of glass index) directly degrades image quality, glass homogeneity was a key performance driver for the lens blanks and we gave vendors a maximum inhomogeneity requirement. Furthermore, we obtained measurements of the glass homogeneity along multiple sightlines through each blank from the vendors and modeled the impact relative to the beam footprint to determine that the delivered inhomogeneity produced an acceptable amount of blur within the corrector budget.

The two all-spherical lenses C1 and C4 were polished by L3 Brashear, and the two aspheres C2 and C3 were polished by Arizona Optical Systems. Rayleigh Optical Corporation polished both ADC lenses (Miller et al. 2018). The polishing vendors supplied all test procedures and data (e.g., interferometric surface maps, wedge measurements) that were used to verify requirements, and we performed an independent evaluation as part of the acceptance process. The two spherical lenses were delivered in 2016 January, and the two aspheric lenses were delivered in 2017 June and October. The spherical lenses consequently took approximately 1 yr from the order of the blanks through polishing, and the aspheres took about 3 yr. The purchase of the lens blanks and the polishing of C2 and C3 were initiated prior to CD-3 approval for the start of construction with support from the Gordon and Betty Moore Foundation and the Heising–Simons Foundation. Given the long procurement times, this early support was crucial to the timely completion of DESI.

The six lenses were coated by Viavi Solutions, Inc. with ion-assisted deposition techniques in a 3 m coating chamber (Kennemore et al. 2018) through funding by the National Council of Science and Technology, Mexico (CONACYT). The size of the lenses and the shapes of the surfaces presented numerous challenges for coating. One was that the radius of curvature of the 12 surfaces ranges from nearly flat to a radius of curvature of ∼600 mm, with a sag of 140 mm. Another is that the angle of incidence for one lens surface varies from nearly normal to 40°. The vendor created a coating design with a wider bandwidth than required in order to compensate for the nonuniformity and angle variation and thus was able to employ a single design for all surfaces with the same substrate. The coatings were measured as averages over 110 mm diameter circular areas. Figure 6 shows average measurements of the coatings on both surfaces of all six lenses. All of the coatings exceeded the requirement of >98.5% averaged over the bandpass. At the extremes of the bandpass, one coating is just barely noncompliant with the >98% minimum. We evaluated this noncompliance and determined that there would be a minimal impact on performance, as the other coatings exceeded the requirements.

Zoom In Zoom Out Reset image size

The corrector meets all requirements. Requirements such as wavelength range, glass transmission, image quality, FOV, size of the focal surface, chief ray deviation, and performance up to 60° from zenith were met by the optical design. The total throughput set requirements on the performance of the coatings, which were met, as shown in Figure 6. The image quality also set requirements on the alignment of the lenses in their cells, the alignment of the cells within the corrector barrel, and the alignment of the barrel with the primary mirror. These requirements are discussed further in the next section on the corrector support system. The final verification of the corrector performance was the demonstration on sky that the image quality requirements were met. Early data were obtained with an r-band filter, as shown in Figures 3 and 4. The performance over the full wavelength range across the entire FOV was ultimately verified with measurements of the total throughput, which is described in Section 13.1.

There are three main classes of requirements on the corrector support system. The first is that all performance requirements need to be met over zenith distance angles from 0° to 60°, which flow from the Level 3 technical requirement L3.3.2 described in Section 2.3. This requirement impacts the design and analysis of all of the components through a joint constraint on the mass and stiffness of this system. The project separated this constraint by establishing an overall mass budget for the upper assembly structure. This mass budget applied to all items installed for DESI that rotate about the decl. axis. This includes the components in this section (barrel, cage, and ring), the lenses described in Section 3.2, the focal plane system in Section 5, and a fraction of the mass of the fiber cables (see Section 6). The total budget corresponds to 10,700 kg at the center of the upper ring, which is 8.5 m from the decl. axis, and corresponds to a minimal change in the mass at the top end of the telescope. That is possible because the larger mass of the corrector and the substantial mass of the focal plane system are approximately compensated by the mass removed with the replacement of the original split top ring with a single ring and the removal of the secondary mirror support structure. As a point of comparison, Dark Energy Camera (DECam) is more than 2500 kg heavier than the previous camera and cage assembly on the Blanco Telescope (Flaugher et al. 2015).

The mass budget and zenith distance range requirements flowed down to requirements on the displacement of the optics due to gravity. There are both static and dynamic lateral and tilt tolerances for the six lenses, and there are additional tolerances between the lenses and cells and between the cells and the barrel. There are also tolerances on the interface flange to the focal plane system. The static tolerances are alignment tolerances and were measured with either a coordinate measuring machine (CMM) or optical metrology with a rotary table in a climate-controlled environment at a specified temperature. The dynamic tolerances limit flexure due to gravity and were calculated with finite element analysis. The total tolerances include both static and dynamic tolerances and both lens-to-cell and cell-to-barrel tolerances. The total lateral tolerances ranged from 75–200 μm, and the total tilt tolerances ranged from 105–250 μrad. There are also requirements on gravity sag and tilt of the C3 mounting flange that connects to the hexapod flange.

The ADC requirements included continuous rotation, rotation rate and accuracy, and the stability and lifetime of the mechanisms. The requirements on the continuous rotation and the rotation speed were motivated by the requirement that the inter-exposure sequence not exceed 2 minutes. Continuous rotation avoids the need for rotations greater than 180° between exposures, and a rotation speed of up to 10° per minute was adopted to ensure that the ADC motion would not contribute to the inter-exposure time. The lifetime requirement was primarily motivated by the bearings, as the motors can be readily accessed for maintenance.

The new prime-focus cage, vanes, and ring were fabricated by CAID Industries, Inc. in Tucson, AZ, who delivered these components to the Mayall. The new top ring mounts to the telescope's Serrurier truss via eight truss members that attach to the ring at four locations, spaced at 90° intervals (e.g., see Figure 1). The top ring provides the interface between the Serrurier truss and the prime-focus instrumentation, as well as a convenient location to mount the calibration lamps. The prime-focus instrumentation is supported by a new prime-focus cage that is constructed from low carbon steel and attached to the top ring with four sets of three vanes. The vanes have adjustment features to center the cage on the optical axis of the primary mirror. There is a requirement that the roll of the cage must be less than 8 μm at the edge of the focal plane during the longest expected DESI exposure times of the order of 20 minutes. To minimize roll, the vanes are not connected to the cage at 90° increments, but rather are displaced by 18 cm along the circumference of the cage such that pairs of vanes move in opposite directions. Another key requirement is that the vertical sag of the cage's position relative to the top end of the Serrurier truss must be less than 0.9 mm at zenith so that it is well within the adjustment range of the hexapod. There are also requirements on the sag and tilt at the Horizon that were developed in conjunction with the specifications for the adjustment range of the hexapod.

The prime-focus cage has three rings that are held in place with four rails. The ring that is closest to the primary mirror supports the light baffles, the middle ring supports the hexapod system and thus the weight of the corrector and focal plane system, and the top ring supports the mass of the focal plane enclosure (FPE). The cage has an open design, which provides easy access to the hexapod and ADC mechanisms. The cage includes covers that protect the barrel from wind and thermal radiation. These components were painted with an optical black Aeroglaze paint, with the exception of precision mating surfaces. The new prime-focus cage and instrumentation for DESI is somewhat larger than the previous cage, and it is larger than the central hole of the primary mirror. As a result, the effective area of the telescope is 8.658 m² on axis.

The hexapod is designed to adjust the barrel in six degrees of freedom to maintain the optical alignment of the prime-focus instrumentation with the optical axis of the primary mirror. For a Cartesian coordinate system with the z-axis aligned with the optical axis, the degrees of freedom are x- and y-decenter, tilt about the x- and y-axes, translation along the z-axis (focus), and rotation about the z-axis. The hexapod has six actuators that form three triangles, and the actuators at the base of each triangle are attached to a plate that is directly attached to the cage. The hexapod is controlled by the active optics system (AOS), which is described in Section 8.5.5. Figure 2 shows the hexapod attached to the corrector barrel.

We purchased the hexapod from ADS International, and it met all of our requirements. These included that the ranges (resolutions) are ±8 mm (±10 μm) for x- and y-decenter, ±10 mm (±5 μm) along the z-axis, ±250'' (±15) in tilt around the x- and y-axes, and ±600'' (±4'') in rotation about the z-axis. The DESI hexapod is very similar to the hexapod used for DECam (Flaugher et al. 2015), with the main difference being that the hexapod for DESI has a somewhat larger diameter (1.75 m versus 1.55 m). The hexapod was shipped directly to the Mayall Telescope and was integrated with the corrector barrel during installation.

The corrector barrel was fabricated from carbon steel by Dial Machine in Rockford, IL. It is divided into three main sections to support integration and testing: the front section includes the C1 and C2 lenses, the middle section includes the two-element ADC and associated mechanisms, and the aft section includes the C3 and C4 lenses (e.g., see Figure 2). The hexapod is mounted to the cage, and attaches to the aft section of the barrel. There is also a section called the shroud that attaches to the front section. The shroud extends somewhat beyond C1, and its primary purpose is to protect that lens. It includes a provision for a protective cover, as well as references for alignment. The barrel also includes a light baffle that faces the primary mirror, and an adapter that supports the majority of the mass of the focal plane system.

The lens cells for C1, C2, C3, and C4 were constructed from a nickel-iron alloy, and the cells for the ADC lenses were constructed from low carbon steel. All of the cells were fabricated by University College London (UCL). Each lens is attached to its corresponding cell with axial and radial pads. The thickness of the radial pads for each cell was customized to accommodate the differences in the coefficient of thermal expansion (CTE) between the lens and cell materials, essentially an athermal design. The nickel-iron cells incorporate flexures to allow for the difference in CTE between the cell and steel barrel. The axial pads in the cells both support the lenses over the full range of gravity vectors (0°–60° zenith angle) and account for surface irregularities that could produce uneven loading of the lenses.

The team at Fermilab meticulously measured the barrel and cells prior to their shipment to UCL. The measurements were to verify the static deflection tolerances and to provide key information in advance of the optical alignment at UCL. The static deflection measurements of the complete barrel were made with a large CMM that Fermilab built for this project. This temperature-controlled facility was used to measure the assembled corrector barrel at a range of orientations relative to gravity. A smaller CMM was used to obtain precision measurements of the lens cells, lens spacers, axial and radial pads, and the barrel flanges. These tests also verified that the barrel could be reassembled to a position accuracy of ∼6 μm by use of a Moglice pinning system.

The tolerance requirements on the ADC lenses apply to the ADC mechanism as well. Each ADC cell is mounted within a large bearing from Kaydon. This bearing is bonded within a large bull gear and rotated with a servo-hydraulic actuator servo motor via an REL-230 motor controller from the Harmonic Drive. Operation tests prior to shipment included a demonstration that it rotates to the commanded angle within ±0015, which substantially exceeded the requirement. Thermal testing included some operation outdoors during winter in Chicago.

Once we completed the measurement and alignment of the corrector barrel and cells at Fermilab, we shipped all of the components to UCL. At UCL, we installed the lenses in the lens cells with custom room-temperature-vulcanizing (RTV) pads for both axial and radial support. The axial RTV pads have a height accuracy of ±25 μm, which we measured with a Micro-Epsilon laser sensor; this is important to ensure the even support of the lens. Before we mated the cell and lens, we mounted them on a rotary table on an independent X-Y-Z support system. We then brought the lens into contact with the axial pads on the cell while ensuring the correct centering of the lens using dial gauges on the cell and lens. Once the lens was supported by the axial pads, we installed the radial RTV pads that are mounted on the cell inserts in opposite pairs with a layer of RTV to glue each pad to the lens. Once the lens was in place, we measured the distance from the surface of the lens to the cell mounting flanges with a Faro gauge arm. We used this value to produce spacers to position the lens at the correct axial position in the barrel. All of the lenses were installed within their cells by 2018 April.

The next step was the integration of the lenses in their cells with the barrel sections. We mounted the C1, ADC1, and C3 lenses by lowering their corresponding barrel section close to the cell. We then inserted the fiducial pins and attachment bolts and used them to carefully raise the cell to the barrel section. We checked the alignment of the first lens in each section with a pencil beam laser alignment system installed on the rotary table. This system directs a narrow laser beam along the axis of rotation of the table, and a camera records the beam location as it is deviated by the rotating lenses in the beam path. We measured the position of the return and transmitted beams to determine the tilt and decenter of the lens. We then lowered the second lens in its cell into the section without disturbing the laser system, and we repeated the laser beam test to determine the position of the second lens. Once both lenses were installed in each of the three barrel sections, we integrated the three corrector sections and mounted the full barrel on the rotary table. Lastly, we measured the position of the three barrel sections and compared this measurement to that made at Fermilab. These tests demonstrated that the static alignment tolerances were met.

Once these measurements were complete, we disassembled the corrector into three sections for shipment to the Mayall. For safety reasons, each section was mounted inside a custom frame, and that frame had vibration isolators to prevent shocks greater than 2 g in any direction. The crates were designed for shock loads up to ±6 g in both the horizontal and vertical directions. We shipped the corrector sections via charter aircraft directly from London to Tucson, AZ. We also shipped the rotary table and pencil beam alignment system with the corrector sections to support the realignment and reassembly. The reassembly of the corrector at the Mayall is described in Section 12.2.1.

The key requirements on the focal plane system flow down from three main functions: accurate initial placement of the fiber tips; maintenance of the fiber tips in position for the duration of each exposure; and rapid reconfiguration to set up the next field. The first two functions flow down from the throughput requirement, while the third function flows down from the survey efficiency requirement.

The accurate placement of the fiber tips on astronomical sources sets requirements on the relative, lateral accuracy of the fiber tips and illuminated fiducials on the aspheric focal surface, precise metrology of the illuminated fiducials relative to the pixels of the guide cameras, and the placement of stars with precise astrometry at specific locations on the guide cameras. The quantitative requirements on the fiber tips are lateral accuracy of ≤10 μm rms of their nominal target position on the focal surface relative to the guide sensors and absolute accuracy of ≤35 μm for each fiber. The fiber tips are also required to be within ±100 μm of the focal surface to maintain focus, and for the ensemble to be oriented within 03 mean absolute error perpendicular to the focal surface to limit the fiber injection angle. One of the two key metrology requirements is the measurement of the relative positions of the illuminated fiducials with 10 μm accuracy. The second is the measurement of the relative position and orientation of each CCD and the nearest illuminated fiducial with 5 μm accuracy. In both cases, these requirements are on measurements in a plane that is locally parallel to the focal surface.

The four GFAs configured as wave front sensors are used to determine and then maintain focus and alignment relative to the primary mirror, while the six configured as guide cameras are used for initial acquisition and subsequently to maintain telescope tracking for the duration of each observation. The guide centroid accuracy requirement is a tracking error signal better than 003. The GFAs are required to deliver images at a refresh rate of 0.2 Hz and deliver the images over ethernet for analysis in ≤2.5 s. This requirement was established based on the known performance of the Mayall Telescope. These high-level guiding and wave front sensing requirements in turn set requirements on the total area, detector properties, and readout that are provided in Section 5.6.

The L3.2.1 requirement on survey area and duration flows down to a requirement on the inter-exposure time of 2 minutes, and one of the main contributors to this time is the alignment of the fiber positioners for the new field. Given the performance of the positioner design, our expectation was that we would need two iterations of the focal plane in this time. As this reconfiguration applies power to 5020 fiber positioners and the 123 illuminated fiducials, reconfiguration produces on the order of 5–10 kW of heat. There are consequently requirements on the cooling of the focal plane system to both minimize thermal gradients that could impact image quality and to avoid damage to equipment due to overheating. These requirements are described in Section 5.7.

The fiber positioners have two rotational axes. The rotation of the first (or θ) axis is centered on the positioner. The second (eccentric or ϕ) axis is centered along an arm nominally 3 mm from the θ axis. The fiber tip is located along a second arm nominally 3 mm from the center of rotation of the ϕ axis. The positioner can consequently locate the fiber tip within a nominal 12 mm diameter patrol region. The positioners were designed to be very compact in order to maximize the number on the focal surface, and may be placed as close as within 10.4 mm of neighboring units. The patrol regions overlap to maximize coverage of the focal surface, although it does also mean that we need software to prevent collisions between neighboring positioners. Figure 8 shows the tips of ∼150 positioners, and illustrates their footprint.

Zoom In Zoom Out Reset image size

One of the key technologies that enabled the compact positioner design are high-precision 4 mm diameter brushless DC gearmotors. These were produced for DESI by Namiki Precision Jewel Co. There is one of these per axis per positioner, for a total of 10,040 motors in the FPA. These motors do not have encoders, and therefore we use the FVC (see Section 5.8) and the PlateMaker software (see Section 8.5.3) to determine if the positioners are in the correct location relative to the fiducials. The motors each have three coils and are driven by pulse-width modulation (PWM) of the current applied to its coils. The PWM frequency is 18 kHz (55 μs); thus, for our controller (operating at 72 MHz) we have 4000 counts of resolution for each 55 μs update period. This gives us fine control over the net current delivered to each coil. To drive the motor, we apply sinusoidally varying currents to the three coils, phased 120° apart. The frequency of this waveform sets the shaft angular speed. The firmware provides a user-facing "duty cycle" scalar parameter, by which the user can scale the pulse-width durations (thus scaling the effective amplitude of the waveform, thus the net current to the coils, thus the torque) by a multiplier between 0% and 100%.

The positioners are powered by a 7.5 V DC supply voltage, and a 3.3 V voltage regulator on the positioner electronics board provides local power. The power draw per positioner with both motors in operation at 100% duty cycle is 3.23 W. Only a few dozen motors in the instrument are presently run at 100% duty cycle in order to overcome internal component misalignment or friction and the remainder are run at 70% duty cycle. The power draw is 180 mW with the motors idle and the electronics available for active communications. The positioner electronics are put into a low-power sleep mode most of the time, including during observations. The power draw is 15 mW per unit in this mode.

The shaft position is controlled open-loop by rotating the sum magnetic field of the three coils. For large motions, the magnetic field, and thus the motor shaft, is ramped up rapidly ("cruise speed") to a rotation rate of 9900 rpm. This is very fast in practical terms: 17607 s⁻¹ at the output shaft. For smaller motions, such as repositioning, the final 3° (∼160 μm) of the move is done at low speed ("creep speed," 150 rpm) and very fine resolution (00003 ≈ 0.002 μm at the output shaft). After positioning, the coils are completely de-energized, and the position remains stable due to a tab that applies friction to the motor shaft. We measured the stability by taking repeated centroids with the FVC before and after de-energizing the motor coils.

There is a positioner electronics board mounted to each positioner that hosts a Cortex microcontroller running custom firmware. This board connects the positioner to power (two wires), a controller area network (CAN) communications bus (two wires), and an independent common logic line (SYNC, one wire) used to coordinate positioner moves. The firmware contains an integral boot loader so that after initial programming, subsequent reprogramming can be performed remotely via the CAN bus command lines. This feature allows full access to the microcontroller while minimizing the wire connection count.

Before a move, the move commands are uploaded to the individual positioners through the CAN bus. The start of the moves is synchronized at the petal level through the SYNC line on the positioner electronics board. Each positioner is uniquely addressed by an ID number and can be physically placed anywhere on the CAN bus. Groups of positioners are connected in parallel, and in any order, to two-line power and CAN signal rails. There are also a temperature monitor and motor current monitors on the positioner electronics board that can be queried through CAN commands.

The illuminated fiducials in the DESI focal plane provide an important dual-purpose reference. They tie the pixel positions of the fibers imaged by the FVC to their physical locations in the focal plane, and tie physical locations in the focal plane to the celestial coordinates of stars imaged by the GFAs. As such, they are crucial to precisely position every fiber on its intended target.

The physical shape of each fiducial is an 8 mm diameter tube that is sized and threaded for interchangeability with the positioners. At the tip of each fiducial, there is a small glass block with a ground finish on the inner surface and an opaque blue chrome finish on the outside. Four 10 μm diameter pinholes in a standard pattern are etched into the chrome finish. In operation, the pinholes are illuminated by a monochromatic 470 nm LED inside the support tube. Each fiducial has a microcontroller, the same as for the positioner, for communication on the CAN bus and control of the LED intensity. The tip of one fiducial is visible in Figure 8.

The fiducials were produced in two types. The vast majority were field illuminated fiducials (FIFs) that are distributed throughout the focal surface. In addition, we produced shorter guide illuminated fiducials (GIFs), two of which were mounted to the GFA camera body, in order to obtain the most precise mapping from the reference frame of the fiducials to the GFA detector pixels.

After we installed the fiducials in the petals, we measured the average position of the four pinholes on each fiducial to high precision relative to a petal reference. The resolution and stability of the FVC are sufficient to measure the centroids of the back-lit fibers and the fiducial pinholes to ∼20 millipixel precision. The fiducial centroids provide important information to calibrate the transformation from fiber centroid coordinates to focal plane coordinates. In practice, we refined these measurements during commissioning as part of the dither tests described in Section 13.3.

Each physical petal has 514 precision holes for the positioners and fiducials. Each hole is machined at a unique angle to orient the fiber positioner or fiducial along the chief ray at its location on the focal surface. The petals are mounted into the focal plane ring (FPR). Boston University (BU) fabricated the petals and contracted Dial Machine (Rockforrd, IL) to fabricate the FPR and have it anodized using an inorganic black process. The FPR was delivered to BU where we inspected it and found it to be satisfactory for flatness and for mounting the petals. BU contracted and closely worked with Zeiss in Boston to validate each petal with a custom CMM program. We shimmed the petals and conducted several test alignments with the FPR at LBNL prior to the installation of the positioners to verify the alignment precision, as well as gain experience with integration. The alignment error of each petal and the FPR was ± 15 μm rms. See Duan et al. (2018) for more details about the fabrication, metrology, and alignment of the petals and FPR.

The petals also include power supplies, control electronics, and substantial cable routing fixtures. The control of the positioners and fiducials is via a small, single-board computer (BeagleBone Black Industrial), which was integrated on a petal control board. This control board is housed in a small box (the "petalbox"), along with circuitry to send and receive external signals, three 12 VDC power supplies, and two commercial eight-channel CAN control boards from Systec for communication with the positioners and fiducials. The petalbox also carries signals for the GFA power supplies, thermoelectric cooler, and fan. The petal control computer in the petalbox communicates with the ICS via ethernet (see Section 8).

There are two 600 W power supplies (from Mean Well) on each petal. These connect to the positioners via a large printed circuit board that runs radially along the back of the petal. This circuit board also carries the CAN and SYNC bus traces. There are then seven transverse circuit boards that are perpendicular to the large printed circuit board. The seven transverse boards carry a total of 10 CAN communication buses for each petal. A photograph of a complete petal is shown in the left panel of Figure 9. We integrated and tested all of the petals at LBNL. In the summer of 2021, we made several upgrades to the petal electronics that are described in Section 13.6.

Zoom In Zoom Out Reset image size

The positioners are moved via open-loop control of the two motor axes. The starting point to calculate a move is the last position measured with the FVC. The move consists of a series of rotations and pauses of the motors that define the path of the positioner to the intended location. Once the move is complete, we measure the position again with the FVC. In order to measure the locations of the fiber tips with the FVC, we back-illuminate the fibers with an illumination system that is mounted to the spectrograph shutters (see Section 7.3) and the locations of the fibers measured with the FVC are converted to the focal plane coordinate system with the PlateMaker software described in Section 8.5.3.

During typical nighttime operations, we use two move iterations to acquire targets for a new field. We call the first move the "blind" move because it is based on the known (expected) position, which is typically based on the FVC image obtained before the prior science exposure. The blind move puts fibers within ∼50 μm rms of their target positions, and therefore does not meet our requirements. The second, correction move is based on an FVC image obtained after the first move, and this meets our requirements with an rms accuracy of better than 6 μm with respect to the commanded position.

The move commands for each positioner have to take into account their neighboring positioners because the patrol regions overlap. This impacts the allowed final positions and the move path (including timing). Our software evaluates and avoids collisions during moves with a model of each positioner as a pair of 2D polygons. The fiberassign software (see Section 9.2.3) that matches fiber positioners with targets is also aware of this geometry and avoids targets that would produce collisions. To avoid collisions during the moves to these positions, the typical motion is to first retract the eccentric or ϕ arm, then rotate about the central axis θ arm, and lastly extend the eccentric arm to the final position. This avoids most potential collisions. We spread out these moves in time so that not all positioners are moving at once, which reduces the instantaneous power draw and further decreases potential collisions. We lastly eliminate any remaining, potential collisions with small adjustments to the move paths. In rare cases, the algorithm finds no solution for a given positioner to reach its assigned target, and that fiber is not moved to a target for that observation.

There are 10 GFA cameras at the periphery of the focal plane. Six of these are configured as guider cameras and four as wave front sensors. Each unit contains a back-illuminated e2v Model CCD 230-42 detector with 2048 × 2064 pixels that are 15 μm square. We use the central 1024 × 2048 pixels for imaging and the two remaining 512 × 2048 regions as a frame store. The centers of the detectors are approximately 398.5 mm from the center of the focal surface. At this distance, the meridional and sagittal plate scales are approximately 76.3 μm/''and 70.4 μm/'', respectively. The imaging region is oriented with the short axis meridional and the long axis sagittal and subtends $3\buildrel{\,\prime}\over{.} 34\times 7\buildrel{\,\prime}\over{.} 27$ on sky. The average scale is about 0205 pixel⁻¹. The GFA cameras were built by a consortium of four Spanish institutions: Institut de Física d'Altes Energies (IFAE), Institute of Space Sciences (ICE-CSIC, IEEC), Centro de Investigaciones Energéticas, Medioambientales y Tecnológicas (CIEMAT), and Instituto de Física Teórica (IFT-UAM/CSIC). Their work included custom packaging to minimize the footprint of the GFAs on the focal surface. As part of this design, the voltage supplies are in a separate electronics box that is located behind the petal, and they are cooled by air flow driven in part by fans mounted behind each camera.

All 10 GFA cameras have an r-band filter directly in front of the detector. The cameras do not have shutters. The filter was chosen to maximize sensitivity to stars. These filters do not transmit the bluer light used by the back-illumination system and the illuminated fiducials. The guide sensors have a 5 mm thick filter and are confocal with the fiber tips. Each wave front sensor has half of the active area covered by a 1.625 mm thick filter, and the other half covered by an 8.375 mm thick filter. Half of each wave front sensor is consequently designed to record extra-focal images, and the other half to record intra-focal images.

The wave front correction requirements are to measure the defocus of the corrector with respect to the primary mirror with 30 μm rms, decenter with 300 μm rms, and tip/tilt with 10'' rms, and report corrections to the hexapod within 5 s. The software that calculates these corrections is the AOS and is based on the similar system developed by Roodman et al. (2014) for DECam. One important difference between AOS for DECam and DESI is that the DESI GFAs do not have shutters, and it is consequently possible to calculate corrections prior to the start of science exposures. The guiding and especially the wave front correction requirements were that the read noise must not exceed 25 e⁻ pixel⁻¹. Due to the tracking requirement, the detectors also have requirements on frame rate and data transfer. The central area of the CCD is read into memory in 2.5 s through four amplifiers. In regular operation, we obtain a 5 s guiding image approximately every 8 s. While this is somewhat below the refresh rate requirement, in practice this frame rate has worked very well.

The FPE maintains the FPR at a constant temperature of 11°C, removes heat generated by the GFAs, positioners, and their associated electronics, monitors the humidity, and maintains the exterior within ±1° C of the dome ambient temperature. We sized the FPE insulation to prevent significant surface temperature differentials with respect to the ambient environment, so as not to induce dome turbulence by natural convection. The air supply to the FPE is provided by an external, facility dry air system and cooling is provided by an external chiller that removes heat with 3M Novec 7100 fluid. There are two distinct volumes of air within the FPE, one between the C4 lens and the focal surface, and the other behind the focal surface that contains the remainder of the FPE, including electrical and other utilities. The FPE has two large fans. One drives air via four ducts into the focal plane adapter volume, where it then passes through the positioners, the GFA cameras, and through narrow gaps between petals. The other fan drives air through a heat exchanger. The heat exchanger transfers heat into the Novec coolant, which then transports it out of the FPE control volume. These fans are counter-rotating to minimize vibrations and drive more air through the positioner array and heat exchanger. The FPE Controller (FXC) continually monitors the temperatures and humidity within the FPA and can adjust the temperature of the refrigerant and the speed of the fans.

There are hardware interlocks within the FPE to protect equipment. One is that the FXC has hardware interlocks that can shut down power to the entire focal plane system if the temperature and/or humidity exceed threshold values. The FXC is powered by an independent 24 VDC line so that it maintains power in the event the interlock is tripped. The FXC will also issue warnings and alarms in case the measured values approach predetermined thresholds. The electronics box for each GFA camera monitors the humidity and temperature of the electronics for that camera. If either exceeds safe limits, the bias voltages for the CCD are turned off. If the electronics exceed the temperature limit, the GFA power is also shut off. There is also an environmental power interlock for the focal plane on the M floor of the Mayall. If smoke is detected or the dewpoint exceeds −2° C, power is shut off to the fans, petalboxes, GFAs, positioners, and fiducials. This interlock system also stops the Novec chiller that provides cooling to the focal plane system. If the temperature exceeds +35° C, then the power is shut off to the positioner power supplies and petalboxes. In addition to these hardware interlocks, there is a fault monitoring system that provides warnings if parameters exceed the nominal range (see Section 8.2.1).

The FVC is designed to measure the locations of the back-illuminated fiber tips relative to the illuminated fiducials. The camera is mounted approximately 1 m behind the primary mirror and about 12.25 m from the focal surface. The detector is a Finger Lakes Instrumentation Microline camera body with Kodak KAF-50100 CCD. This CCD has a 6132 × 8176 pixel format with 6 μm pixels, although we just read out the central 6132 × 6132 pixels. The key requirements on the FVC system are to return centroid measurements with <3 μm accuracy with a <5 s cycle time. The FVC meets these requirements, and Baltay et al. (2019) describe the FVC system in detail.

The FVC system observes the location of the fiber tips and the fiducials through its own lens and the DESI corrector system, and therefore we require accurate knowledge of the distortions of both optical systems to compute the transformation from coordinates on the FVC to the focal plane coordinate system. Through the end of 2019, the FVC had a 600 mm f/4 Canon telephoto lens (described in Baltay et al. 2019); although like most telephoto lenses, it had significant, nonaxisymmetric distortions. We ultimately decided to switch to a simpler, achromatic lens in 2020 to minimize distortion. The new lens is a 25.4 mm diameter singlet lens with a 600 mm focal length. There is also a 50 nm wide filter centered at 475 nm in front of the lens. The singlet is a plano-convex N-BK7 lens that is part KPX119AR.16 from Newport, and the filter is part 86-950 from Edmunds Optics. We used an optical model of the FVC and the corrector to predict the distortion pattern that should be observed by the FVC and then made small adjustments to the distortion pattern based on the locations of fiducials measured by the FVC.

We thoroughly tested each petal prior to shipment. The test suite for each petal included electrical tests, mechanical and optical metrology, characterizations of the GFA noise properties, measurements of the fiber throughput, and fiber positioner performance. The metrology included measurement of the locations of the two GIF fiducials relative to the GFAs, measurement of alignment features on the petal relative to the FPR, and measurements of the relative locations of all of the FIF and GIF fiducials and alignment features. The electrical tests confirmed the safe operation of the petal electrical systems. We characterized the performance of each GFA camera after integration with its petal and measured an average read noise of 18 e⁻ and an average gain of 3.7 e⁻ per ADU. This meets the read-noise requirement of 25 e⁻ per pixel. The fiber throughput measurements are described at the end of the next section.

Measurement of fiber positioning speed, accuracy, and precision was a major activity prior to shipment, both to confirm requirements and to further develop the software and procedures for operations. The key measurement was the lateral accuracy of the fiber tips, which we conducted in a clean room at LBNL. The test setup included a system to back-illuminate the fiber tips that was based on the shutter illumination system, a spare FVC to measure the locations, and substantial specialized software. We measured the lateral accuracy ¹⁰¹ with an "XY Test" in which we moved each positioner to a series of points on an evenly spaced 5 × 5 grid of 24 targets that was approximately 3 mm on a side. There was no central target because we cannot position at the exact center of the patrol field due to small differences in the arm lengths. For each position, we first execute a blind move of all positioners and measure the errors in their locations with the FVC. We then used the error vectors from that FVC measurement to calculate and send correction moves. Finally, we measured the errors in the correction moves with a second FVC image. The distribution of the uncertainties in the blind and correction moves are shown in Figure 10. The blind move typically achieved a median accuracy of 29 μm in the lab. The median accuracy was 2.2 μm after the first correction move, which met our requirement of ≤10 μm rms lateral accuracy. Note that this measurement did not include several important effects that impact performance on sky, including astrometric transformations, optical distortions, dome turbulence, and uncertainties in the metrology for the complete focal plane system after assembly at the Mayall. We obtained on-sky measurements that included these effects during commissioning and describe those results in Section 13.

Zoom In Zoom Out Reset image size

The key requirements on the fiber system relate to the throughput. The throughput of this system is impacted in three ways: (1) Fresnel losses; (2) attenuation of light as it propagates through the fiber; and (3) losses due to the degradation of the output beam. The throughput requirement from Fresnel losses is >99% over nearly the entire wavelength range, except for percent-level decreases below 375 nm and above 900 nm. This requirement is imposed as both an average over 5000 fibers, and that no fiber may be below 85% of the nominal value at any wavelength. The bulk attenuation requirement is determined by what is achieved with Polymicro "FBP" broad spectrum optical fiber. This ranges from a minimum of 52.6% at 360 nm to a maximum of 97.2% at 850 nm for our typical cable length. The degradation of the output beam is an increase in angle of the output beam compared to the input beam, which is commonly referred to as FRD. This characterizes how much energy is within some output cone that is accepted by the spectrograph, and impacts the fraction of the total light within the PSF. Our requirement is that the encircled energy should be ≥90% within an f/3.57 output beam when illuminated by a uniform f/3.9 input beam that mimics the corrector, including a scaled obscuration at prime focus. There are also requirements on the throughput stability, the PSF stability, and the fraction of intact fibers at the time of integration with the telescope.

The DESI fiber system has 10, identical fiber cable bundles, one between each petal and spectrograph. Each cable bundle includes about 530 fibers, more than the 502 positioners per petal so that some spares are available in case of breakage before integration with the petal and slithead. Our Polymicro FBP fiber has a core diameter of 107 ± 3 μm. Inside of the fiber cable bundle there are 11 protection tubes from Miniflex, 10 with 53 fibers each and one with three sky fibers. We used a boron nitride lubricant and a fiber control guide to avoid tangles that might induce stress when inserting the fibers into these tubes. The 11 tubes were then helically wound around a tensile element that limits stresses due to axial loads and temperature changes, and then inserted into a protective conduit from Adaptaflex. The middle panel of Figure 9 shows one of these fiber cable bundles. See Schmoll et al. (2018) for more details about the fiber cable production.

One end of each fiber cable bundle terminates at the slithead (Tyas et al. 2018), which consists of 500 fibers in a linear array that is grouped into 20 blocks of 25 fibers each. Each block has 25 parallel V-grooves machined in fused silica that hold the fiber ends, and consequently each block points at a slightly different angle along the linear slit. To assemble each block, we bonded the individual fibers between the V-grooves and a cover plate, and co-polished the fiber ends and the block. We then bonded a slit cover window with a UV-transparent glue (EPO-TEK 301-2) onto each fiber block. The slit cover windows were AR-coated by Infinite Optics. A side view of one slithead is shown in the left panel of Figure 11, and a view along the optical axis of one fiber block is shown in the right panel. The other end of each fiber cable bundle initially terminated in a fiber spool with sufficient excess fiber for splicing to the PFAs. This process is described in the next subsection.

Zoom In Zoom Out Reset image size

The FRD of the fiber cables was characterized by collimated (or ring) tests. In this test, a collimated light beam at 625 nm is injected into a fiber, which azimuthally scrambles the light to form a ring illumination pattern. The diameter of the ring is a direct measurement of the incidence angle of the input light, and the thickness of the ring in the radial direction is a measure of the FRD. While not as quantitative as a measurement of the angular diffusion of a filled light cone matched to the nominal f-ratio, the collimated FRD test is very simple to perform and interpret. We performed both cone and collimated FRD tests (see Section 6.4) on a subset of fibers and used the results to calibrate the collimated FRD tests that were applied to a larger sample. Upon the completion and verification of each fiber cable bundle and slithead, we shipped the unit from Durham to LBNL for integration with the PFAs and the focal plane system.

A PFA is a single, ∼3 m strand of the same fiber employed in the fiber cable bundles and slithead. These 3 m strands exist because the positioners were designed to be loaded into the petal from the front, and we did not want to pull the entire ∼50 m fiber associated with each positioner through the petal before wrapping the fiber cable. The assembly of the PFAs was carefully optimized to minimize degradation of the optical performance. One end of each fiber is precision-cleaved with a diamond cleave blade while under tension in order to produce a high-quality optical surface. We optimized the tension for the cleave with feedback provided from FRD measurements. We chose to optimize the precision cleave to produce a high-quality surface that did not require additional polishing. This eased requirements on the hardness of the adhesive we used to bond the fiber into the ferrule, and we chose a low-stress option that cured very quickly under UV illumination. There is also a polyimide tube bonded to the back of the ferrule to provide strain relief and reduce stress when the positioner moves. The assembly of each PFA took only three minutes per fiber, in large part due to the short cure time, and this helped us assemble and test these units fairly quickly.

Once the PFA was complete, we had the input face of each PFA AR-coated by Infinite Optics. We loaded sets of 320 PFAs into custom coating jigs that could be placed directly into the coating chamber, and therefore minimize handling by the vendor. After coating, we inserted each PFA into a positioner and used optical metrology to check the fiber tip location with respect to the focus direction against the ±100 μm requirement on all fiber tips relative to an ideal focal surface. Finally, the positioner with the PFA was installed into the petal from the side of the petal that faces the focal surface.

After the 500 positioners were installed in a petal, the PFAs were connected to the Spool Box at the end of the fiber cable bundle, which contained a sufficient excess of fiber for the splicing process (see Figure 12). The fibers were connected with fusion splicing, which proved to provide superior performance to other splicing and mechanical connector technologies (Fagrelius et al. 2016). To employ this method, we carefully aligned the two uncoated ends of fibers, melted them with an electric filament, and pushed them together to create a solid bond. The result of one splice is shown in the bottom panel of Figure 12. This technique is used in the telecommunications and medical industries, and several commercial machines are available for this procedure. We experimented with various fusion splicing parameters and optimized our choice of parameters based on feedback from the collimated FRD performance.

Zoom In Zoom Out Reset image size

We performed FRD and absolute flux throughput measurements of the complete fiber system at 625 nm for each petal after it was assembled. The FRD measurements used the collimated (or ring) illumination test described in Section 6.2. The average FWHM of the output ring was 192, and the distribution is shown in Figure 13. We empirically determined the relationship between the measured FWHM and the inferred FRD throughput with measurements of multiple fibers with both full cone illumination and the collimated FRD test (Poppett et al. 2018). The FRD throughput contribution corresponds to the encircled energy within an f/3.57 output beam. The average inferred FRD contribution to the throughput is 90.3%, which met the requirement of 90%. The other contribution to the fiber throughput is the flux transmission, which includes both surface losses and attenuation in the fiber. We measured the flux transmission of each fiber with measurements relative to a well-calibrated test fiber. The fiber throughput is the combination of the FRD contribution and the flux transmission. The average fiber throughput is 89.1%. Figure 13 also shows the distribution of the FRD throughput contribution and total fiber throughput for all 10 petals.

Zoom In Zoom Out Reset image size

A series of Level 4 technical requirements was imposed upon the spectrograph design, and these requirements were used to also generate Level 5 specifications for the cryostats, detectors, Rack, and Shack. One class of Level 4 requirements stipulated conformance with the design adopted for other subsystems, namely the fiber diameter of 107 μm, the input light cone of f/3.57, the spectral bandpass from 360–980 nm, and accommodation of 5000 total fibers. The other class of requirements specified the resolution, throughput, PSF stability, and noise. Those key Level 4 requirements and Level 5 specifications are summarized in the relevant subsections.

The spectrograph design has a reflective, f/3.57 spherical Schmidt collimator and three refractive, f/1.7 cameras. This produces a magnification of 0.48 of the 107 μm fiber diameter onto the 15 μm pixels in the three wavelength channels. Light enters the spectrograph through a 121 mm pseudo-slit composed of 500 fibers from one of the petals in the FPA. This pseudo-slit protrudes through a slot in the NIR dichroic such that it is immediately behind the exposure shutter. Collimated light is reflected directly back toward the slit. Light for the NIR channel passes through the NIR pass dichroic, the NIR VPH grating, and into the NIR camera. The slithead vignettes approximately 1% of the light that enters the NIR channel. Light in the red and blue channel wavelength range is reflected off the NIR pass dichroic, and the blue light reflects a second time off of the red pass dichroic, and then is dispersed by the blue VPH grating and enters the blue camera. Light in the red channel wavelength range passes through the red dichroic, is dispersed by the red VPH grating, and enters the red camera.

The collimator mirror was fabricated by Winlight Optical Systems and coated by Infinite Optics with an enhanced silver coating. While the spectrographs are maintained at a constant temperature, the collimator mount design includes athermalization to further maintain stability in the event of small temperature fluctuations. The mount also includes adjustment features for tip/tilt and focus, and the Hartmann door mechanisms described in Section 7.3.

The NIR pass dichroic transmission requirements were >95% from 772–980 nm, and reflectance >95% from 360–757 nm. The average incidence angle is 254. The red pass dichroic requirements were >95% transmission from 593–772 nm, and >95% reflectance from 360–566 nm, and the average incidence angle is 321. The substrates for both types of dichroic are fused silica, and all of the dichroics were produced by Materion. The mount design includes several degrees of tip/tilt adjustment to aid the optical alignment.

There is a distinct grating design for each wavelength channel. All three gratings have physical dimensions of 180 × 160 mm², are 16 mm thick, and the incident beam diameters are approximately 140 mm. The VPH grating fringe frequencies and average incidence angles are 1103.5 l/mm and 1148 (blue), 1157.4 l/mm and 1873 (red), and 992.5 and 2119 (NIR). The fringe angles for all three were 3° to avoid Littrow ghosts, and in all cases the diffraction angle is <16°. There is also an AR coating on the exterior faces of the gratings. All of the gratings were produced by Kaiser Optical Systems, Inc. and mounted to the cameras by Winlight. The resolutions of the blue, red, and NIR channels are approximately 2000–3000, 3500–4500, and 4000–5500, respectively. These values meet the resolution requirements of FWHM (λ/Δλ) ≥ 1500 from 360–555 nm, ≥3000 from 555–656 nm, and ≥4000 from 656–980 nm.

There are different optical designs for the blue, red, and NIR cameras. The optical designs for the cameras each use a triplet, a singlet, and a field lens that also serves as the window of the cryostat. These designs produce excellent image quality, and achieve the throughput requirement. The average rms radius is better than 6 μm, and the maximum is approximately 10 μm. The blue camera has two large (∼160 mm diameter), convex lenses fabricated from calcium fluoride crystals that sandwich a convex N-BAK2 lens. Due to the mismatch in CTE between these materials, and the fragility of CAF2, Winlight used a fluid-coupled solution for this triplet that was based on the solution designed by O'Brien et al. (2014) for the C/KOSMOS instruments (Martini et al. 2014). The triplets in the red and NIR cameras were fabricated from optical glasses with similar CTE (S-BSM4, S-TIM22, and S-LAL12) and cemented. All three field lenses were constructed from fused silica. All of the lenses were polished by Winlight and coated by Oerlikon Balzers. Winlight built and aligned the cameras.

There are four mechanisms in each spectrograph: two shutters and two Hartmann doors. The exposure shutter is the first element in the light path after the slithead, and the operation of this shutter sets the length of exposures. The exposure shutter also contains the back-illumination system for the focal plane system. The second shutter is located between the slithead and the NIR camera. Both shutters have pneumatic seals that produce an extremely light-tight seal around the shutter blade when inflated. When the back-illumination system is in use, both of the shutters are closed and the seals are inflated; otherwise just the shutter doors are closed. The control system includes interlocks to prevent the activation of the back-illumination system when the shutter is not closed and the seal is not inflated. The two Hartmann doors are integrated with the collimator mirror mount. The locations of these mechanisms are shown in Figure 14.

The main specifications on the exposure shutter were that it move from completely open to completely closed in less than 0.5 s, and that the actuation time be uniform to 0.1 s. These specifications minimize uncertainties in the true length of each exposure. The illumination system included specifications for total flux within the 400–550 nm bandpass to ensure adequate S/N and minimize chromatic aberrations in the centroids measured by the FVC, and to avoid contamination through the r-band filters mounted in front of the GFA detectors. Lastly, the shutters were required to transmit <10⁻⁷ of the flux at the fiber tip into the red and blue channels so the back-illumination system could be used when the detectors are still reading out the previous exposure. Derwent et al. (2016) present a detailed description of the exposure shutter design. The NIR shutter is identical to the exposure shutter, except it does not have a back-illumination system, and it has a somewhat smaller aperture on the shutter housing due to the smaller beam size at its location. The two Hartmann doors are used to measure the focus of the three optical channels through a classic Hartmann test, and consequently, each door is actuated by a separate motor.

All control electronics for the spectrograph mechanisms are housed in an electronics box called the spectrograph controller, and there is one per spectrograph. The control box includes a digital stepper motor controller for each of the four mechanisms, an Next Unit of Computing (NUC) PC that handles communications with the ICS, an air cylinder for the shutter seals, solenoid valves that control the air to the seals, and other control electronics to read various sensors. Each motor controller has a distinct command set to perform specific actions, such as to open or close a shutter, as well as to provide low-level equipment safety. More details of the control system were presented in Coker et al. (2016).

All three spectrograph channels have back-illuminated CCDs with 15 μm pixels, four readout channels, and a readout rate of 100 kHz. The blue devices are 4096 × 4096 STA4150 CCDs from Semiconductor Technology Associates that were processed and packaged by the University of Arizona Imaging Technology Laboratory (ITL) to meet high efficiency and low noise requirements. The quantum efficiency requirements included better than 75% from 360–400 nm and ≥85% from 450–600 nm. The read-noise requirement was better than 3 e⁻ rms, and the dark current requirement was less than 10 e⁻ pix⁻¹ hr⁻¹. There were also numerous other requirements on quantities such as charge transfer efficiency and cosmetics. All of the devices were tested at ITL prior to shipment and then retested with the DESI electronics. ITL delivered 13 science grade devices, and we mounted the best 10 plus one spare into cryostats.

The devices for the red and NIR channels were especially developed for DESI by LBL with wafers fabricated by Dalsa. These are thick, high resistivity, fully depleted p-channel 4114 × 4128 CCDs that provide very high quantum efficiency in the NIR. The quantum efficiency requirements included better than 85% from 600–900 nm and better than 60% from 900–980 nm. The read-noise requirement was better than 3 e⁻ rms, and the dark current requirement was less than 10e⁻ pix⁻¹ hr⁻¹. These devices achieved higher quantum efficiency in the NIR and lower noise than the previous generation, such as those for the Dark Energy Survey (DES; Flaugher et al. 2015), through two main developments. First, improvements to the AR coating through the addition of higher-index, ZrO₂ to the multilayer coating increased the quantum efficiency. Second, the development of a buried-contact technology combined with optimizations to the transistor size substantially reduced the read noise (Holland et al. 2014; Bebek et al. 2015). We mounted the best 20 devices plus two spares into cryostats.

The cryostat system has one self-contained vacuum vessel for each spectrograph channel. The requirements on the cryostats include precise ±1 K and stable ±0.1 K temperature control, and the ability to accommodate the different operational temperatures of the devices (163 K for the blue channel, and 140 K for the red and NIR channels). The cryostats maintain temperature control with Thales linear pulse tube cryocoolers. An ion pump maintains the pressure below 3 × 10⁻⁷ mBar for all cryostats (well below the <10⁻³ mBar requirement). The cryostats are monitored with a slow control system that includes setting and regulation of the CCD temperature, setting the pulse tube frequency and electrical power, and control of the warm-up and cool-down process. The field lenses for the spectrograph cameras form the windows of the cryostats. The detectors and field lenses were integrated with the cryostats at CEA Saclay, tested, and then delivered to Winlight for integration with the spectrographs. CEA/Saclay also reintegrated the cryostats with the spectrographs and tested them after they arrived at Kitt Peak.

The 10 spectrographs are supported in two rows of five on a structure called the Rack, which is shown in Figure 15. The main design requirements on the Rack were to place the spectrographs as close as practical to the entry point of the fiber cables into the Large Coudé Room in order to minimize the total fiber length and to minimize vibration frequencies. The Rack is approximately 8.5 m long and 1 m deep and constructed with an extruded aluminum frame with steel braces that provide both structural rigidity and the mounting surface for the spectrographs. The Upper Rack is a fixed, interconnected structure that supports five spectrographs. The upper spectrographs were installed directly into position on the Upper Rack via an overhead crane. The Lower Rack comprises five, low-profile wheeled carts. The lower spectrographs were installed onto these carts with the overhead crane and then wheeled into position under the Upper Rack. The Upper Rack includes cable trays along its length on both sides, which support all utility lines such as communications, power, and air lines. The Upper Rack also has shelves to support the Slitheads when they are removed for special calibrations. As several spectrographs had not yet been installed after installation of the focal plane system, those Slitheads were placed on the shelves with an illuminator similar to the one on the exposure shutter. This stand-alone fiber illumination system was used to test the focal plane system before all of the spectrographs arrived.

The 10 spectrographs are installed in a clean, thermal enclosure called the Shack. The Shack is a commercial lab enclosure that we purchased as a kit from Western Environmental Corporation (WEC) to our specifications and then assembled in the Large Coudé Room. The key specifications for the Shack included thermal stability of 20° ± 2°, relative humidity of <50%, and the ability to provide a clean room environment better than class 10,000 when the optics are exposed for >12 hr service operations. The thermal specification optimizes the spectrograph optical design performance, and the humidity specification is to prevent condensation on the Dewar windows. The exterior dimensions of the main room of the Shack are 10.7 × 4.7 m, and the exterior height is nearly 4 m. The interior height is 3.5 m, which includes space for the operation of an overhead gantry crane. There is also an Annex that is 4.4 × 2.5 m with a lower ceiling height (the crane does not extend into the Annex) that was intended to support major maintenance activities. The dimensions of the Large Coudé Room were the most significant limitations on the size of the Shack, and this area was carefully surveyed before the start of the detailed design work.

There are two industrial control electronics boxes for the Shack. One was supplied by WEC, which controls the thermal environment, as well as reads telemetry data from all of the components supplied by WEC. These include temperature, humidity, and pressure, as well as information about the HVAC system. The second control box is called the Shack controller, and it provides remote power control to the 10 spectrographcontrollers, monitors the air pressure of the shutter seals, operates the stand-alone fiber illumination system, and reads the telemetry data from the WEC control box. The ICS regularly queries and stores these telemetry data in the project's database.

The calibration system for the spectrographs includes a new dome screen and new calibration lamps. The key specifications included continuum and spectral line coverage across the DESI bandpass, azimuthally averaged pupil uniformity to 20%, field uniformity over the 32 diameter FOV to 5%, and a specification on the flatness of the continuum emission. The pupil uniformity requirement is designed to limit PSF inhomogeneities, while the field uniformity is designed to minimize sky subtraction residuals. The spectral line coverage is motivated by the wavelength calibration requirements needed to achieve the Level 3 requirements on systematic redshift uncertainties, as well as the relative wavelength calibration between fibers to minimize sky subtraction residuals.

The new dome screen is a replacement of the previous screen, which was too small for DESI. It has a usable diameter of 5173 mm, which corresponds to an FOV of 34, and this allows for a lateral placement error of ±230 mm. The screen is coated with Permaflect (sold by Labsphere), and has reflectivity >90% over the entire DESI bandpass that is independent of incidence angle. The new dome screen was completed in 2017. The calibration lamps are mounted in four boxes that are attached to the top ring. One calibration lamp box is shown in Figure 16. Each calibration lamp box has slots for six drawer units that contain lamps. The first drawer of each box contains a set of LEDs that together produce a reasonably flat, broad-based SED, and the second drawer contains two halogen lamps, one with and one without a blue filter. The remaining drawers have Cd, Xe (2/4 boxes) or Ne (2/4 boxes), Kr, and HgAr wavelength calibration lamps. There is a Raritan Power Distribution Unit (PDU) for each lamp box, and the lamps are activated through the PDU. The PDUs also have temperature and humidity sensors to protect each lamp box. Figure 17 shows the spectral energy distributions of the LED and arc lamps.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

DESI uses the Dynamic Exposure Time Calculator (ETC) to actively estimate the spectroscopic S/N while exposures are in progress, and adjusts the exposure time as needed to ensure the S/N requirements are met. The inputs to the Dynamic ETC include real-time measurements of the image quality, transparency, and the sky continuum brightness. The first two quantities are measured with the guider CCDs, and the sky continuum brightness is measured with the sky monitor. This sky continuum measurement agrees within ∼3% with the sky level measured in the spectrographs in the same bandpass (D. Kirkby et al. 2022, in preparation).

The sky monitor was designed to take input from 20 dedicated sky fibers mounted in fiber positioners on the periphery of the focal plane, with two per petal. The sky fibers are part of the fiber bundle from the focal plane system to the Shack, although they terminate at SubMiniature version A (SMA) connectors on the side of each fiber spool box, rather than in the slithead in each spectrograph. Short segments of DESI fiber with SMA connectors link the spool boxes to one of two dedicated camera systems that sit below the spool boxes on the exterior of the Shack inside the Large Coudé Room. Each unit has an SBIG STXL-6303e CCD camera, an eight-position FW8S-STXL filter wheel, a Nikon 50 mm f/1.2 lens that reimages the fiber tips onto the CCD, and a back-illumination system for the fiber tips. Tie et al. (2020) describe the sky monitor in further detail.

Each spectrograph was extensively tested and characterized before shipment to the Mayall. These tests included focus, optical performance, CCD properties, throughput measurements for each channel, and stray light measurements and functional checks of the mechanisms. This lab system included a range of LEDs, arc calibration lamps, and a white light source that illuminated an integrating sphere. An Offner relay then reimaged the output into DESI-like fibers that comprised a sparse test slit with just 21 fibers, rather than 500. We performed the focus tests with the Hartmann doors, and then used arc calibration lamps to quantify the image quality, resolution, and dispersion. We also used the sparse test slit, in conjunction with a calibrated photodiode, to measure the throughput of each spectrograph as a function of wavelength (see Figure 18). Finally, we measured the CCD readout noise and gain from data with a separate, flat field slit, which was illuminated with a tungsten lamp. These tests verified that the spectrographs met their requirements.

Zoom In Zoom Out Reset image size

The calibration system and sky monitor underwent numerous pre-ship tests. The key requirements for the calibration system were the bandpass coverage, pupil and field uniformity, spectral line coverage, and continuum flatness. Most of these quantities were measured with lab tests prior to shipment, with the exception that the pupil uniformity was verified by analysis. The continuum flatness required a maximum variation of no more than a factor of three in any spectrograph channel, whereas we only achieve a factor of five with many LEDs. In practice, this variation is acceptable, as we simply take more exposures and coadd the results to obtain high-S/N flat fields across the bandpasses. For the sky monitor, we verified the readout cadence and heat dissipation prior to shipment. The remaining requirements were verified after installation as part of commissioning the Dynamic ETC (see Section 8.5.6). The verification of the temperature stability, humidity control, and cleanliness of the Shack environment after installation is described in Section 12.2.2.

The ICS requirements flow from the Level 3 requirements on survey efficiency and duration. The overall requirement is that the ICS must perform all control and monitor functions to enable successful operation of the instrument and the completion of the science program. One of the main performance requirements is that the time between successive exposures, called the inter-exposure time, must be less than 2 minutes. ¹⁰² Another is that the system up time must be greater than 97% by the end of the first year of the survey. There are several monitoring requirements that include storage of instrument telemetry, provision for interfaces to the telemetry data, and a warning and alarm system. The warning and alarm system is informational by design, as ultimately, hardware protection was a requirement on each subsystem. The instrument control requirements include a user interface, connection to the hardware subsystems, and the Dynamic ETC. Lastly, the ICS has to provide computing hardware to perform nighttime survey operations, as well as sufficient storage for at least three nights of observations, should there be a loss of off-mountain data links. The off-mountain data transfer, along with data processing, is the responsibility of the DESI data systems group. That work will be described in a separate series of papers.

The design of the DESI online system is modeled on the readout and control system architecture developed for DECam (Diehl & Dark Energy Survey Collaboration 2015), which has been successfully used for both the DES and other DECam observing programs. There are detailed descriptions of the DECam data acquisition system in both Honscheid et al. (2012) and Honscheid et al. (2014). A schematic view of the overall architecture for DESI is shown in Figure 19.

Zoom In Zoom Out Reset image size

The core of the system is the OCS, which manages the complex DESI exposure sequence. The OCS interacts closely with the focal plane system, which includes the fiber positioners, the FVC, and the GFA cameras. Image acquisition originates with the GFA and spectrograph CCD controllers, and then flows through the image processing and analysis stage to a storage device. At that point, the images are transferred and further processed by the data systems group. The ICS connects to hardware components, including the spectrographs, hexapod, cryostats, and ADC, as well as the Mayall Telescope Control System (TCS) and telemetry database.

The readout system is tightly integrated with the DESI instrument monitoring and control system. This system monitors every component of the instrument and includes detailed information about the instrument status, operating conditions, and performance. All such data are archived in the DESI operations database. The monitoring and control system provides all components of the instrument with access to the telemetry and alarm history databases, and can do so even if the rest of the OCS and the readout system are offline.

8.2.1. Infrastructure Software

The DESI mountain-top computer system consists of four server class machines for the database and the software and user home areas (including two hot standby servers for these services) and 17 rack-mounted Dell PowerEdge R540 workstations. Each workstation is configured with two Intel Xeon Gold 6126 processors running at 2.6 GHz and with 96 GB of memory. Dual network interfaces and dual hot-plug redundant power supplies increase the overall system reliability. The workstations and servers are connected by a 10G-base-T network using Cisco network switches. NFS-shared 44 TB disk arrays in a RAID-10 configuration attached to each of the server machines provide ample storage for observer data, ICS software, and observation and other telemetry data in case the network connection off the mountain is down for an extended period. Intel NUC computers are used to control the FVC, the spectrograph mechanisms, and the spectrograph shack environmental system. The fiber positioners in each of the 10 focal plane petals are controlled by BeagleBone Black microcontrollers. Custom FPGA-based controllers with an ARM processor core are used to readout the spectrograph CCDs and the GFAs. Apple iMac Pro computers, each equipped with three high-resolution 5K Retina displays, are used as observer consoles in the control room at the Mayall Telescope.

The DESI observer console acts as the primary observer interface for day-to-day operations. This set of GUIs includes elements for system and exposure control, alarm displays, and telemetry monitors. Figure 20 shows the DESI observer console GUI. Combined, they present the user with the most commonly used information and everything that is needed to operate the DESI instrument. Near-real-time performance is needed to provide quick access to more specific information and to achieve the responsiveness expected from a modern system. The DESI GUI architecture follows the model-view-controller pattern that is based on the realization that all applications are essentially interfaces that manipulate data. This popular architecture for web applications allows us to provide users with different interfaces (views) for different circumstances, such as a text-based console and an interactive, graphical web page, and helps with support for multiple users both on site and remote. On the ICS server side, we have implemented the controller component as a web server so that we can use standard web browsers to render the views. Web-based user interfaces provide many desirable features such as platform independence, remote access, a large number of third-party tools, and also a certain level of security. Thanks to developments that have enabled faster rendering and JavaScript execution, as well as standards such as HTML5 and the Web-Socket API that allow for advanced functionality, this approach now provides equal performance to desktop GUI toolkits. The client side (web browser) code for the DESI user interface is based on HTML/CSS, JavaScript, and SproutCore, an HTML5 application framework. Websockets, defined as part of the HTML5 standard, are used to provide bidirectional information exchange and a responsive user experience.

Zoom In Zoom Out Reset image size

Exposure sequences contain the information necessary to set up the instrument and point the telescope for an observation. The ICS was structured to allow both direct control of exposure sequences and nearly autonomous operation through an application called the NTS (see Section 8.5.1). The direct control capability was designed to support instrument testing, commissioning, and other specialized observations, and the NTS is used to initiate each exposure sequence during survey operations. In either case, the exposure sequences are loaded into a request queue and managed by the OCS. The architecture of the OCS is pipelined such that the setup of multiple components will proceed concurrently as long as there are no conflicts accessing instrument hardware or certain software components.

Processing of the next exposure sequence begins with a notification from the Dynamic ETC that the current exposure is about to end within the next minute. The activities required to set up for the next exposure are shown schematically in Figure 21. The sequence starts by processing the information provided in the exposure request by the NTS. We determine the telescope, ADC, and hexapod settings and then transform the sky coordinates of the targets to physical focal plane coordinates. These coordinates are then sent to the petal software to prepare for the next positioner move, which also runs the anticollision algorithm. When the spectrographshutters close to end the previous exposure, the OCS instructs the telescope to slew to the new field and adjusts the ADC and the hexapod, and then commands the positioners to execute the first or blind move. Once the telescope, dome, ADC, and hexapod are ready, we obtain a 15 s acquisition image with the six guide GFA cameras that is then processed by the astrometry module (GFAPROC) of the PlateMaker application. Each tile design includes preestablished guide stars that must be placed on specific pixels on the guide cameras to align the targets with fibers. The GFAPROC outputs are used to offset the telescope pointing to be exactly on the field center and to initialize the guider with the expected guide star positions in pixel coordinates for each of the GFA cameras.

Zoom In Zoom Out Reset image size

The next steps are performed concurrently: begin guiding, start observations with the four focus and alignment GFAs, and complete the first iteration of the fiber positioning loop by taking and processing an FVC image. The fiber positioning loop uses real-time feedback from an FVC image of the back-illuminated focal plane to measure the locations of the fiber positioners (see Section 8.5.2). The PlateMaker FVCPROC module converts these measured positions in FVC pixels into focal plane coordinates, determines the corrections needed to place each fiber on target, and those correction moves are sent to the fiber positioning system to prepare and execute the second iteration (correction move) of the fiber positioning loop. The correction move begins as soon as the analysis of the FVC image taken after the blind move is complete. Upon completion of the correction move, we automatically obtain another FVC image to record the final positions of the fiber positioners and fiducials. The fiber back illumination and fiducials are then turned off, while in parallel the spectrograph CCDs are prepared for the next exposure. Once the illumination is off, the spectrograph shutters open and the exposure begins. The analysis of the final fiber positions occurs in parallel with the start of the spectrograph exposure. The exposure loop for the sky monitor cameras starts when the spectrograph shutters open. Optimization of these steps continued into the start of operations, and we present the latest time estimates below in Section 13.5.

The spectrograph exposure time is determined by the Dynamic ETC, which uses information from the sky monitor system and full-frame images from the guide GFAs to analyze the observing conditions in real time and to calculate the accumulated effective exposure time of the spectroscopic exposures. Once the requested effective exposure time has been reached, the Dynamic ETC triggers the OCS to close the spectrograph shutters to end the exposure. In order to control the amount of cosmic-ray contamination on the spectrograph CCDs, the Dynamic ETC can trigger the OCS to initiate an exposure split. This is a spectrograph readout followed by the restart of the exposure at the same location with only minimal adjustments to fiber positions to correct for field rotation and similar effects. At the end of an exposure, the OCS initiates the readout and digitization of the CCDs. While exposing, the AOS analyzes the images obtained with the focus and alignment GFAs. The focus and alignment loop takes 60 s exposures, and each frame is analyzed by the AOS to determine any alignment corrections. Focus corrections are sent to the hexapod and applied while exposures are in progress.

8.5.1. Next Tile Selector

8.5.2. Focal Plane Imaging and Analysis

8.5.3. PlateMaker

8.5.4. Guider

8.5.5. Active Optics System

8.5.6. Dynamic Exposure Time Calculator

The ICS is connected to every component of the instrument as well as the Mayall Telescope. In order to manage the large number of interfaces, we developed ICDs that detail the hardware connection and define deliverables and responsibilities. These documents were reviewed by the project teams and were placed under change control. While the hardware components were under development, we created software simulators for each interface to allow early testing and validation. We adopted a spiral development approach with increasingly complex systems and included a number of functional verification tests throughout the construction project. Some of these tests were coordinated with the project to support other activities, for example, spectrograph testing and evaluation. The four key DOS system tests were: ICS–Mayall TCS integration; single spectrograph support; ProtoDESI, described in Section 11.1; and a full system test using the emulated hardware components.

The ICS has been in routine operation since the start of the DESI precommissioning activities described in Section 11. During DESI commissioning, a steady stream of improvements, bug fixes, and feature enhancements was applied to the ICS. The requirements to provide telemetry storage, data mining, and alarm and error handling infrastructure were demonstrated before the start of commissioning. Since the start of DESI operations, the robustness and ICS availability have surpassed the required uptime of 97%, and the typical inter-exposure times have achieved the 2 minute design goal. For reference, the average exposure time is ∼1100 s for the dark program and ∼800 s for the bright program; so, for example, a 10 s change in the inter-exposure time has a ∼1% impact on the survey margin.

Data systems developed and implemented the target selection, survey plan, fiber assignment, and spectroscopic data processing capable of producing a data set meeting the science requirements in Tables 4 and 5 given the raw data from the DESI instrument. The derived technical requirements for data systems include the assignment of 10 standard stars and 40 sky fibers per petal for calibration, the need to transfer and back up data within 12 hr of sunrise, and to provide fully processed data to the collaboration within 48 hr of observation. The detailed performance of data systems relative to the requirements will be provided in the papers cited above.

9.2.1. Target Selection

9.2.2. Survey Plan

9.2.3. Fiber Assignment

Fiber Assignment selects which targets should be assigned to which fibers for each tile, while also assigning fibers to sky locations and standard stars for calibrations. Lower-density targets such as QSOs are given higher priority and assigned first, followed by higher-density lower-priority targets such as LRGs and ELGs. Standard stars and sky locations are initially assigned to positioners that cannot reach any science targets. Any disabled positioners that are serendipitously pointing at blank sky locations are also used as sky calibration fibers. If needed, low-priority science targets are replaced with standard star and sky assignments if required to achieve the minimum calibration target density (10 standard stars and 40 sky fibers per petal). Sky fibers are also distributed such that groups of 50 fibers on the focal plane will have at least one sky assignment to ensure uniform coverage over both the FOV and across the spectrograph CCDs. Figure 22 shows an illustration of fiber assignment for a very small subset of the focal surface.

Zoom In Zoom Out Reset image size

Since neighboring positioners have overlapping reach, fiber assignment also checks for and excludes target positions in which the mechanical components of neighbors would interfere. Since preplanned fiber assignments cannot perfectly know the exact transforms between sky coordinates and focal plane coordinates (e.g., due to the exact airmass of observation), fiber assignment includes a buffer zone of 200 μm in total reach and an extra 50 μm around each positioner (in addition to the anticollision safety buffer zone used in operations) so that small variations between the expected and actual transforms do not introduce new collisions. Although fiber assignment does not purposefully make assignments that would collide, this is additionally checked by the online code using the final as-observed conditions, and colliding assignment requests are not fulfilled.

9.2.4. Data Transfer, Archive, and Distribution

9.2.5. Spectroscopic Pipeline

9.2.6. Nightwatch

The data systems code was designed from the beginning to run in parallel at a high-performance computing (HPC) center such as NERSC. This ability has been critical for rapid turnaround of nightly data for survey operations and timely processing of internal data assemblies that process all data using a tagged software release. DESI uses message passing interface (MPI) as its primary mode of parallelism due to its performance and ubiquity at HPC centers; some algorithmic steps also use Python multiprocessing. The largest production runs so far have used 40,800 Intel KNL cores simultaneously. Work is in progress to adapt the slowest pieces of the DESI pipeline to GPUs (PSF fitting, spectral extractions, and redshifts) for the next-generation Perlmutter machine at NERSC, which will have dramatic performance improvements. For example, the speed up in the spectral extraction code should be of the order of 20 × per-node throughput relative to the previous generation CPU-only system (Margala et al.2021).

While designing for HPC deployment from the start, data systems also made some pragmatic design choices to maximize the efficiency of human contributors over raw computing performance. Most DESI code is written in Python, leveraging compiled code "under the hood" via numpy, ¹¹³ scipy, ¹¹⁴ and numba. ¹¹⁵ All algorithmic steps can be run on a Linux or Mac laptop using files for input and output, without requiring MPI, centralized databases, GPUs, or other HPC or pipeline infrastructure. This use of Python and separation of algorithms from pipelining and parallelism has enabled non-computing-expert collaborators to make significant contributions, and greatly facilitated the speed with which new algorithm ideas can be implemented and tested.

All data systems code repositories are open source at https://github.com/desihub, with significant contributions from both the DESI project staff and from members of the science collaboration. Developers make branches of the code, implement their features, push the branch to GitHub, and open a pull request for review before integration into the main branch. Each repository includes unit tests that run automatically when pull requests are opened.

Additionally, each night a cronjob updates the main copy of each repository at NERSC and re-performs the unit tests to ensure cross-repository consistency. This cronjob also performs an integration test by simulating a small amount of pixel-level raw data, processes it with the spectroscopic pipeline as if it were real data, and verifies the resulting redshifts. Any integration test failures generate email alerts.

Quarterly software releases of version-tagged repositories include additional testing on real data and end-to-end tests of target selection, fiber assignment, survey operations (via simulation), and the spectroscopic pipeline. These software releases are used for interim internal data releases, processing all available data to date.

During the DESI construction project, extensive simulations guided the code development. These included pixel-level simulations of raw data, higher-level (faster but less detailed) simulations of raw extracted spectra (including sky and not flux calibrated), and rapid simulations of fully sky-subtracted flux-calibrated spectra with added noise based upon the instrument model. Any of these levels of simulation could be inserted into the spectroscopic pipeline for processing as if it was real data.

This extensive simulation and testing program enabled the code to work well from the very beginning of operations. Although we continue to make improvements, the core pipeline met the science requirements and scaling performance needs from the start of commissioning.

The key technical requirement on the telescope upgrades was that they do not degrade the delivered image quality relative to the baseline adopted to plan the DESI survey. That baseline was obtained from the study by Dey & Valdes (2014), who measured the delivered image quality for the Mayall from thousands of images obtained with the MOSAIC prime-focus imager between 2006 and 2011. Those data cover a range of wavelength, airmass, temperature differences between the mirror and ambient environment, and exposure time. Dey & Valdes (2014) reported values of 12–13 for the median of the nightly median values in images obtained in the reddest filters (i, I, z), and that the median of the best values from each night ranges from 093 to 1''.

There were also requirements on improvements to electrical, cooling, and communications infrastructure, as well as facility improvements to support installation. The electrical upgrades included additional capacity in the prime-focus cage for new electronics, in the computer room for additional computing and storage capacity, in the Large Coudé Room for the thermal enclosure and spectrographs, at the top end to support the calibration lamps, and in the facility uninterruptible power supply. Cooling capacity was added to both the Large Coudé Room and the top end for the spectrographs and focal plane system, respectively. The TCS upgrades included the capability to interface with the DESI ICS. Other facility improvements and additions included a new, larger control room on the "utility" or U floor.

The Mayall TCS was modernized as part of the preparations for DESI installation (Abareshi et al. 2016; Sprayberry et al. 2016). Key motivations for this project were to minimize the time between successive exposures and to guide accurately during the ∼20 minutes planned for the longest science exposures. The guiding enhancements included an interface between the DESI guider system and the TCS.

The modernization project drew substantially from the TCS modernization project at the 4 m Blanco Telescope that prepared for the DES (Warner et al. 2012). The main changes were the replacement of the old VxWorks TCS computer, new tape encoders on the hour angle and decl. axes, and new digital servo controllers for the telescope drives. We also replaced the old 2D polynomial pointing model with the commercial TPOINT software. These updates occurred during 2014, and regular operations shifted over to the new TCS at the start of the 2015B semester.

Pointing map measurements with the new TCS showed rms errors of 2''–3'' at that time, which is substantially better than the 10''–20'' errors typical of the previous pointing map and the old, mechanical encoders. The unguided or open-loop tracking errors after the upgrades also exhibited substantial improvement. The best model yielded equivalent open-loop tracking errors of 16 hr⁻¹ in R.A. and 01 hr⁻¹ in decl., far better than the 30'' hr⁻¹ performance typical of the previous polynomial pointing model with the old mechanical encoders and analog servo controllers.

We evaluated the impact of the TCS upgrade on the image quality with 329 nights of data obtained between 2015 December and 2018 February for the Mayall z-band Legacy Survey (Dey et al. 2019), which was the largest program on the Mayall Telescope since the Dey & Valdes (2014) study. The seeing in these images was characterized in a similar manner as the Dey & Valdes (2014) study, namely we first calculated the median seeing and the best seeing for each night, and then calculated the median of the median values and the median of the best values. The median of the median values of each night was 103, and the median of the best values per night was 084. Both metrics indicate that the upgrades prior to installation somewhat improved the delivered image quality.

Some significant improvements for DESI included increases in the capacity of the building's electrical wiring and battery backup, cooling system enhancements, and bandwidth upgrades to the building's ethernet system (Sprayberry et al. 2014). Updates to the facility electrical system and battery backup capability were needed to support the greater power draw from DESI components, and especially to maintain and protect them against interruptions that would impact survey efficiency. The updates to the cooling system were needed to support the stable thermal environment for the spectrographs, the linear pulse tubes for the spectrograph cryostats, and the additional processing and data storage in the computer room. The DESI focal plane system has a separate cooling system described in Section 5.7.

One major change was the surveying and clearing of the Large Coudé Room. This room is located on the "main" or M floor, the bottom floor of the dome enclosure of the telescope. The Large Coudé Room was originally intended for a large spectrograph, and was used for a number of instrument setups, although over the years the room also became a convenient storage area. This room was carefully surveyed in 2014 to support the design of the thermal enclosure or Shack for the 10 spectrographs (see Section 7.5), and then cleared of its venerable equipment collection and the heavy fixtures intended to support the original large Coudé spectrograph in order to prepare for the assembly of the Shack.

The DESI project also needed infrastructure improvements to the computer room, which is located on the Mezzanine level just above the M floor. Mayall staff cleared a number of obsolescent computer racks and tape storage files out of this room and increased the cooling capacity of this climate-controlled room to support the expanded computer processing and storage requirements of the DESI project. Much of this work was completed by 2016 to support the early integration efforts of the ICS with the TCS.

The DESI installation plan included reassembly and testing of the new optical corrector on site prior to integration with the new top ring and subsequent integration with the telescope. The reassembly and testing of the corrector occurred on the ground floor of the telescope building. This area has access to the exterior via a large garage door, high ceilings, and is connected to the telescope level with a pair of cranes (5 ton and 50 ton capacity) that may extend through a hatch in the M floor. Mayall staff cleared the ground floor area to support the delivery of the many large crates that contained the corrector components, as well as numerous other components that moved through this space. They also erected a large clean tent (4 m × 4 m footprint, and high enough to support a crane with a 6 m hook height) to support the reassembly and testing of the corrector.

The operations model for DESI is to have two personnel on site every night: a highly trained expert observer (called the Lead Observer) from the collaboration and an NOIRLab staff Observing Assistant, who is responsible for telescope operations and personnel safety. Observers drawn from the collaboration membership also connect remotely every night and assist with observations (Observing Scientist) and data QA (Data Quality Scientist). Pre-COVID, the operations model had been for the Observing Scientist and Data Quality Scientist to be on site as well.

Many more people were expected to be on site during the installation, functional verification, and commissioning periods. As the original control room on the "console" or C floor could not accommodate more than five people, the Observatory built a new, larger control room on the "utility" or U floor, which is several floors below the M floor. The new control room accommodates up to 15 people, is more easily and safely accessible than the original control room, and reduces the thermal load on the dome environment.

The main purpose of ProtoDESI was to test the fiber positioners, including both the physical mechanisms and the acquisition procedure. It was also useful to test a GFA prototype and improve the guiding algorithm. The instrument included three fiber positioners, 16 fiducials, one GFA camera, a back-illumination system for the fibers, and a backup finder scope camera in the event of difficulties with the fiber alignment. The three fiber positioners each had a 3 m length of fiber that was reimaged on an SBIG STF-8300M camera. ProtoDESI was mounted to the f/3.2 refractive Mosaic corrector at the prime focus. We operated ProtoDESI with an early version of the FVC and the ICS and therefore we were able to test key software components several years in advance of commissioning.

We used ProtoDESI to test the fiber positioner technology in conjunction with an early test of the fiber positioning software PlateMaker, which calculates the coordinate transforms necessary to place fibers on targets (see Section 8.5.3). The alignment of fibers with targets required that ICS coordinate the activation of the back-illumination system with exposures from the FVC, use the measurements from the FVC to determine the projected location of the fiber tips on the focal plane, and combine these data with astrometry from the GFA to project the fiber tip locations into sky coordinates. The observations with ProtoDESI were extremely valuable to develop and test all of these software tools, as well as to develop tools and procedures for the eventual calibration and commissioning of the complete FPA. In addition to fiber positioning, we used ProtoDESI to characterize the GFA detectors, such as to demonstrate that the GFA CCD met the quantum efficiency requirement. We also used these observations to develop more optimal GFA exposure times to minimize errors due to atmospheric scintillation, decrease heating of the GFA unit, and improve the guiding algorithms.

The ProtoDESI experience was invaluable to help plan DESI commissioning. One impact was that we more thoroughly defined the purpose, conditions, data requirements, analysis plan, criteria for success, time estimate, and one or more responsible parties for each commissioning task. Another was that we identified a number of new tasks. These included the development of robust tools for automatic focus sequences, use of the GFAs to refine fiducial positions, telescope tracking and guiding tests, and positioner dither tests. We also identified multiple precommissioning tasks that better prepared us to analyze data once we went on sky. These tasks included a more complete development of data models and interfaces, a more thorough plan for the metrology of the FPA, enhanced lab testing to minimize the number of systems that would be tested on sky for the first time, and motivated us to substantially expand the core group of experts who would contribute to commissioning.

Development of the CI began in 2017 in order to take advantage of a forecast 2 month schedule gap between the installation of the new corrector system and the readiness of the FPA for installation. The CI was designed to realize an early start on commissioning tasks related to the telescope, the performance of the corrector and corrector system, guiding algorithms, the AOS, the FVC, and the ICS. We determined that tasks from these categories could readily fill the expected 2 month period with a fairly simple arrangement of imaging cameras combined with illuminated fiducials. For this reason, and due to the earlier success of ProtoDESI, we decided that the CI would not include fiber positioners, nor a fiber feed to a spectrograph. To obtain good value from the CI, we used the GFA requirements to establish similar requirements on the location, tilt, FOV, noise, pixel size, and readout time on the imaging cameras that would be part of the CI.

The main components of the CI were five commercial SBIG STXL-6303 cameras and 22 illuminated fiducials. These SBIG cameras were chosen because their CCD array and pixel size (3072 × 2048, 9 μm pixels) produced a 28 square arcminute FOV that was comparable to the 25 square arcminute FOV of the GFAs, and with somewhat higher sampling (013 pixel⁻¹ versus 02 pixel⁻¹) and better noise properties, although somewhat lower efficiency. The fiducials were identical to those produced by Yale (Baltay et al. 2019) for the FPA. The cameras and fiducials were supported by a short, steel cylinder called the CI FPA. Four of the five commercial cameras were located at the same off-axis distance as the GFAs on the focal plane system, while the fifth provided unique, on-axis imaging through the corrector. Behind the cameras was an electronics rack that contained a dedicated NUC PC for each of the five SBIG cameras, a simplified DESI petal controller electronics box supplied by the University of Michigan to operate the illuminated fiducials, temperature sensors, an ethernet switch, and a Raritan PDU for remote power control. These electronics are shown in the left panel of Figure 23. Each camera had an SDSS r-band filter mounted immediately in front of its shutter to mimic the bandpass of the GFA cameras. Most of the cameras and their filters are visible in the middle panel of Figure 23. One of the requirements of the CI FPA was that it would match the 815 kg mass and moment expected for the FPA in order to assess the expected pointing and tracking performance of the telescope with the FPA. We met this requirement by surrounding the CI FPA with a larger, more massive steel cylinder. An additional 221 kg of counterweights were attached to the corrector cage to mimic the mass of the FPE.

Zoom In Zoom Out Reset image size

The greatest challenges in the CI development were the strict alignment and metrology requirements. The five cameras and 22 fiducials needed to be aligned to the aspheric focal surface. The alignment requirements were that the full area of each CCD and each fiducial had to be within 50 μm of the focal surface, and that each CCD had a tilt of no more than 005 relative to the best-fit local tangent. The metrology requirements were to measure the x, y positions of each fiducial with ±10 μm accuracy, and the orientation of each CCD and the nearest fiducial with ±5 μm accuracy. We met these requirements with a careful metrology program that was based on a combination of CMM touch probe metrology and a custom microscope with an integrated projection system (Coles et al. 2018).

The 2 month CI campaign was very successful because it retired multiple risks and identified many areas for further work with sufficient time to address them (approximately 6 months) before the start of commissioning. One major accomplishment on the first night was the demonstration that the new corrector system could produce subarcsecond images across the entire FOV, including at the center where the focal plane system does not have an imaging capability (see Figure 3 and 4). One important discovery was that the historical value for the radius of curvature of the primary mirror was incorrect by 18 mm, which corresponds to a shift in the position of best focus by 9 mm. Fortunately the hexapod had a focus range of ±10 mm and was able to compensate for the difference. In the midst of the CI campaign we used this measurement to adjust the position of the cage to place the center of the hexapod range at the best focus. Another measurement with the CI was the amount of field rotation during exposures as a function of position on the sky due to a combination of polar axis misalignment and variations in atmospheric dispersion. We used these data to develop a model for field rotation that we subsequently refined during commissioning of the full focal plane system. We also made substantial progress on many commissioning tasks, including a new pointing model for the telescope, an initial assessment of the throughput of the corrector system, further development of guiding algorithms, measurement of the performance of the ADC, and development of ICS, PlateMaker, and the AOS, including mapping the hexapod settings to keep the corrector aligned to the primary mirror over the full range of telescope pointing for the survey. Lastly, we obtained substantial experience with scheduling personnel, documentation, and organization of commissioning activities that helped the subsequent commissioning proceed more smoothly.

The completion of the subsystem acceptance process marked the formal hand off of each subsystem to the installation team. In most cases the subsystems were provided by collaboration member institutions (or partners), including several international collaborators, and not directly by LBNL. As a consequence, the acceptance process often represented the formal delivery of equipment provided as a contribution to membership, and in some cases marked the point when the scientists and engineers who developed a subsystem would complete their work with DESI and move to other projects. As a result, the acceptance process was used to ensure we had sufficient information to complete installation and integration and maintain the subsystem during operations (in practice, a subset of those groups formed our commissioning team and the current operations team, and the remainder are available for consultations as needed). We consequently created a Subsystem Acceptance Board to review the requirements on each subsystem, verify that we had sufficient documentation, and to make a recommendation on acceptance to the Project Manager. The Subsystem Acceptance Board was chaired by the Instrument Scientist and also included the two Project Scientists, the Systems Engineer, and the Acceptance, Integration, & Testing (AI&T) Scientist.

Each hardware subsystem had a review prior to shipping. In practice, this was a dry run for the formal, post-delivery acceptance review. The pre-ship review included documentation for all requirements that could be verified prior to integration with other subsystems. These include laboratory tests such as precision metrology, analyses such as the performance of the as-built lenses in an optical model, and documentation of requirements met by design, for example, that the petals had the correct number of fibers for the sky monitor. The review ensured that the system was in compliance with the relevant ICDs, that the ICD documentation was mature and complete, and that the documentation described the procedures for installation and long-term maintenance in sufficient detail. The pre-ship review frequently helped identify missing documentation well in advance of the final subsystem acceptance review.

We anticipated that not all subsystems would meet all requirements and maintained margin against this eventuality. We also created a Nonconformance Review Board to review any instance where a subsystem failed to meet requirements. This Board initially comprised the AI&T Manager, L2 Manager, Project Manager, System Engineer, and Instrument Scientist, and we typically added one or more other experts from within the collaboration. The Board performed an initial judgment of the scope of any cost and/or schedule impact of the nonconformance. If the scope was judged to be major, the Board was expanded to include the Project Scientists and Director. Some conditions that led to nonconformances were that a subsystem failed to meet performance, technical, or material requirements or that it failed to meet a requirement in ICDs or other design documents. If the nonconformance could not be easily addressed, the Board conducted an analysis of the potential implications for higher-level requirements and drafted a waiver request. Some examples of waiver requests include that the efficiency of the blue gratings in three spectrographs was somewhat below the requirement and that the ADC rotator exhibited some vibrations at the required speed. In the case of the blue gratings, the throughput of other spectrograph components exceeded their specifications, so the net throughput of the spectrographs were compliant. In the case of the ADC rotator, further study showed that the vibration was minimized at 50% nominal speed, and that this speed was sufficiently fast relative to the telescope and dome that it would not affect the length of the inter-exposure sequence. Waiver requests were submitted to the Subsystem Acceptance Board as part of the documentation for subsystem acceptance.

Subsystem acceptance occurred after post-delivery inspection. The rationale for this timing was to place the responsibility for shipment on the partner who would prepare it for shipment. The partner would also retain responsibility for the subsystem through post-delivery inspection and testing, as the partner would be best equipped to address any issues that arose during shipment. The degree of post-delivery inspection and testing varied substantially between subsystems. In some cases, such as the spectrograph cryostats, it was not practical to conduct on-site testing until after the cryostats were installed on the spectrographs in their final location, so the post-delivery inspection was simply a visual inspection for shipping damage. In other cases, such as the corrector system, there were substantial verification tests during the reassembly of the corrector prior to installation. Once the post-delivery inspection and testing were complete, the Subsystem Acceptance Board reviewed the subsystem. In all cases, the Board recommended acceptance to the Project Manager.

The planning and execution of DESI integration with the Mayall occurred over many years, starting with the upgrades to the TCS and other facility improvements described in Section 10. In this subsection, we describe the integration stages of the new instrumentation. These include the removal and replacement of the top end and the installation of the new corrector, modifications to the bottom end, construction of the thermal enclosure and support structure for the spectrographs, installation of the new focal plane instrumentation, and routing of the fiber cables and other utilities. In addition to the CD reviews, there was a dedicated DOE review of the installation planning in 2017 April. This planning included careful attention to personal and equipment safety procedures, as well as the detailed installation and lift procedures for all components. The peak of the installation activity occurred during shutdowns of the telescope for the corrector system installation from 2018 February to 2019 March and for the focal plane system installation from 2019 May through 2019 September. During both of these periods, the telescope was locked into position. The interlude between these two phases corresponds to the CI campaign, which was described in Section 11.2. See Besuner et al. (2020) for a more detailed description of the entire installation process.

12.2.1. Top End and Corrector

We began preparations for the removal of the top end with the removal of the KOSMOS (Martini et al. 2014) instrument from the Cassegrain focus in order to center an alignment telescope on the rotator axis with the aid of a cross-hair marked on the f/8 secondary mirror. Our next steps were to remove the Mosaic prime-focus imager (Dey et al. 2016) and corrector optics (Sawyer et al. 2010), the f/8 secondary mirror, and associated utilities. We then disconnected or severed the air, data, and electrical connections running to the top ring assembly and added temporary counterweights to account for the removal of these items. At this point we locked the telescope into position at the zenith and started to disassemble the bottom end, which includes the Cassegrain guider and rotator and the central baffle assembly. We then removed the primary mirror, which was re-aluminized and stored safely off the telescope until the completion of the work on the top end; however, we did reinstall the empty primary mirror cell and the Cassegrain rotator to facilitate later alignment of the new DESI top ring. As the Serrurier truss would be unstable without the upper ring in place, we braced the truss to prevent any relaxation or rebounding of the truss upon removal of the old ring. We also constructed a two-level work platform (visible in Figure 24) to provide personnel access to the upper assembly. These access platforms were useful for the removal of the remaining hardware on the top rings in preparation for the removal of the ring assembly. We designed and built a temporary hydraulic jack system to aid in the separation of the ring from the truss.

Zoom In Zoom Out Reset image size

We hired a 450 ton, 300 foot crane to remove the old Mayall top ring and transfer the new ring from the ground level into the dome. To remove the old ring, we first used the jack system attached to the truss members to slightly raise the ring off of the truss pads. This was more straightforward than feared, as fortunately the ring had not become rusted in place over the preceding 45 yr. On 2018 June 14, we used an exterior crane to lift the old top end, maneuver it through the dome slit, and place it on a flatbed truck that carried it to its retirement home in the parking lot by the Kitt Peak Visitor Center. We then used the same crane to lift the new top ring from the ground floor into the dome. Both of these operations were completed with an exterior crane, rather than the 50 ton dome crane, because the top ring is too large to fit through the hatch between the dome floor and the ground floor of the Mayall unless the ring were rotated such that the longest dimension was aligned with the diagonal of the hatch. The exterior crane was a less risky and less complex solution.

The new ring, vanes, and cage for the top end arrived on 2018 April 19, before the removal of the old top end assembly; they were stored in the ground floor garage until the old assembly was removed. The first step of their installation was to match-drill the locations of the alignment pins and bolts that would be used to fasten it to the truss. As the ring experiences stress and some deformation with the mass of the corrector installed in the cage, we put the ring into a similar state of stress and tension in order to accurately mark the locations for the pins and bolts. We reproduced the expected deformation (a few millimeters) with an assembly that replicated how the vanes attached to the cage and provided a comparable amount of tension.

The next stage of installation was to match-drill the pretensioned upper ring. We lifted the ring with the pretensioning system to the top of the truss with the 5 ton dome crane, used the alignment telescope previously centered on the Cassegrain rotator axis to align the ring to the axis, and match-marked the holes for the alignment pins and bolts. We then lowered the ring to the M floor, drilled and tapped for the pins and bolts, and lifted to the C floor for integration with the cage and vanes. Prior to the start of this final assembly, we added strain gauges to the vanes to help balance the forces on them. We used a laser metrology system to center the prime-focus cage within the top ring.

The new corrector and ancillary alignment equipment arrived on 2018 July 13 by charter aircraft from the UK. We carefully unpacked each of the three corrector segments, inspected them on a rotary table, and reassembled and tested the complete barrel. We then integrated the corrector with the hexapod, which required the temporary separation of the forward section of the corrector from the other two sections. All of this work occurred on the ground floor of the Mayall. In early 2018 November, we used the 50 ton dome crane to lift the corrector plus hexapod to the dome floor, where we integrated it with the new prime-focus cage in the new top ring. Finally, on 2018 November 28, we used the 50 ton dome crane to install the 10,700 kg top end assembly onto the top of the Serrurier truss.

12.2.2. Spectrograph System

12.2.3. Focal Plane and Fiber System

We began preparations for the assembly of the focal plane system at the Mayall during the CI campaign in 2019 April and May. As described in Section 5, we chose to ship individual petals from LBNL and integrate the system on the Mayall after review of the relative merits of integration on the telescope compared to shipment of the fully integrated focal plane system to the Mayall. The greatest challenge in the on-site integration was the insertion of the 10 petals, as each petal is a large, complex, delicate piece of equipment with only 0.6 mm of clearance relative to its neighbors. Each petal had to be inserted at a unique angle (increments of 36° about the corrector axis) relative to its neighbors. In addition, the insertion direction was tilted by ∼72 relative to the horizon, as that corresponds to the orientation of the telescope when locked into place at the Southeast Annex maintenance position.

To overcome these challenges, we built a petal insertion structure called the "Sled" (see Figure 25). The Sled consists of two large, welded steel frames connected to each other by a set of six adjustable-length struts that control six degrees of freedom. The lower frame of the Sled is rigidly mounted to the Southeast Platform of the Mayall facility. The upper frame supports a pair of linear motion guide rails and a central lead screw. Prior to petal insertion, we adjusted the struts to align the rails parallel to the axis of the telescope when it is parked in the maintenance position. To install a petal, the Petal Mount Adapter (PMA) is driven along the Sled's linear rails, and the base plate of the PMA has linear bearings and a coupling to the Sled's lead screw. Six adjustable-length struts connect this base plate to a roughly tetragonal steel frame above it, and mounted to this frame is a steel rotation arm. We attach a petal to one end of the PMA arm and a counterweight to the other. The rotation axis of the arm is nominally colinear with the corrector axis. This allows us to clock the petal to its designated angular position in the focal plane. We then use the PMA struts to finely align the petal to the focal plate adapter ring. At this point the petal is ready for insertion by driving the PMA down the Sled rails. All driving torques during the petal insertion are provided via handwheels, with significant gear ratios between the operator and the load. Each strut has an inline load cell, so that we can quantitatively monitor the net reaction forces between the telescope and the petal during insertion and load transfer. The cells have a resolution better than ∼10 N.

Zoom In Zoom Out Reset image size

One key tool for the alignment process was a Faro VantageE Laser Tracker, combined with Spherical Mount Retroreflectors (SMRs) that were attached to both the focal plane adapter ring on the telescope and to each petal. The SMRs are mounted in "nests" that magnetically hold them against repeatable contact points, and the locations of the contact points were measured at LBNL with a CMM. We measured the relative locations of the SMRs on the focal plane adapter ring and on each petal during installation. Typically these measurements had uncertainties of less than 100 μm, and better than 50μm was achieved under good conditions. Shourt et al. (2020) provide a more extensive description of the alignment and integration of the FPA with a laser tracker.

While the CI campaign was in progress, we conducted a mock installation exercise on the ground floor of the Mayall building. This included a custom corrector simulator with an identical interface flange to the end of the corrector, oriented at the same ∼72 angle relative to the ground, as well as used dummy petals with the same envelope, mounting features, mass, and center of mass. This mock installation led to a number of improvements in the installation procedure.

As soon as the CI was removed from the telescope, we moved the installation fixtures for the focal plane system from the ground floor to the Southeast Annex. The first petal arrived on 2019 May 8, and the remainder followed at intervals throughout June and July. Before installation, each petal was briefly powered on, back illuminated, and tested to determine if any damage had occurred during shipment. We installed the first petal on 2019 June 25. The installation of the remaining petals proceeded smoothly, and we installed the final petal on 2019 July 23. Figure 25 is a photo of the first petal ready for installation.

One unforeseen challenge in this process was a small drift between the telescope and the Southeast Annex platform with time. This drift occurs because the telescope is mounted to a rigid, concrete pier that is mechanically isolated from the rest of the Mayall building in order to maximize its stability. During the petal installation, we observed differential motion as large as a few tenths of a millimeter in tens of minutes, which we attribute to differential thermal effects on the Mayall building and the pier. We adjusted our procedure to minimize the time between the final measurement and adjustment and the insertion of the petal. Throughout this process we enclosed the workspace as much as practical with plastic sheeting and applied a slight overpressure to maintain a clean and moth-free environment.

Once the FPA was complete, we installed the FPE and routed the fiber cables to the spectrographs. The FPE is mounted directly to the cage, rather than the corrector, to reduce the total mass and moment attached to the hexapod. The FPE thermally isolates the FPA at a constant temperature, as well as provides a clean and low-humidity environment. The frame of the FPE includes four rigid "tower" assemblies that are parallel to the corrector axis and extend the full length of the FPE. The fiber cables, electrical cables, and hoses route through these towers. From the FPE, the fiber cables route to the upper ring along the Southeast and Northeast upper spider vanes, down the telescope truss, and through the hour angle and decl. axes. NOIRLab staff designed and built a series of cable carriers for the fibers that avoid twists and enforce their minimum bend radius of 200 mm. The fiber cables then extend through custom penetrations in the walls of the Large Coudé Room and into the Shack. Once we completed the fiber cable installation, we installed the four calibration lamp boxes on the Mayall's top ring.

The two goals of the Functional Verification (FV) phase were to verify that subsystems were not damaged during installation and to integrate them with the other subsystems. The main focus of this integration was software, as hardware integration occurred during installation. We chose to not spend time on the re-verification of requirements during this phase. Instead, we emphasized the start of on-sky commissioning as soon as practical, as that phase provides more and better opportunities for the verification of the instrumentation.

The emphasis on software integration meant in practice that FV for each subsystem was complete upon the successful integration of that subsystem with the ICS. As a consequence, ICS was the first subsystem to begin FV and the last to finish. Functional verification for ICS began early in the project with the TCS integration in advance of ProtoDESI. FV for the other subsystems occurred as soon as they were installed. In practice this meant that the Shack was the next subsystem to go through the FV process, as it was needed to install the spectrographs and cryostats. The Shack was followed by the corrector, as it was needed before the installation of the CI, and then the FPA. Spectrograph FV occurred after each spectrograph arrived and was installed in the Shack.

The main step for the Shack FV was the ingestion of telemetry data (e.g., temperature and humidity) from the WEC control computer to the ICS, which we achieved with a dedicated communications link from the telemetry available from the control computer to our Shack controller electronics box. The Shack controller also reads the pressures of the air supplies for the spectrograph seals and the spectrograph purge system, and it has the capability to power cycle any of the spectrograph controller electronics boxes. During FV of the focal plane, the Shack controller was also used to control the stand-alone fiber illumination system.

Spectrograph and cryostat FV proceeded as each spectrograph and its cryostats were installed in the Shack. The tests for each spectrograph included both a repeat of all pre-ship functional tests and a subset of the pre-ship qualification tests. The functional tests included the operation of the mechanisms, operation of the fiber illumination system, and readout of the detections. The qualification tests were measurements of the read noise, well capacity, gain, nonlinearity, dark current, and CCD cosmetics. Once the focal plane and calibration systems were installed, we measured light leaks, image quality, spectral resolution, crosstalk, and stability. We also used the Hartmann doors to check the focus and make minor adjustments to each cryostat. As the last spectrograph was delivered after the start of commissioning, spectrograph FV was not complete until several months into the commissioning period (2019 December). By that time, we had gained sufficient experience with the system that we established a daily calibration protocol. This includes zeros, darks, flat fields, and arc calibration exposures that are typically obtained in the late afternoon. The flats and arc calibrations use the calibration lamp system mounted to the top ring. Figure 26 shows an example of the data from one (SP4) of the 10 spectrographs after the removal of the overscan and bias subtraction steps.

Zoom In Zoom Out Reset image size

FV of the corrector system included the two ADC motors and the hexapod. Both of these units were extensively tested with ICS in advance of installation and this integration was consequently straightforward. Upon installation of the CI, both it and the FVC went through FV. The main step was illumination of the fiducials on the CI, and imaging those spots with the FVC. This produced the first recorded images through the new corrector. On-sky observations, described in Section 11.2 and shown in Figure 3, demonstrated the superb performance of the corrector optics.

Functional verification of the focal plane and fiber system occurred in three phases. The first phase was a basic checkout of the cooling services. This included leak and pressure checks, the operation of the liquid chiller, the cooling fans, temperature sensors, and humidity sensors, and their integration with ICS. The second phase was a basic functional test of the positioners and GFAs. The positioner operation was first checked with visual inspection, and then the illuminated positioners were imaged with the FVC. At this time, we verified that ≥99% of the fibers were intact after installation. The GFAs were also turned on and operated under ICS control. Finally, the third phase combined the calibration of the positioner parameters with the mapping of the coordinate transformations from the pre-ship metrology of each petal to the FVC. This process included accounting for the distortion caused by both the corrector optics and the FVC optics. The positioner calibration included precise measurements of the six parameters per unit, which are required to accurately transform between motor rotation angles and fiber tip coordinates on the focal surface. This mapping is critical for accurate alignment of the fiber tips with astrophysical sources and to avoid collisions between positioners. Larger uncertainties in these parameters would require larger tolerance regions around the moving components, which would impact fiber assignment. The instrument was ready for on-sky observations in 2019 October upon completion of FV for the focal plane and fiber system.

The total system throughput was one of the most important parameters throughout the project. As described in Section 2.1, we developed a detailed total throughput budget early in the development of DESI to set requirements on the instrument subsystems. We updated this budget throughout the project as we obtained empirical data for many contributions (e.g., optical coatings, FRD, detector response). We obtained our first opportunity to measure the total system throughput with observations of the standard stars that we include in all petals on every field we observe, and we gradually refined these throughput measurements as we acquired more data and improved the fiber positioning accuracy. Figure 27 shows the total measured throughput from standard stars observed over many photometric nights, both with and without corrections for atmospheric absorption based on the extinction curve for Kitt Peak. The measurement with atmospheric extinction has been corrected to airmass X = 1.

Zoom In Zoom Out Reset image size

The measurements shown in the figure are based on observations over nights that exhibited excellent image quality (seeing between 07 and 09 FWHM). We determined the fractional flux in a fiber for each observation based on stacked images of the guide frames that were obtained in conjunction with the spectroscopic observations. This fiber aperture loss is about 40% for our typical seeing of 11 and increases as the seeing degrades. The total throughput shown in Figure 27 is corrected for fiber aperture loss, and this correction takes into account the variation in the projected fiber size with position on the focal surface. We assessed that we obtain a consistent throughput using data observed with worse seeing, which imply a higher correction. The figure also shows the predicted throughput from the system engineering model, which is the total throughput budget based on the best empirical data or analytic estimate for individual contributions. The agreement between the data and the engineering model is extraordinary for the blue channel and very good, within 10%, for both the red and NIR channels. The mild disagreement between the prediction and model for the two redder channels is likely due to many small discrepancies in the engineering model.

The previous subsection demonstrated how DESI provides exceptional throughput. The next subsections show how well the telescope, corrector, and focal plane deliver the light from our targets to the fibers. The top-level requirement on the telescope guiding performance (L3.3.3) is the assumption that the telescope guiding accuracy is 100 mas rms 1D. Multiple factors contribute to the guiding accuracy, including the telescope pointing model and tracking, the performance of the algorithm that identifies and measures sources on the guider chips, the algorithm that combines the measurements to create correction signals in R.A. and decl., and the correction for field rotation with the hexapod.

We used the guider cameras to validate the performance of the corrector and telescope after the installation of the focal plane system. Figure 28 shows the distribution of measured seeing values from guider images obtained between 2019 October 23 and 2020 March 15. These FWHM measurements were obtained from the measured radial profile of stars on a guider chip. The best FWHM measurements are between 06 and 065 and demonstrate the excellent performance of the corrector system. The median is 111, which is superior to the median of 12–13 measured by Dey & Valdes (2014). For typical seeing of 11, about 60% of the flux from a point source enters the fibers. Figure 29 shows the FWHM as a function of zenith distance, which demonstrates the excellent performance of the ADC and the overall stability of the top end and hexapod control of the corrector system.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

We obtained several new data sets to update the pointing model before optimizing the guiding performance (see Section 10.2). The latest model from 2021 April 1 yielded excellent pointing and tracking performance. The median performance was about 100 mas from 2021 January through April on each axis, and this is shown in the right panel of Figure 28. The performance is slightly better along the decl. axis, as expected for an equatorial mount telescope. The measurement is after a correction for field rotation during the spectroscopic exposure. We still plan to implement a range of filters on the raw guide signal to improve this performance.

The overall performance of the GFAs has been excellent. We regularly monitor the dark current and read noise, and these studies have shown very good overall stability. The dark current tracks the CCD temperature in a consistent manner over the typical operational temperature of 10°–11° C. There is a modest amount of variation in the read noise that we attribute to low-level pattern noise. The read noise has consistently remained below the requirement of 25 e⁻ rms per pixel.

Our method to ensure the 107 μm diameter fibers are centered on astronomical targets combines many complex elements. These include the alignment of the fiber tips with the aspheric focal surface of the corrector, the tilts of positioners relative to the focal surface, precision metrology of illuminated fiducials and GFA pixels, optical distortion models of both the corrector and the FVC camera lens, and a model for atmospheric refraction. To verify all of these individual steps, we developed an end-to-end test based on a series of small, random offsets of each fiber that we refer to as fiber dithering.

Fiber dithering treats the calculation of fiber positioning errors as a nonlinear optimization problem. The model parameters are the flux of each star assigned to a fiber, the 2D misalignment of each fiber relative to the nominal location, the seeing, the transparency, and any telescope offsets between each exposure of a dither sequence. The basic scheme is that we take some number of exposures ${N}_{\exp }$, with some number of fibers N_fibers assigned to stars. Between each exposure, we dither each fiber in a separate, random direction by some known amount. We then measure the integrated flux of the source in each fiber in each exposure with the spectrographs, as well as measure the precise, relative motion of the fiber positioners with the FVC. The dither pattern needs to match the PSF size to get good centroid measurements for the positioners. If the positioning errors are much larger than the PSF size, then positioners may have no dither positions that collect the light of the target star, and we would not measure their centroids. In this case, we still dither the fibers by the PSF size between exposures, but we also add larger (∼5'') dither steps onto each target to attempt to align an occasional positioner with the target star on different parts of the focal plane.

We used this dithering analysis on multiple occasions during commissioning as an integrated check of all of the components of fiber positioning. The analysis is especially sensitive to large-scale features, such as shifts between the GFA pixels and other focal plane metrology or errors in the distortion models. Based on the results of several fiber dither sequences at a range of zenith distances, we rechecked the raw focal plane metrology data and replaced the original FVC lens with the simpler design described in Section 5.8. Within about 2 months, these improvements and checks led to improvements in fiber positioning from initially ∼10'' rms to the current performance of ∼01. For reference, offsets of 01 (02) correspond to throughput losses of 0.07% (3%) for a galaxy with a half-light radius of 035 observed in 11 seeing. Figure 30 shows one illustrative result of the dither analysis from 2021 December 19 with data from the red channel of the spectrographs. The dithering analysis also provides a direct way to measure the total throughput independent of fiber positioning errors.

Zoom In Zoom Out Reset image size

We attribute the remaining radial pattern to residual errors in the distortion model. These residual errors vary between fields and therefore have not been removed. Both of these variations between fields, and variations between the blue, red, and NIR channels, are of the same order of magnitude as the variations shown in the figure. In addition to this distortion pattern, dome seeing and turbulence effects also contribute to fiber positioning inaccuracies. Nonmoving devices with known stationary positions in the focal plane—both fiducials (123) and nonfunctional positioners (of the order of 700)—provide a measure of the turbulence field at each iteration of the positioning loop. We fit this field and correct all measured positions for the turbulence field. Based on a dither analysis with and without these corrections, we find that the turbulence correction produces a ∼10% improvement in positioning.

The spectral extraction code employs a model of the spectrograph PSF to accurately measure the flux, noise, and spectral resolution of every spectrum and subtract night sky lines. Requirement L3.1.6 states that the PSF bias shall not exceed 1%, where the PSF bias is δ F/F = Σ_pi q_i /Σq_i q_i − 1; the two quantities are the intensity distribution of the true PSF p_i and the linearly interpolated PSF q_i based on the arc calibration data, and the sum is over all i pixels that contribute to the PSF. This bias captures the fractional difference between the true PSF shape and the linearly interpolated model, and determines the sky subtraction accuracy. We measure the PSF from arc calibration frames obtained as part of the daily calibration protocol (typically afternoon calibrations). The main factors that are expected to produce changes in the PSF are changes in forces on the fiber cables and environment-induced changes in the spectrographs. The forces on the fiber cables change when the telescope moves to different positions on the sky and when the positioners move the fiber tips within their patrol regions. Changes in the spectrographs could be produced by changes in the temperature and humidity of the Shack environment.

We have measured the size of the PSF bias with numerous, dedicated tests to determine if daily (afternoon) measurements of the PSF from arc calibration data are sufficient. For one test, we fixed the telescope at the altitude of the dome flat screen, moved the telescope to different azimuth locations in the dome, rotated the dome to place the flat field screen in front of the telescope, and took a sequence of arc calibrations. For a second test, we kept the telescope in the same position and took a sequence of arc calibrations with the fiber positioners in different configurations. Figure 31 shows the variation in the PSF bias measured with calibration data obtained on three consecutive afternoons after nighttime observations. After each night of observing, the fiber positioners are in different locations, as they are left in the configuration for the last observation of the night. The PSF bias is <1% for nearly all fibers, and it is sufficiently stable to obtain a sky subtraction precision better than 1%. Figure 32 shows two examples of the superb sky subtraction with spectra of ELGs at z > 1. These are particularly challenging objects because redshift measurements require detection of the faint, marginally resolved [O ii] doublet, which is in the midst of significant atmospheric emission at these redshifts. The sky subtraction algorithm and PSF stability will be described further in J. Guy et al. (2022, in preparation).

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

In addition to this superb stability, we assessed that the spectrographs delivered the expected spectral resolution. We measure the spectroscopic resolution with the afternoon arc calibration described in Section 12.3. Specifically, we measured the 2D PSF of each fiber in each spectrograph as a function of wavelength, convolved this model with the CCD pixels, projected the model onto the dispersion axis, and then measured the FWHM of the profile. Figure 33 shows a collection of resolution measurements for a subset of fibers in all 10 spectrographs from 2022 January. This figure demonstrates that the spectral resolution exceeds the requirement. In the red and NIR channels, this resolution is about twice that of SDSS and BOSS, and this enables DESI to observe the faint continuum between the OH sky lines and resolve the [O ii] doublet from high-redshift star-forming galaxies.

Zoom In Zoom Out Reset image size

During commissioning and early in the early operations phase, we worked to minimize the length of the inter-exposure sequence, which is described in Section 8.5 and summarized in Figure 21. Our requirement is that the inter-exposure sequence be less than 2 minutes, which is a Level 4 requirement that flows down from the requirement to maximize the survey efficiency.

The times for the spectrograph CCD readout and the setup of the telescope are as short as practical, and both begin as soon as the shutter closes. The spectrograph readout time is 60 s. This readout occurs in parallel with other tasks during the inter-exposure sequence, and those take sufficiently long that the inter-exposure sequence is not limited by the spectrograph readout time. The time to move the telescope, dome, and ADC depend on the relative locations of the current and next field. Early in the project, we used the move parameters of the telescope's hour angle and decl. drives, the dome motors, and the ADC motors to simulate the distribution of times for each of these motions to complete for simulated survey observations. These actions happen in parallel, and the telescope dominates 88% of the time, while the dome dominates the remaining 12%. The mean (median) time for these moves to complete is 41 s (27 s). Based on our operations experience to date, moves complete somewhat more quickly, possibly because we have continued to optimize the survey software to minimize slew distances. While the telescope is in motion, we adjust the location of the hexapod for the new position based on a static look-up table and perform the blind of the fiber positioners to set up on the new field. The hexapod adjustment takes approximately 5 s, and the execution of the blind move takes 35 s.

Once the telescope is in position, we obtain and analyze a 15 s acquisition image with the guider GFAs. We have preestablished guide stars for every field, and we have preestablished where those guide stars need to be located on the GFAs in order to ensure that the fibers are aligned with their targets. It takes 4 s to analyze the GFA images and about 7 s to adjust the telescope pointing, start the hexapod rotator, and initialize the guider. In total, we typically complete the field acquisition and are ready to begin guiding about 30 s after the end of the telescope slew. For very small slews, we are ready to begin guiding as soon as 40 s after the spectrograph shutters closed to end the previous exposure. Once we begin guiding, we are ready to back-illuminate the fibers, turn on the fiducials, obtain an FVC image, and calculate and execute the correction move. The correction move is typically complete 51 s after guiding begins. Once this step is complete, we take a second FVC image to record the final locations of the fiber positioners, turn off the back illumination and fiducials, and are ready to start the exposure 15 s after the completion of the correction move. At present we have achieved inter-exposure times as short as 118 s between fields that are close together on the sky.

Early in the operations phase, we experienced problems with the CAN communication to positioners. This would manifest as a loss of communication with an entire CAN bus and cause the temporary loss of communication to the devices on that bus. This would result in the loss of 28–78 devices. While we could recover from this error condition, the recovery procedure was sufficiently long that it proved more efficient to observe the rest of the night without these positioners and to perform the recovery procedure the next day. We traced this problem to communications problems with individual positioners that would disrupt an entire bus. To minimize disruption, we redesigned and changed all of the transverse boards to add a relay for each positioner. This provided the flexibility to switch off individual problematic positioners, rather than risk temporary loss of an entire bus. We also increased the number of CAN channels from 10 to 20, which reduced the maximum number of devices per CAN bus to 28. As the transverse boards are directly behind the petals, this required a major operation: the disassembly of the FPE and the removal of the petals to obtain sufficient access. We successfully conducted this upgrade during the summer 2021 monsoon season, and we are now able to maintain higher observing efficiency.

The other main performance issue that we identified during early testing at the Mayall was that some positioners would occasionally stop moving the commanded amount, which we traced to performance problems with the motor for either the ϕ or θ arm. The most common failure mode was that the motor would start to move a fairly constant fraction of the requested move, and we referred to this as the "linear ϕ" (or θ) problem. When the petals were removed in the summer of 2021, we extracted several of these problematic positioners that were near the edge of a petal and consequently possible to extract with minimal risk to neighbors. High-resolution CT scans of these positioners showed the pinion gear inside of the motors had cracked radially through the wall thickness, which allowed the gear to slip relative to the motor shaft. The pinion gear had been attached to the motor shaft via a press fit. We hypothesize that some assemblies may have exceeded the (very challenging) tolerance limits that press fits at this small scale can require, and that the stress induced in these cases initiated a slow-growing crack in the affected gears. A particular challenge in the detection of the failures and then the identification of the root cause was that affected gearboxes may spin properly in the horizontal orientation (typical of our lab tests) and only become symptomatic when pointed downward (as they are at the zenith position of the telescope). While the cracked motors cannot be replaced without rebuilding the petals, we are exploring operational changes to regain the use of the motors that show linear behavior. As of 2022 January, about 14% of positioners are not used on targets due to mechanical or electrical problems, although most of these naturally fall on blank sky positions and are therefore still useful. Some preliminary data suggest that of the order of one new positioner becomes problematic per week, although we are also actively engaged in remediation efforts to restore some of the unused positioners to regular operations.