Brought to you by:
Paper

VirtualDose: a software for reporting organ doses from CT for adult and pediatric patients

, , , , , , and

Published 2 July 2015 © 2015 Institute of Physics and Engineering in Medicine
, , Citation Aiping Ding et al 2015 Phys. Med. Biol. 60 5601 DOI 10.1088/0031-9155/60/14/5601

0031-9155/60/14/5601

Abstract

This paper describes the development and testing of VirtualDose—a software for reporting organ doses for adult and pediatric patients who undergo x-ray computed tomography (CT) examinations. The software is based on a comprehensive database of organ doses derived from Monte Carlo (MC) simulations involving a library of 25 anatomically realistic phantoms that represent patients of different ages, body sizes, body masses, and pregnant stages. Models of GE Lightspeed Pro 16 and Siemens SOMATOM Sensation 16 scanners were carefully validated for use in MC dose calculations. The software framework is designed with the 'software as a service (SaaS)' delivery concept under which multiple clients can access the web-based interface simultaneously from any computer without having to install software locally. The RESTful web service API also allows a third-party picture archiving and communication system software package to seamlessly integrate with VirtualDose's functions. Software testing showed that VirtualDose was compatible with numerous operating systems including Windows, Linux, Apple OS X, and mobile and portable devices. The organ doses from VirtualDose were compared against those reported by CT-Expo and ImPACT—two dosimetry tools that were based on the stylized pediatric and adult patient models that were known to be anatomically simple. The organ doses reported by VirtualDose differed from those reported by CT-Expo and ImPACT by as much as 300% in some of the patient models. These results confirm the conclusion from past studies that differences in anatomical realism offered by stylized and voxel phantoms have caused significant discrepancies in CT dose estimations.

Export citation and abstract BibTeX RIS

1. Introduction

X-ray computed tomography (CT) has experienced tremendous technological advances in recent years and is one of the most useful diagnostic imaging modalities today. Driven particularly by advanced multi-detector CT (MDCT) technologies, the number of CT scans performed each year in the United States had reached nearly 81.2 million in 2014(IMV's 2014 CT Market Outlook Report 2015). The potential radiation risk to the patient population, particularly children, has led to increasing attention from the radiology community in the past few years (Berrington de Gonzalez et al 2009, Sodickson et al 2009, Brenner 2010, Ding et al 2010, Boone et al 2012, Pearce et al 2012). For a long time, the as low as reasonably achievable (ALARA) principle (ICRP 1977) has been widely adopted in the radiation protection of patients undergoing diagnostic imaging including CT (Slovis 2003, Kalra et al 2004, McCollough et al 2009, Dougeni et al 2012). In its Publication 102, the international commission on radiological protection (ICRP) emphasized the importance of managing patient dose, particularly from repeated or multiple examinations (ICRP 2007b). Recently, several task groups from the american association of physicists in medicine (AAPM) developed methodologies for the evaluation of CT doses (AAPM 2008, 2010), including the size-specific dose algorithm for pediatric and adult CT examinations (AAPM 2011). Public campaigns such as the dose index registry (DIR) (ACR 2012), Image Wisely (Wisely 2013), and Image Gently (Gently 2013) have been initiated to engage the radiology community. In January of 2015, revised elements of performance (EPs) for organizations that provide diagnostic imaging services have been finalized by the Joint Commission and will go into effect on July 1, 2015(The Joint Commission 2015).

Presently, three types of dosimetric quantities are used in CT dosimetry (Tack and Gevenois 2007, Mahesh 2009, Seeram 2009): 1) weighted CT dose index (CTDIw) and volume CT dose index (CTDIvol), which provide an indication of the average absorbed dose to a cylindrical phantom in the scan region, 2) dose-length product (DLP), which integrates the dose along the length of the scan, and 3) effective dose (E), which is a risk-related method for comparing whole-body patient radiation doses across different imaging procedures. To respond to the increasing trend in CT dose, the States of California, Connecticut, and Texas in the United States have mandated the CT dose reporting in terms of CTDIvol and DLP. However, there are some concerns about the use of CTDIvol as a metric for patient dose, because it does not account for the size or anatomy of the patient (Brenner 2006, Dixon 2006, McCollough 2006, Boone 2007, McCollough et al 2011).

Radiation-induced health effects are correlated with the mean absorbed dose to organs and tissues. The absorbed dose is determined as the quotient of mean energy imparted from any type of radiation and the mass of any irradiated material of interest. To quantify the whole-body risk, the ICRP recommends the effective dose as a radiation protection quantity, which is based on the weighted sum of selected major radio-sensitive organs or tissues according to:

Equation (1)

where DT,R is the average absorbed dose in tissue T from the radiation type R. wR is a radiation weighting factor accounting for the relative biological damage of different types of radiation (and is always unity for x-rays), and wT is a tissue weighting factor for T derived from that tissue's relative radio-sensitivity. The set of 'tissue weighting factors' has been revised periodically to reflect the latest epidemiological information, most recently in ICRP Publication 103(ICRP 2007a) which replaces the recommendations in IRCP Publication 60 (ICRP 1991). Although, the ICRP developed the concept of effective dose for the purpose of setting occupational dose limits for radiation protection, and stated that the effective dose concept should not be used to indicate risk for specific individuals, the quantity is still widely used by the radiology community to compare risk for patients who undergo x-ray imaging (McCollough and Schueler 2000, McNitt-Gray 2002, Brenner and Huda 2008). The effective dose is defined only for the ICRP reference adult models (ICRP 2009), but the methodology has been applied to other computational phantoms (Xu 2014).

Several CT organ dose calculation tools are currently available (Kalender et al 1999, Stamm and Nagel 2002, CT Dose 2008, Ban et al 2011, eXposure 2012, ImPACT 2012). Most these existing packages are based on stylized patient phantoms developed prior to the 1980s using overly simplified anatomies. Although stylized phantoms were utilized worldwide both for external and internal dosimetry studies, the stylized models had been found to result in significant dose errors when compared against anatomically realistic patient models (Zanki et al 2002, Liu et al 2010, Lee et al 2011, 2012, Xu 2014). Furthermore, most existing software packages do not consider patient populations other than averaged-sized adults, ignoring pediatric, pregnant, and obese patients.

This paper describes the development and testing of a new web-based software, called 'VirtualDose', for reporting organ doses to patients who undergo diagnostic CT examinations. Funded by a grant from the national institute of biomedical imaging and bioengineering (NIBIB) for commercial development, VirtualDose is designed to improve upon existing software packages by considering validated CT scanner models and scanner-specific correction factors, latest ICRP recommendations, advanced 'software as a service (SaaS)' delivery mode, and a family of 25 anatomically realistic patient phantoms which includes a set of voxel phantoms covering median (50th percentile) adults, children at different ages, pregnant females at three gestational stages, and obese patients of different body mass.

2. Materials and methods

2.1. Twenty-five 'virtual patient' models

Computational phantoms can be divided into three generations with increasing anatomical realism and geometrical sophistication: 1) stylized phantoms developed prior to the 1980s, 2) voxel phantoms developed since the late 1980s, and 3) Boundary Representation (BREP) phantoms developed since the mid of 2000s (Xu 2014). This study took advantage of a total of 25 whole-body BREP phantoms that were previously developed at rensselaer polytechnic institute (RPI) and the university of florida (UF). These phantoms included reference adults representing the ICRP-89 50th percentile (median) of adults (named RPI-Adult-Male (RPI-AM) and RPI-Adult-Female (RPI-AF)) (Zhang et al 2009b, Na et al 2010), pediatric patients at different ages (newborn, 1-, 5-, 10-, and 15 year-old) (Bolch et al 2010), and pregnant females at three gestational stages (named RPI-Pregnant 3-,6-, and 9 month) (Xu et al 2007). A newly developed set of phantoms representing overweight and obese patients (Ding et al 2012) were also adopted for the development of VirtualDose. Figure 1 depicts these phantoms which were originally designed in various BREP data formats that are easy to deform (Xu and Eckerman 2009). They were converted to voxel-based phantoms, as summarized in table 1, to perform MC organ dose calculations.

Table 1. Phantom parameters and body mass index (BMI) of the voxel-based 'Virtual Patient' models used in VirutalDose.

  Mass (kg) Height (cm) BMI (kg m−2) Voxel size (mm3)
Pediatric patient models        
Newborn male 3.27  47.8 14.3 × 2  ×  2
Newborn female 3.27  47.8 14.3 × 2  ×  2
1 year male 9.39  76.6 16.0 × 3  ×  3
1 year female 9.39  76.6 16.0 × 3  ×  3
5 year male 16.45 110.4 13.5 × 3  ×  3
5 year female 16.45 110.4 13.5 × 3  ×  3
10 year male 30.16 140.1 15.4 × 3  ×  3
10 year female 30.16 140.1 15.4 × 3  ×  3
15 year male 53.13 166.5 19.2 × 3  ×  3
15 year female 52.24 161.7 20.0 × 3  ×  3
Pregnant female patient models        
3 month pregnant 61.9 163.2 23.2 × 3  ×  3
6 month pregnant 66.6 163.5 24.9 × 3  ×  3
9 month pregnant 72.4 163.5 27.1 × 3  ×  3
Average adult patient models        
Average adult male 73 176 23.6 × 3  ×  3
Average adult female 60.1 164 22.3 2.5 × 2.5 × 2.5
Obese patient models        
Normal body-weight male 72.7 176 23.5 3.5 × 3.5 × 3.5
Normal body-weight female 63.5 163 23.9 3.5 × 3.5 × 3.5
Over-weight male 85.7 176 27.7 3.5 × 3.5 × 3.5
Over-weight female 75.3 163 28.3 3.5 × 3.5 × 3.5
Obese level-I male 103.1 176 33.3 3.5 × 3.5 × 3.5
Obese level-I female 90.6 163 34.1 3.5 × 3.5 × 3.5
Obese level-II male 117.0 176 37.8 3.5 × 3.5 × 3.5
Obese level-II female 102.4 163 38.5 3.5 × 3.5 × 3.5
Morbidly-Obese male 139.4 176 45.0 3.5 × 3.5 × 3.5
Morbidly-Obese female 123.3 163 46.4 3.5 × 3.5 × 3.5
Figure 1.

Figure 1. 3D rendering of whole-body BREP phantoms used in this study: (a) UF pediatric male and female patients at different ages (newborn, 1-, 5-, 10-, and 15 year-old), (b) RPI adult male and female patients matching with the 50th percentile of population, (c) RPI pregnant female patients at three gestational stages (3-,6-, and 9 month), and (d) RPI obese patients.

Standard image High-resolution image

2.2. CT scanner models

Two CT scanner models (the GE LightSpeed Pro 16 and Siemens SOMATOM Sensation 16) were explicitly constructed in the Monte Carlo code, MCNPX v2.6 (Pelowitz 2005). The GE LightSpeed Pro 16 scanner (figure 2), operated at different tube voltages (i.e., 80, 100, 120, and 140 kVp) with different beam collimations (1.25, 5, 10, and 20 mm), was developed and validated using a previously validated method by Gu et al (2009) that was later refined by Ding (2012). The SOMATOM Sensation 16 was simulated by Lee et al (2011), operated at 80, 100, 120, and 140 kVp with two beam collimations of 10 and 24 mm.

Figure 2.

Figure 2. Typical equipment used in CTDI measurements including electrometer, ionization chamber, and a CTDI phantom (CTDI head or body phantom). The measured CTDI values were used to validate the CT scanner model in MCNPX code by comparing with the simulated results (photo taken at Massachusetts General Hospital, Boston, MA).

Standard image High-resolution image

It had been shown by Turner et al (2010) that organ doses normalized by CTDIvol were practically independent of the scanner type. Therefore, CT scanners other than the scanners validated in this study were corrected by the measured CTDIvol normalized to a tube current of 100 mAs, as described below in section 2.4. In this way, a small number of fully validated CT scanner models can be used to represent nearly any modern CT scanner.

2.3. Monte Carlo organ dose calculations

With detailed geometric and compositional information for dozens of well identified organs, a computational phantom contains necessary anatomical data for radiation dose calculations using a MC radiation transport code, such as MCNPX (Pelowitz 2005). Coupled with a model of the radiation produced by the CT scanner, a MC radiation simulation can produce a detailed distribution of radiation dose across various organs and tissues of the body.

For the purposes of reporting organ doses for a particular patient undergoing a specific CT scan, the scan range was first deconstructed into the individual tube rotations or slices of the scan. A series of separate axial scans from head to toe (as shown in figure 3), were successively simulated by using each specific tube voltage and each transverse beam width in the MCNPX code. For each slice simulated, the direct dose within the scan volume and the scattered radiation dose outside of the scan volume were calculated. The procedure was repeated for pediatric, pregnant female, and adult male/female phantoms, over a very large number of simulations. The MCNPX code can handle voxels efficiently using MCNPX's 'repeated structures' feature, therefore an in-house voxelization algorithm (Zhang et al 2009b) was developed to convert these BREP phantoms into a voxel-based data. The number of source photons was selected to ensure that the calculated organ doses had an acceptable level of statistical uncertainty—relative errors of  <1% in most organs near the primary beam and  <5% for organs with very small volumes or located at large distances from the primary beam. Dose to skin and also bone surfaces and red marrow were handled using the methods demonstrated in previous studies (Zhang et al 2009b, Johnson et al 2011).

Figure 3.

Figure 3. Scheme of the contiguous axial MC organ dose simulations on the patient phantom using a validated CT scanner model: a MC simulation mimics a series of continuous axial scans covering a phantom from the head to toe.

Standard image High-resolution image

The MC simulation results provided organ dose in units of MeV per gram per source particle and they must be adjusted according to the integrated x-ray tube current, which was expressed as the product of tube current (mA) and the exposure time (s). The proprietary nature of the x-ray tube and bowtie filter assembly makes it difficult to quantify the x-ray photon output directly. As a result, an empirical conversion factor (CF) was used to convert the tally output to absorbed dose per unit integrated tube current (in units of mGy/100 mAs). These CFs were unique to each combination of beam energy (E) and beam collimation (NT). A series of CFs were calculated using methods described in previous studies (DeMarco et al 2005, Gu et al 2009):

Equation (2)

where ${{\left(\left(\text{CTD}{{\text{I}}_{100}}\right)_{\text{in}-\text{air}}^{\text{Measured}}\right)}_{\text{E},\text{NT}}}~$ is the measured air kerma (CTDI100)in−air values in units of mGy/100 mAs by using the ionization chamber in air at the CT scanner iso-center for a single axial scan; ${{\left(\left(\text{CTD}{{\text{I}}_{100}}\right)_{\text{in}-\text{air}}^{\text{Simulated}}\right)}_{\text{E},\text{NT}}}~$ is the corresponding air kerma values in units of MeV per gram per source parcel acquired by simulating the ionization chamber in the MCNPX code under the same CT scan scenario. The units of ${{\left(\text{CF}\right)}_{\text{E},\text{NT}}}~$ are expressed as in units of (mGy·gram·source particle)/ (MeV·100 mAs). The CTDIvol, which includes the effect of the bowtie filter at 'off-center' positions, was used to scale the organ doses for different scanners as mentioned in section 2.4.

By using these CFs, the simulated results from the MCNPX code can be easily converted to the absorbed dose according to the following conversion equation:

Equation (3)

where ${{\left({{D}_{\text{absolute}}}\right)}_{\text{E},\text{NT}}}~$ is the absorbed dose in unit of mGy, ${{\left({{D}_{\text{Simulated}}}\right)}_{\text{E},\text{NT}}}~$ is the MCNPX simulation results in the units of MeV per gram per source particle, and ${{\left(\text{CF}\right)}_{\text{E},\text{NT}}}~$ is the conversion factor for the beam energy E and beam collimation NT.

Effective dose (E) was first calculated as a weighted average of the equivalent doses to selected body organs or tissues using the tissue weighting factors specified by the ICRP-60 (ICRP 1991) according to the equation (1). In addition, we adopted the latest ICRP-103 (ICRP 2007a) definition of E as being a sex-averaged value calculated from the averaged equivalent doses of the male and female phantoms using the equation:

Equation (4)

where $H_{\text{T}}^{\text{F}}$ and $H_{\text{T}}^{\text{M}}$ are the equivalent doses for organ or tissue T of the female and male phantoms, respectively. $w_{\text{T}}^{{}}$ is the updated tissue weighting factor for T provided in ICRP Publication 103. $D_{\text{T,R}}^{\text{F}/\text{M}}$ is the average absorbed dose in tissue T of the female or male phantoms from the radiation type R. $w_{\text{R}}^{{}}$ is a radiation weighting factor.

2.4. Organ dose reporting algorithms in virtualdose

Once the axial slice-by-slice dose database has been established, organ doses from a contiguous axial scan corresponding to a specific protocol can be obtained by directly summing the corresponding single axial slices in the scan range. When no-integer number of slices appears in the dose accumulation, a linear interpolation algorithm (based on the scan covered anatomy length among one piece axial scan distance) was applied to interpolate the data at the starting and ending places. In the helical scan mode, the dose calculation depends upon the 'pitch' of the scan, which was the ratio of the patient shift (table movement) during one rotation to the width of the beam. For a helical scan covering the same scan length and a pitch of 1, approximately the same radiation dose results as for a contiguous axial scan resulted in approximately the same radiation dose with the same technique factor (McNitt-Gray et al 1999). For noncontiguous (pitch  >  1) or overlapping (pitch  <  1) helical scans, radiation dose is inversely proportionally to the pitch value if all other scan parameters remain unchanged.

To correct for the use of CT scanners other than the scanners validated in this study, the organ radiation dose DH can be estimated by:

Equation (5)

where DC is the organ dose from CT scans as reported by the scanners in VirtualDose, in unit of mGy, ${{\left({{\left(\text{CTD}{{\text{I}}_{\text{vol}}}\right)}_{\text{E},\text{NT}}}\right)}_{\text{Scanner}}}~$ is the CTDIvol value of the scanner being used in practice for a specific tube voltage (E) and beam collimation width (NT),$~{{\left({{\left(\text{CTD}{{\text{I}}_{\text{vol}}}\right)}_{\text{E},\text{NT}}}\right)}_{\text{VirtualDose}}}$ is the CTDIvol value used in VirtualDose for the same tube voltage and beam collimation width. Because the organ dose in the final slice-by-slice dose database was given per 100 mAs tube current time, the final dose result will be multiplied by the ratio of total mAs to 100 mAs.

In helical CT scans, additional rotations at the starting and ending points of the scan area of interest along the Z-axis were always needed, called Z-over scanning, for the purpose of image reconstruction of the first and last slices (Mahesh 2009, Seeram 2009, Tack and Gevenois 2007). The additional x-ray tube rotations must be included in the scan length and total integrated dose. To account for the z-over scanning, the total radiation dose $D_{\text{H}}^{\prime}$ is calculated in VirtualDose as:

Equation (6)

where ${{\left({{D}_{\text{os}}}\right)}_{{{\text{Z}}_{+}}}}$ and ${{\left({{D}_{\text{os}}}\right)}_{{{\text{Z}}_{-}}}}$ represent the radiation dose from the over scan length as specified by the user (due to the variability of over-scan length with different protocols, techniques, and scanner features).

As a method to reduce the total applied radiation dose, automatic exposure control (AEC) had become commonplace on modern MDCT scanners. AEC used tube current modulation (TCM) to automatically adjust the tube current according to the size and attenuation characteristics of the patient body region (McCollough et al 2006). To determine dose from CT scans using AEC technologies, the average tube current per rotation is used, such that:

Equation (7)

where ${{D}_{i}}$ is the organ dose result (in units of mGy /100 mAs) calculated from the axial MC simulations for the i-th slice in user specified scan region, ${{w}_{i}}$ is the tube current weighting factor (the actual mAs in the i-th image divided by 100 mAs). The slice-averaged tube current, along with CT output (e.g. CTDI, DLP), CT scanner setting parameters (e.g. kVp, mAs, scan protocol) and patient information (weight, height, age, and gender) information may be extracted from the CT digital imaging and communications in medicine (DICOM) files and be imported automatically into VirtualDose to calculate the organ dose. If patient height and weight information is not available, VirtualDose will automatically select an adult phantom as default. Only the data necessary for the calculation need be extracted, so no protected patient information is transmitted to the VirtualDose.

2.5. SaaS architecture and web services interface design

SaaS is a modern software distribution method that hosts all its associated data and up-to-date resources centrally on a remote computer server (SaaS 2012). Different from the traditional software distribution model that requires a user to install and configure the application first, SaaS does not require any installation on user's computers. Using a license pre-assigned by an SaaS provider, users can remotely access the software application, typically through an internet browser on demand at any time from any computer that is connected to the internet.

In this study, VirtualDose was designed as an SaaS application to allow multiple users to simultaneously access the software functions via the Internet. To implement this, a 'service-orientated architecture (SOA)' design was adopted (Erl 2005). As illustrated in figure 4, the SOA architecture in VirtualDose includes 3 different interface layers: the user, the service, and the data. Two main parts of the software design are needed: the client-side and the server-side. The client-side provides an interactive graphical user interface (GUI) within which visitors can provide the necessary scan parameters. The server-side hosts all the data and web service functions. After a user input is specified, the organ dose and effective dose can be calculated and tabulated instantly on the data grid panel embedded on the client-side GUI.

Figure 4.

Figure 4. Schematic diagram of the SOA architecture for VirtualDose which includes 3 interface layers: the user, the service, and the data. Web-based interface uses javascript object notation (JSON) to send user's request and interpret the response messages from the remote server.

Standard image High-resolution image

Using the SaaS platform, VirtualDose consists of many functional modules that were developed using several programming languages or technologies, including C-sharp (C#), JavaScript, hypertext markup language (HTML) specification, and cascading style sheets (CSS). For the client-side scripting, the HTML specification was used to define the content elements used in the web page; CSS was used to control the appearance and formatting of marked-up contents when presented to the front-end user. JavaScript was used to manipulate the contents of HTML content elements and receive/respond to the user interaction, which can be embedded directly into the HTML web page. For the server-side scripting, C# was used as the primary programming language and all the service-side codes are implemented in C#. Each data model or object on the server-side was mapped directly to an individual HTML tag on the client-side and the entire web page was treated as a tree of HTML document object model (DOM) objects. JavaScript and HTML DOM are both used in the programming to better monitor and handle client-side events, such as when a user clicks a control, changes the value in an input control, moves the mouse over or away from a control button. As all the programming codes are stored and executed on a remote website host server but are invoked by users in the client side, the active server pages (ASP).NET model-view-controller (MVC) pattern was selected as the main development framework (Microsoft 2012b).

Another benefit of adopting the ASP.NET MVC is its high security for data protection. When programming under the ASP.NET MVC framework, the client-side code is prevented from directly reading a file or fetching data from a database hosted on the remote host server. Instead, when executing a web service hosted on the remote server-side, the client-side code sends a request message over the hypertext transfer protocol (HTTP) connection to the server. The request is a unique uniform resource locator (URL) for a web service hosted on the remote server-side which was essentially the endpoint of the user's HTTP request connection. This endpoint contains all information about the service function (Pathak 2011). A javascript object notation (JSON) request-response interaction pattern was used as a shortcut method for obtaining data from the remote server asynchronously (Richardson and Ruby 2008, Richardson et al 2013). JSON provides a compact way to either serialize or de-serialize the data from a remote host server to the client-side web page. In this way, the requested data can be loaded quickly from the remote server-side asynchronously to the client-side web page and rendered within the same browser without a visible page refresh.

In VirtualDose, all the detailed distributions of radiation doses to different organ/tissues are derived from a large MC-simulated organ dose database. To efficiently handle this database into VirtualDose, an 'entity framework' (EF) technology in the .NET development environment (Microsoft 2012a) was used to create various types of entity data models (e.g. patient phantoms, dose for each different beam thickness).

In this study, Microsoft Visual Studio 2010 Professional was used as the development tool for the designs of client- and server-side interfaces and functions developments. Microsoft SQL Server 2008 was used to process and compile the organ dose data.

2.6. Testing of virtualdose for routine CT scans

A testing of the CT radiation dose reporting functions in VirtualDose was also performed. Organ dose and effective dose reports were generated for four routine CT scan protocols of the head, chest, abdomen–pelvis (AP), and chest–abdomen–pelvis (CAP). The technical parameter settings for each of these sample scans were 120 kVp tube voltage, 100 mAs integrated tube current, head (for head scan) and body (for chest, AP, and CAP scans) bowtie filters, 10 mm collimation, and a pitch of 1. The scan ranges of these protocols were obtained from AAPM CT scan protocols collections (www.aapm.org/pubs/CTProtocols) and summarized in table 2.

Table 2. Definition of anatomical boundaries using AAPM CT protocols.

Name of the CT protocol Anatomic coverage
Head From top of C1 lamina through top of calvarium
Chest From top of lungs through the bottom of lungs
Abdomen–Pelvis From top of liver to the pubic symphysis
Chest–Abdomen–Pelvis From top of lungs to the pubic symphysis

2.7. Comparison of organ dose data in virtualdose with other existing data

CT scans use low x-ray energies and the resultant CT scan doses are sensitive to small anatomical details in the phantom. CT dose software packages such as the ImPACT CT patient dosimetry calculator (here inafter referred to as 'ImPACT') and CT-Expo (Stamm and Nagel 2002) are based on stylized patient phantoms with overly simplified anatomical information. A previous study has found that these stylized models could present significant dose discrepancies, particularly for low-energy x-rays, when compared against anatomically realistic patient models (Liu et al 2010). To further demonstrate the CT radiation dose reporting functions in VirtualDose, in this study we extended the comparison with the CT-Expo and ImPACT software by considering pediatric, adult, and pregnant patients.

3. Results and discussion

3.1. Monte Carlo organ dose database

Table 3 summarizes the comprehensive organ dose database that was based on extensive MC simulations of a total of 25 voxel phantoms covering pediatric, pregnant female, adult, and obese patients. Each CT scan of the phantom required a separate MC simulation, leading to more than 60 000 MC simulations to cover these phantoms involving different beam thicknesses and tube voltages in the MCNPX code. Results from each set of MC simulations were processed to generate a datasheet showing organ names and corresponding dose results for the axial continuous slice-by-slice scans. The datasheets were then integrated into a comprehensive organ dose database which was compiled using Microsoft SQL server 2008, as illustrated in figure 5.

Table 3. Summary of patient phantoms and CT scan parameters used in the slice-by-slice MC simulations.

  Bowtie Filter Beam Collimation (mm) kVp CT scanner
Pediatric patient models
Newborn male       SiemensSOMATOMSensation 16
Newborn female      
1 year male Head 24 80,100,120
1 year female 10 80,100,120
5 year male      
5 year female      
10 year male Body 24 80,100,120,140
10 year female 10 80,100,120,140
15 year male      
15 year female      
Pregnant patient models
  Head 20 80,100,120,140 GE LightSpeed Pro 16
3 month pregnant6 month pregnant9 month pregnant 10 80,100,120,140
5 80,100,120,140
  1.25 80,100,120,140
Body 20 80,100,120,140
10 80,100,120,140
5 80,100,120,140
Average adult patient models
  Head 20 80,100,120,140 GE LightSpeed Pro 16
  10 80,100,120,140
  5 80,100,120,140
Average adult maleAverage adult female   1.25 80,100,120,140
Body 20 80,100,120,140
  10 80,100,120,140
  5 80,100,120,140
Obese patient models
Normal body-weight male Head 201051.25 80,100,120,14080,100,120,14080,100,120,14080,100,120,140 GE LightSpeed Pro 16
Normal body-weight female
Over-weight male
Over-weight female
Obese level-I male Body 20105 80,100,120,14080,100,120,14080,100,120,140
Obese level-I female
Obese level-II male
Obese level-II female
Morbidly-Obese male
Morbidly-Obese female
Figure 5.

Figure 5. An example of how the slice-by-slice organ dose datasheets retrieved from MC simulations were incorporated into a comprehensive dose database compiled using Microsoft SQL Server 2008. In each datasheet, each data column was for one of axial continuous slice-by-slice scans that covered from head to toe for each phantom.

Standard image High-resolution image

3.2. VirtualDose

3.2.1. A Platform-independent CT dose reporting SaaS.

VirtualDose was designed as an SaaS for CT dose reporting and it involved a web-based dynamic GUI compatible with numerous operating systems. The SaaS deliver mode can also be accessed from portable or mobile devices to be useful to users away from the office environment.

The main interface consists of parameter selection panel, a patient model/scan range display, and a dose result display. The parameter selection panel provides the options for a user to specify the operating conditions of a particular CT scan. Table 4 summarizes the available input features in VirtualDose. When the patient model is selected from a drop-down list of 25 phantoms, an image of that phantom appears on the patient model/scan range display window, with default coverage of the selected CT protocol superimposed on the phantom. The default scan length/position can be accepted, or the boundaries can be adjusted with click-and-drag controllers. To assist in the selection of the proper scan boundaries, transverse pseudo-CT slices of the anatomy for the scan range are also displayed.

Table 4. Parameter selection features available in VirtualDose.

Features and functions defined in VirtualDose
1. Patient phantom library
2. CT scanner manufacturer and model
3. A list of pre-defined CT scan protocols
4. X-ray tube voltage (kVp)
5. Bowtie filter type
6. Beam collimation
7. 2D whole-body cross-section landmark
8. Scan mAs
9. CTDIw specification, with default value provided
10. Pitch specification
11. Z-over beaming length specification
12. DICOM file reader
13. ICRP organ weighting scheme

Based on the user-specified scan parameters, VirtualDose fetches and calculates the patient-specific organ dose data from the remote server-side database. The results are then displayed as a table and a figure, as illustrated in figure 6. In addition to the manual entry of scan parameters, VirtualDose has the option to read the parameters from the DICOM file header, followed by the similar steps as the above to generate the dose report as above. In cases when some parameters are not available from a DICOM file, some default parameters will be selected automatically and users are also allowed to change them later on the interface.

Figure 6.

Figure 6. The results of organ doses were tabulated and plotted in the web-based GUI.

Standard image High-resolution image

3.2.2. RESTful web service API available for third-party application integration.

As a web-based SaaS, VirtualDose offers the representational state transfer (REST) web service using the JSON (Richardson and Ruby 2008, Richardson et al 2013) data exchange interface. This RESTful feature allows VirtualDose to be integrated seamlessly in a third-party software using an application programming interface (API). To retrieve the CT dosimetric metadata from VirtualDose server, a user can send a HTTP POST request including the necessary input parameters (e.g. patient gender, height, weight, kVp, pitch, mAs, TCM data, etc) to the target RESTful link via Internet connection. The remote server can then fetch asynchronously the parameters from user's request message and then return the requested CT dose results through the JSON data communication. In this way, the client software simply parses the contents of the HTTP response and creates its own dosimetric object before continuing processing.

3.3. CT doses reporting for routine CT scans using virtualDose

To demonstrate the CT dose reporting capabilities for different individual populations (small, average, and large patients), 3 pairs of male and female phantoms (1 year-old, median adult, and morbidly-obese) in the phantom library of VirtualDose were selected for four routine CT scan protocols of the head, chest, AP, and CAP. For each CT examination type, organ dose results for the small (1 year-old), average (adult), and large (morbidly-obese) males and females were reported by VirtualDose in tables 58. As CTDIvol was an effective parameter for the comparison between different MDCT scanners (Turner et al 2010), all the reported CT organ doses in tables 58 were normalized by the corresponding CTDIvol of the CT scanner validated in this study.

Table 5. Normalized organ doses in units of mGy per 100 mAs per CTDIvol reported by VirtualDose for the small, average, and large males and females for head examinations.

Patient size Small patients Average patients Large patients
Patient phantom 1 year-M 1 year-F Adult-M Adult-F Morbidly-obese-M Morbidly-obese-F
Scan length (cm) 11.2 11.2 12.7 12.2 12.7 12.2
Bone Surface 0.34 0.34 0.07 0.13 0.11 0.15
Brain 0.96 0.96 0.83 0.87 0.83 0.85
Breast 0.01 0.01 0.01 0.01 0.01 0
Colon 0 0 0 0 0 0
Esophagus 0.05 0.05 0.01 0.02 0.02 0.02
Gonads 0 0 0 0 0 0
Liver 0.01 0.01 0 0 0 0
Lungs 0.02 0.02 0.01 0.01 0.01 0.01
Red Bone Marrow 0.25 0.26 0.05 0.09 0.08 0.11
Salivary Glands 0.18 0.17 0.15 0.39 0.47 0.48
Skin 0.17 0.18 0.05 0.06 0.04 0.04
Stomach 0.01 0.01 0 0 0 0
Thyroid 0.07 0.07 0.02 0.04 0.03 0.04
Urinary Bladder 0 0 0 0 0 0

Table 6. Normalized organ doses in units of mGy per 100 mAs per CTDIvol reported by VirtualDose for the small, average, and large males and females for chest examinations.

Patient size Small patients Average patients Large patients
Patient phantom 1 year-M 1 year-F Adult-M Adult-F Morbidly-obese-M Morbidly-obese-F
Scan length (cm) 9.9 9.9 26.7 24.8 27 24.8
Bone Surface 0.52 0.52 0.45 0.46 0.38 0.38
Brain 0.04 0.04 0.04 0.04 0.05 0.05
Breast 1.83 1.88 1.16 1.12 0.94 0.8
Colon 0.07 0.08 0.08 0.1 0.05 0.05
Esophagus 1.33 1.34 0.81 0.94 0.47 0.57
Gonads 0.01 0.03 0.02 0.02 0.01 0.01
Liver 0.66 0.67 0.53 0.55 0.29 0.3
Lungs 2.04 2.05 1.32 1.36 0.97 1
Red Bone Marrow 0.53 0.54 0.37 0.37 0.29 0.29
Salivary Glands 0.18 0.18 0.13 0.1 0.09 0.09
Skin 0.36 0.36 0.3 0.31 0.26 0.28
Stomach 0.63 0.63 0.54 0.64 0.4 0.4
Thyroid 0.65 0.66 0.37 0.34 0.28 0.34
Urinary Bladder 0.02 0.02 0.02 0.02 0.01 0.01

Table 7. Normalized organ doses in units of mGy per 100 mAs per CTDIvol reported by VirtualDose for the small, average, and large males and females for abdomen–pelvis examinations.

Patient size Small patients Average patients Large patients
Patient phantom 1 year-M 1 year-F Adult-M Adult-F Morbidly-obese-M Morbidly-obese-F
Scan length (cm) 19.7 19.7 32.7 30.3 32.9 30.2
Bone Surface 0.48 0.49 0.47 0.54 0.25 0.28
Brain 0.01 0.01 0.01 0.02 0.01 0.02
Breast 0.13 0.14 0.08 0.09 0.14 0.19
Colon 2.23 2.25 1.28 1.4 0.57 0.58
Esophagus 0.25 0.25 0.04 0.05 0.03 0.05
Gonads 0.72 2.16 0.2 1.18 0.12 0.34
Liver 1.98 2 0.98 1.06 0.68 0.76
Lungs 0.45 0.46 0.15 0.16 0.14 0.17
Red Bone Marrow 0.52 0.52 0.41 0.47 0.2 0.22
Salivary Glands 0.04 0.04 0.03 0.02 0.02 0.02
Skin 0.73 0.74 0.37 0.38 0.31 0.32
Stomach 1.91 1.93 0.74 0.71 0.36 0.38
Thyroid 0.1 0.1 0.04 0.04 0.03 0.03
Urinary Bladder 2.11 2.14 0.96 1.16 0.37 0.36

Table 8. Normalized organ doses in units of mGy per 100 mAs per CTDIvol reported by VirtualDose for the small, average, and large males and females for chest–abdomen–pelvis examinations.

Patient size Small patients Average patients Large patients
Patient phantom 1 year-M 1 year-F Adult-M Adult-F Morbidly-obese-M Morbidly-obese-F
Scan length (cm) 28.4 28.4 57.6 53.4 57.4 53.4
Bone Surface 0.95 0.95 0.89 0.98 0.61 0.65
Brain 0.04 0.04 0.05 0.06 0.06 0.06
Breast 1.93 1.98 1.22 1.2 1.04 0.96
Colon 2.28 2.31 1.34 1.48 0.61 0.62
Esophagus 1.51 1.53 0.84 0.98 0.49 0.61
Gonads 0.72 2.18 0.21 1.2 0.12 0.35
Liver 2.35 2.37 1.36 1.44 0.89 0.98
Lungs 2.34 2.36 1.44 1.49 1.06 1.12
Red Bone Marrow 1 1.01 0.76 0.82 0.48 0.5
Salivary Glands 0.21 0.2 0.15 0.12 0.11 0.11
Skin 1.05 1.05 0.65 0.67 0.54 0.58
Stomach 2.29 2.32 1.18 1.26 0.69 0.72
Thyroid 0.73 0.74 0.4 0.37 0.3 0.36
Urinary Bladder 2.12 2.15 0.98 1.18 0.38 0.37

For head scans, as shown in table 5, brain and salivary glands received considerably higher radiation doses as compared to other organs. The brain was fully included in the head scan region and the CT radiation doses per tube current to the brain in the small, average, and large patient phantoms were found to be very close. However, the doses to the salivary gland increased slightly with increasing patient size. As the salivary gland was not fully covered within the scan region, these increasing doses mostly resulted from the increasing covered portion of the salivary gland in the CT scans of the larger patients. The organ dose results from chest scans were tabulated in table 6 and it was found that, for the constant scan settings, the CT radiation dose per tube current to most organs in the small patient phantoms, particularly to those fully covered in the scan region (e.g. the breast, esophagus, lungs, etc) were significantly higher than those in the average and large patient phantoms. For example, a 54–110% decrease in the lung dose in the male phantoms and a 50–105% decrease in the lung dose in the female phantoms were observed with increasing patient size. As for abdomen–pelvis scans in table 7, these doses were found to decrease even more significantly as a function of size. For the colon, the results showed a decrease of 74–291% in the male phantoms and 61–288% in the female phantoms. For the stomach, these dose differences were even larger, ranging from 158–431% in the male phantoms and 171–408% in the female phantoms, mostly due to the fact that the stomach was highly shielded by adipose tissues abundant in the large patient phantoms. It was also interesting to observe that the doses to the ovaries in the 1 year-old female phantom were notably higher (up to six-fold) than observed for the larger phantoms. This was most likely due to body size and shape of this organ in the 1 year-old female phantom. Similar trends of dose per tube current versus patient size were found in most organs (e.g. breast, colon, liver, lung, stomach, gonadal, and so on) for chest–abdomen–pelvis scan, as shown in table 8. In summary, results showed that under the constant CT scan settings, smaller patient phantoms received considerably higher radiation dose, particularly for those organs within the CT scan region. This test illustrates how VirtualDose was used to evaluate patient organ doses for a given CT protocol.

3.4. Comparison of organ dose data in virtualdose with CT-Expo and ImPACT

3.4.1. Adults and children patients.

A series of CT scans with the same CT scan parameters—120 kVp, 100 mAs, head (children) and body (adult) scan mode, 10 mm collimation and a pitch of 1—were performed using VirtualDose, CT-Expo (v2.3), and ImPACT (v1.0). In VirtualDose, 6 pairs of male and female phantoms (New-born, 1 year-old, 5 year-old, 10 year-old, normal weight and morbidly-obese) and 1 pregnant female phantom (3 month) were selected to generate a comprehensive set of organ dose data for the comparison. In CT-Expo, 3 pairs of male and female phantoms (baby, child, and adult) were used to generate the organ dose data for pediatric and adult patients. In ImPACT, the only available hermaphroditic stylized phantom was used. As previously illustrated by Lee et al (2012), it would be very difficult to define scan ranges in the pediatric phantoms of CT-Expo using anatomical landmarks. Therefore, to eliminate the errors caused by the definition of scan ranges, a series of whole-body scans were successively performed using CT-Expo, ImPACT, and VirtualDose on the selected phantoms. All the reported CT organ doses were normalized by their corresponding CTDIvol values of the CT scanner used in this study. Since only one hermaphroditic stylized phantom is available in ImPACT, for the purpose of comparison, the organ doses to adult phantoms reported by CT-Expo and VirtualDose were all averaged to obtain the sex-averaged values (except for the case of the pregnant female phantoms, as there is no corresponding male phantoms). Those sex-averaged organ doses from VirtualDose were then normalized to those reported by the ImPACT and CT-Expo.

Figures 7 and 8 summarized the comparative organ dose values for a whole-body scan involving each pair of pediatric (New-born, 1 year-old, 5 year-old, and 10 year-old) and adult (normal weight and morbidly obese) phantoms from VirtualDose with the CT-Expo (baby, child, and adult) and ImPACT (adult) stylized phantoms. Figures 7(a) and (b) showed that most organ doses from CT-Expo in the whole body scan for the baby and child phantoms agreed with the values for the pediatric phantoms from VirtualDose within 34%. However, significant discrepancies were observed between the dose values for bone surface and breast from CT-Expo and VirtualDose pediatric phantoms. For example, in figures 7(a) and (b), doses for the bone surface in the baby and child phantoms in CT-Expo were 3-fold greater than those for the pediatric phantoms in VirtualDose. Figure 7(a) also showed that the breasts of the baby and child male phantoms receives 100% less dose than those for the pediatric male phantoms in VirtualDose, which was the reason why CT-Expo did not provide breast dose values for the male pediatric phantoms.

Figure 7.

Figure 7. Plots of the organ dose differences reported by CT-Expo and VirtualDose for the whole-body CT scan on the pediatric (a) male and (b) female phantoms. The organ dose results for 'Baby' and 'Child' phantoms reported by CT-Expo were compared with those for newborn, 1 year-old, 5 year-old, and 10 year-old phantoms reported by VirtualDose by using the formula of (DoseCT-Expo  −  DoseVirtualDose)/ DoseVirtualDose * 100%.

Standard image High-resolution image
Figure 8.

Figure 8. Plot of the sex-averaged organ dose differences reported by CT-Expo, ImPACT, and VirtualDose for the whole-body scan on the stylized and voxel-based adult normal weight (NW) and morbidly obese (MO) phantoms by using the formula of (DoseCT-Expo/ImPACT  −  DoseVirtualDose)/ DoseVirtualDose * 100%.

Standard image High-resolution image

Figure 8 showed results for adult phantoms from CT-Expo, ImPACT, and VirtualDose. When compared to those of the stylized phantom in CT-Expo and ImPACT, most organ doses from VirtualDose were found to differ by  −45 to 58% for the normal weight adult phantom. The differences were found to be significantly in the morbidly obese phantom for the same scan settings. For example, VirtualDose reported a reduction of 91 and 96% in the colon dose, 104 and 105% in the stomach dose, and 128 and 145% in the urinary bladder dose, when compared with those reported by CT-Expo and ImPACT, due to the shielding effect of the extra adipose tissues in morbidly obese phantoms used in the VirtualDose.

As three separate series of whole-body scans were independently performed on the patient phantoms in CT-Expo, ImPACT, and VirtualDose using the same CT scan settings, organs in both the stylized phantom and voxel-based phantoms were entirely covered in the scan region. Thus, these dose discrepancies can be attributed mostly to the anatomical variations. These results confirm those were reported previously by Liu et al (2010) and Lee et al (2011, 2012) who concluded that the lack of realism offered by stylized phantoms caused significant discrepancies in reported CT dose results.

3.4.2. Pregnant patients.

For pregnant patients, the primary focus of the comparison for pregnant patients was on CT radiation dose to fetus which was either partially or fully covered in the scan region. As no fetal organs were available in the ImPACT software, ImPACT used the uterus as a surrogate to estimate the fetal dose, as is done in similar situations (Angel et al 2008, Gu et al 2013). In VirtualDose, the CT radiation doses to the fetal brain, fetal skeleton, fetal soft tissues, and fetus total can be reported. The CT dose results from 2 routine CT scan protocols of the chest, and abdomen–pelvis from ImPACT and VirtualDose were tabulated in tables 9 and 10, respectively. As shown in table 9 for abdomen–pelvis scans, when the fetus is fully covered (for medical reasons) in the scan range, the CT radiation dose to the fetus reported by ImPACT was found to be 15% larger than that reported by VirtualDose for the voxel-based pregnant female at three-month gestational stage. However, for chest CT scans when the fetus was only partially covered, the results in table 10 show that the dose to the fetus reported by ImPACT can be even smaller (~15 times) compared to the value reported by VirtualDose. These dose differences were mostly due to the fact that the stylized phantom in ImPACT does not contain a representation of the fetus, using only a surrogate organ. The size and position of the uterus in the stylized phantom introduce significant overestimating or underestimating of the fetal radiation dose. Furthermore, the CT radiation doses to other fetal organs (e.g. brain, skeleton, and soft tissues) were not available in ImPACT. The developing fetus is very sensitive to radiation and the risk of developing leukemia varies with the gestational age (Chatterson et al 2011). For this reason, the fetal dose information provided by VirtualDose can be helpful to the physicians and pregnant patients.

Table 9. Normalized fetal organ (mGy per 100 mAs per CTDIvol) reported by ImPACT and VirtualDose for pregnant patients from abdomen–pelvis examinations.

Organs ImPACT VirtualDose-P3
Fetal_Brain NAa 1.09
Fetal_Skeleton NAa NAb
Fetal_Soft_Tissue NAa 1.14
Fetus_Total 1.3 1.13

aImPACT does not report fetal organ dose. bVirtualDose-P3 does not contain fetal skeleton.

Table 10. Normalized fetal organ (mGy per 100 mAs per CTDIvol) reported by ImPACT and VirtualDose for pregnant patients from chest examinations.

Organs ImPACT VirtualDose-P3
Fetal_Brain NAa 0.02
Fetal_Skeleton NAa NAb
Fetal_Soft_Tissue NAa 0.03
Fetus_Total 0.002 0.03

aImPACT does not report fetal dose. bVirtualDose-P3 does not contain fetal skeleton.

In summary, for relatively low x-ray energies, CT doses depend on organ shape, size and position. Therefore, VirtualDose, which is based on the latest patient phantoms, can be used to significantly improve data for organ doses to patients undergoing CT examinations.

3.5. Limitations of this work

Although the VirtualDose software represents a significant improvement on realism and accuracy of patient modeling and Monte Carlo dose calculations, there are several remaining technological limitations that may introduce uncertainty in the calculation organ dose for a specific patient undergoing CT scans. First, the MC dose calculations performed were based on a limited number of validated CT source term models, namely, the GE LightSpeed Pro 16 and Siemens SOMATOM Sensation 16. Applications of the correction factor techniques to extend to other scanners, while yielded reasonably good agreements with those reported in the literature, may introduce an uncertainty in the estimated organ doses (Turner et al 2010, Li et al 2011, Lee et al 2012, Sahbaee et al 2014). Second, the software does not currently support tube current modulation employed by a modern CT scanner, but an interface is being developed that will address the limitation in the near future. When projecting through different body part of the patient, the tube current value can vary in the xy axis (angular modulation), z-axis (longitudinal modulation), or both. However, the angle-specific tube current information is not currently captured in the scan record, and only the slice-averaged tube current data are available in the DICOM file. When such slice-averaged information is used to compute organ dose, some uncertainties can be introduced, particularly for organs with highly asymmetric placement in the body, although the effect was found to be small in most cases (Khatonabadi et al 2012). Third, the contiguous axial scans with a specific pitch value were used to approximate the radiation dose for a helical scan covering the same scan length. This axial scan approximation method may introduce an uncertainty due to the surface dose variation effect when a low pitch value is used (Zhang et al 2009a). An ideal solution is to simulate real-time helical scans in a Monte Carlo code instead of using the pre-calculated axial scan data. A fast Monte Carlo code for CT dose calculation is being developed to address this issue (Xu et al 2014). Finally, the software assumes the patient's arms are in the overhead position for scans of the body which may introduce an uncertainty in usual CT scans where the arms are inside the scan range (such as for patients in the emergency situations that are sedated, or that are unable to hold their arms above the head). The effects of arm positioning have been discussed else (Liu et al 2015).

4. Conclusions

Based on an extensive and latest library of 25 patient phantoms of both genders and various ages, VirtualDose has been shown in this study to be fully functional in reporting organ doses for a variety of patient types. When compared against the CT-Expo and ImPACT software that are based on anatomically simplified models, VirtualDose is found to be more accurate owing to anatomically realistic geometries. These results confirm those were reported previously by Liu et al (2010) and Lee et al (2011, 2012) who concluded that the lack of realism offered by stylized phantoms caused significant discrepancies in CT dose estimations. The development of VirtualDose as an SaaS platform allows multiple users to access the software simultaneously via Internet without having to install the software locally. The web-based GUI design and reporting features of VirtualDose are designed to cover a large list of CT manufacturers and scanner types and current ICRP recommendations. The SaaS framework and object-oriented programming methods can provide the necessary flexibility to generate accurate dose estimates for arbitrary scan protocols defined by a user. Furthermore, with innovative software engineering features such as the RESTful web service API, VirtualDose permits seamless integration with a third-party picture archiving and communication system (PACS) software package. Effort is on-going to continue to improve both the accuracy and usability in reporting CT doses for more than 50 current users worldwide. More information about VirtualDose is available from www.virtual-dose.com.

Acknowledgments

The development of VirtualDose (www.virtul-dose.com) as a commercial software was supported by an STTR grant to Virtual Phantoms, Inc. (www.virtualphantoms.com) from the national institute of biomedical imaging and bioengineering (NIBIB) (1R42EB010404). Patient phantoms used in VirtualDose were developed previously by grants to Rensselaer Polytechnic Institute from the national cancer institute (NCI) (R01CA116743) and the national library of medicine (NLM) (R01LM009362; R01LM009362-03S1) and a grant to the University of Florida from NIBIB (R01EB00267).

Please wait… references are loading.