The study was conducted under the auspices of the Transdisciplinary Research In Cancer of the Lung (TRICL) Research Team, which is a part of the Genetic Associations and MEchanisms in ONcology (GAME-ON) consortium, and associated with the International Lung Cancer Consortium (ILCCO). Ethics: All participants provided written informed consent. All studies were reviewed and approved by institutional ethics review committees at the involved institutions. Sequencing data are derived from four substudies. The substudies that contributed include Harvard, Liverpool, Toronto, and IARC. The Alpha-Tocopherol, Beta-Carotene Cancer Prevention (ATBC) Study is a randomized primary prevention trial including 29,133 male smokers enrolled in Finland between 1985 and 1993. Participants ranged between ages of 50 to 69 at enrollment and were randomized in a factorial design to take either 50 milligrams of d-alpha tocopheryl acetate (Vitamin E), 20 mg of all-trans-beta-carotene, both or placebo. The study continued to monitor cancer incidence through 2012 and total mortality through December 2013. The CAncer de PUlmon en Asturias Study (CAPUA) is a hospital-based case-control study conducted in Asturias, Spain by the University of Oviedo. Lung cancer cases were recruited in three main hospitals of Asturias, following an identical protocol from 2002 to 2012. Eligible cases were incident cases of histologically confirmed lung cancer between 30 and 85 years of age and residents in the geographical area of each participating hospital. Controls were selected from patients admitted to those hospitals with diagnoses unrelated to the exposures of interest and individually matched by ethnicity, gender, age (± 5 years) and hospital. Epidemiologic data were collected personally through computer-assisted questionnaires by trained interviewers during the first hospital admission. Structured questionnaires collected information on sociodemographic characteristics, recent and prior tobacco use, environmental exposure (air pollution and passive smoking), diet, personal and family history of cancer, and occupational history from each participant. Peripheral blood samples (or mouthwash samples when they refused to donate blood) were collected from all participants. Coding of histology was based on 2001 WHO/IASLC. Genomic DNA was extracted based on standard protocol. The Canadian Screening Study includes the nested case-control samples from 3 screening programs: IELCAP-Toronto: Ever smokers of more than 10 pack-years age 50 and above were eligible for the I-ELCAP screening program since 2003, and a total of 4782 individuals have been enrolled in the Greater Toronto Area. Participants were administered a LDCT scan along with a standard study questionnaire at baseline. Blood samples were systematically collected at baseline since 2006. Participants who had an abnormality in a CT scan were followed up every 1 to 2 years. The screening program was organized by the Princess Margaret Hospital. PanCan: Ever smokers between the ages of 50-75 with no previous history of invasive cancer are eligible to participate in the study. The study was carried out across Canada in Vancouver, Calgary, Hamilton, Toronto, Ottawa, Quebec, Halifax, and St. John's. A total of 2537 smokers have been screened from 2008 to 2011. All study participants completed a detailed questionnaire, spirometry, collection of blood specimens for biomarker measurement and LDCT at baseline. All participants are followed for a minimum of 3 years. On yearly follow up, an updated shorter questionnaire is administered, blood is collected and CT scans are performed. Blood samples are available from all 2537 individuals. BCCA Screening Program: From 1990 to 2007, 4274 smokers above 40 years old who had smoked 20 pack-years or more were enrolled at BCCA. Upon enrollment, subjects completed a questionnaire for their lifestyle and medical history. Baseline spirometry was conducted using a flow-sensitive spirometer in accordance with the American Thoracic Society recommendations. Since 2000, a LDCT was obtained in 2440 individuals. The participants were followed prospectively to determine whether they developed lung cancer. A total of 9759 individuals participated in the CT screening program in Canada from these 3 programs. The samples included in this project is based on a subset of nested lung cancer case-control pairs based on 1:2 ratio. The Carotene and Retinol Efficacy Trial (CARET) was a randomized, double-blind, placebo-controlled trial of the cancer prevention efficacy and safety of a daily combination of 30 mg of beta-carotene and 25,000 IU of retinyl palmitate in 18,314 persons at high risk for lung cancer. CARET began in 1985, and the intervention was halted in January 1996, 21 months ahead of schedule, with the twin conclusions for definitive evidence of no benefit and substantial evidence of a harmful effect of the intervention on both lung cancer incidence and total mortality. CARET continued to follow and collect endpoints on their participants through 2005. Pathology reports and medical records were reviewed to confirm cancer endpoints, and death certificates obtained to capture cause of death. During the active intervention phase of CARET, serum, plasma, whole blood, and lung tissue specimens were collected on participants. These biospecimens make up the CARET Biorepository. For the OncoArray Project, CARET provided DNA extracted from whole blood of lung cancer cases and controls matched on age at baseline (± 4 years), sex, race, baseline smoking status, history of occupational asbestos exposure (asbestos vs heavy smoker), and year of enrollment (2-year intervals). The European Prospective Investigation into Cancer and Nutrition (EPIC) study is a multi-center cohort study involving 521,000 study participants from 10 European countries. The current study involved EPIC participants from 7 countries (Greece, Netherlands, UK, France, Germany, Spain, and Italy), including 1223 incident lung cancer cases and 1249 smoking matched controls. The Kentucky Lung Cancer Research Initiative is a study conducted by the Markey Cancer Center Cancer Center and the University of Kentucky using a population-based, case-control framework to study the extraordinarily high rates of lung cancer in Southeastern, Appalachian Kentucky. Cancer cases were recruited from the Kentucky Cancer Registry at the time of diagnosis and controls were recruited from a random digit dialing process from the same region. Study accrual began in January 5, 2012 and completed on September 5, 2014 and 520 subjects were recruited in a 4:1 ratio of controls: cases from Appalachian Kentucky. Of the 520 subjects recruited, 231 are included in the OncoArray analysis, including all 93 cancer cases, and 123 controls. Newly diagnosed lung cancer cases and controls underwent blood, toenail (for trace element analysis), urine, buffy coat, water, soil, and radon collection, residence GPS mapping, as well as an extensive epidemiologic, occupational, and health history questionnaire (Clinical Trials.gov Identifier: NCT01648166). The Harvard Lung Cancer Study (HLCS) is a case-control study based at Mass General Hospital (MGH) in Boston, Massachusetts from 1992 to 2004. Details of the study were described previously. Briefly, eligible cases included any person over the age of 18 years with a diagnosis of primary lung cancer that was further confirmed by an MGH lung pathologist. Controls were recruited from the friends or spouses of cancer patients or the friends or spouses of other surgery patients in the same hospital. Potential controls were excluded from participation if they had a diagnosis of any cancer (other than non-melanoma skin cancer). Interviewer-administered questionnaires, a modified version of the standardized American Thoracic Society respiratory questionnaire, collected information on demographics, medical history, family history of cancer, smoking history, and a detailed work history, including job titles and tasks. Genome-wide genotype data were first generated using Illumina Human 610-Quad BeadChips and then imputed by MACH against the 1000 Genome Project dataset (http://browser.1000genomes.org/index.html). The Institutional Review Board of MGH and the Human Subjects Committee of the Harvard School of Public Health approved the study. The Israel study (NICCC-LCA) is an ongoing case-control study of newly diagnosed lung cancer cases of any histology and population age/sex/ethnicity-matched "healthy" controls. All participants undergo face-to-face interviews, provide a venous blood sample (separated into DNA, Sera, lymphocytes) after signing an IRB-approved form. Histology reports, FFPE blocks and clinical follow-up are available for most cancer cases. The MD Anderson Cancer Center (MDACC) Study. Lung cancer cases and frequency-matched controls were ascertained from a large ongoing case-control study at the University of Texas MD Anderson Cancer Center (UTMDACC) since 1991. Detailed study description was provided previously (Spitz et al 2007). In brief, cases were newly-diagnosed and histologically confirmed lung cancer patients recruited from UTMDACC. Controls were healthy individuals without a history of cancer (except for nonmelanoma skin cancer) and recruited from the Kelsey-Seybold Clinics, the largest private multispecialty physician group in the Houston metropolitan area. Controls were frequency-matched to cases on age (±5 years), sex, and race/ethnicity. After providing written informed consent, each study participants completed an in-person interview by staff interviewers to collect information on demographics, smoking status, etc. Blood samples were also drawn from all the study participants. This study was approved by institutional review boards of UTMDACC and Kelsey-Seybold Clinics. The Malmö Diet and Cancer Study (MDCS) is a population-based prospective cohort study that recruited men and women aged at 44 to 74 years old of living in Malmö, Sweden between 1991 and 1996. The main goal of the MDCS is to study the impact of diet on cancer incidence and mortality. It consists of a baseline examination including dietary assessment, a self-administered questionnaire, anthropometric measurements and collection of blood samples. A total of 165 incident lung cancer cases and 174 individually smoking-matched controls were available for this analysis. The Multiethnic Cohort (MEC) Study includes 215,251 men and women aged 45-74 years at recruitment, primarily from five ethnic/racial groups - African Americans and Latinos mostly recruited from CA (mainly from Los Angeles County) and Japanese Americans, Native Hawaiians and whites (mostly recruited from HI). The cohort was assembled in 1993-1996 by mailing a self-administered questionnaire to persons identified primarily through driver's license files. The baseline questionnaire obtained information on demographics, anthropometry, smoking history, medical and reproductive histories, family history of cancer, diet and physical activity. Incident cancer cases are identified by regular linkage with the State of California Cancer Registry and the Hawaii Tumor Registry, both members of the SEER Program of the NCI. In 2001-2006, a prospective biorepository was assembled by collecting a pre-diagnostic blood specimen from 67,594 surviving MEC members. At the time of blood collection a short questionnaire was administered that included information on smoking during the previous 15 days. For this study, cases were all lung cancer cases incident to blood draw and diagnosed before December 2012. For each case, a control was selected among unaffected MEC participants who were alive at time of the case's diagnosis and matched on study site, sex, race/ethnicity, age (age at diagnosis for cases; age at blood collection for controls), and date of blood collection. The Mount-Sinai Hospital-Princess Margaret Study (MSH-PMH) was conducted in the greater Toronto area from 2008 to 2013. Lung cancer cases were recruited at the hospitals in the network of the University of Toronto. Controls were selected randomly from individuals registered in the family medicine clinics databases and were frequency matched with cases on age and sex. All subjects were interviewed, and information on lifestyle risk factors, occupational history and medical and family history was collected using a standard questionnaire. Tumors were centrally reviewed by the reference pathologist, a member of the International Association for the Study of Lung Cancer (IASLC) committee, and a second pathologist in the University Health Network. If the reviews conflicted, a consensus was arrived at after discussion. Coding of histology was based on 2001 WHO/IASLC. Genomic DNA was extracted based on standard protocol. The New England Lung Cancer Study (NELCS) is a population-based case-control study of lung cancer among residents of Northern and Central New Hampshire counties and the bordering region of Vermont. Cases with histologically confirmed primary incident lung cancer were identified from 2005 to 2007 using the New Hampshire State Cancer Registry and the Dartmouth-Hitchcock Medical Center (DHMC) Tumor Registry. Control participants were identified using a commercial database and matched to lung cancer cases within 5-year age groups, sex and county. Genomic DNA was isolated from blood or buccal specimens provided by consenting participants. The study complied with requirements of the Dartmouth College's Committee for Protection of Human Subjects. The Nijmegen Lung Cancer Study. The Netherlands patients with lung cancer were identified through the population-based cancer registry of the Netherlands Comprehensive Cancer Organisation in Nijmegen, the Netherlands. Patients who were diagnosed in one of three hospitals (Radboud University Medical Center, Canisius Wilhelmina Hospital in Nijmegen, and Rijnstate Hospital in Arnhem) since 1989 and who were still alive at April 15th, 2008 were recruited for a study on gene-environment interactions in lung cancer. 458 patients gave informed consent and donated a blood sample. This case series was expanded with 94 patients to a total of 552 by linking three other studies to the population-based cancer registry in order to identify new occurrences of lung cancer among the participants of these other studies. All three other studies (i.e., POLYGENE, the Nijmegen Biomedical Study, and the Radboudumc Urology Outpatient Clinic Epidemiology Study) were initiated to study genetic risk factors for disease and participants to these studies gave general informed consent for DNA-related research and linkage with disease registries. Information on histology, stage of disease, and age at diagnoses was obtained through the cancer registry. Lifestyle information was collected through a structured questionnaire and whole blood for DNA isolation was collected by the regional thrombosis services. The cancer-free controls (46% males) were selected from participants of the "Nijmegen Biomedical Study" (NBS), an age- and sex-stratified random sample of the general population of the municipality of Nijmegen, The Netherlands. All participants provided extensive lifestyle information by structured questionnaires and blood samples for DNA isolation, serum and plasma. All controls are of self-reported European descent. The study protocols of the NBS were approved by the Institutional Review Board of the Radboudumc and all study subjects signed a written informed consent form. The Northern Sweden Health and Disease Study (NSHDS) encompasses several prospective cohorts. The current study involves participants from the Västerbotten Intervention Project (VIP), a sub-cohort within NSHDS. VIP is an ongoing prospective cohort and intervention study intended for health promotion of the general population of the Västerbotten County in northern Sweden. VIP was initiated in 1985 and all residents in the Västerbotten County were invited to participate by attending a health check-up at 40, 50 and 60 years of age. Participants were asked to complete a self-administered questionnaire including various demographic factors such as education, smoking habits, physical activity and diet. In addition, height and weight were measured and participants were asked to donate a fasting blood sample for future research. A total of 243 incident lung cancer cases and 266 individually smoking-matched controls were available for this analysis. Norway National Institute of Occupational Health Study. Early-stage NSCLC cases and healthy controls at the time of enrollment were Caucasians of Norwegian origin and were recruited from the same geographical region (Western Norway). The patients were enrolled in the study, whenever practically feasible among patients admitted for lung cancer at the Haukeland University Hospital in Bergen, Norway. The informed written consents covering analysis of molecular and genetic markers was signed by the patients prior to surgery. Only patients with histologically confirmed early-stage NSCLC were included in our study. The subjects included in this project are a subgroup recruited into the project "lung cancer genetics" at NIOH. The controls were recruited from the same geographical region of Western Norway and frequency-matched with cases on cumulative smoking dose (pack-years). Pack-years smoked [( 20 cigarettes per day) x years smoked] were calculated to indicate the cumulative smoking dose. The Cases and controls were interviewed using similar questionnaires and were categorized as never smokers, ex-smokers or current smokers. Never smokers are subjects indicating having smoked less than 100 cigarettes in their life time. Ex-smokers were defined as those having quitted at least 1 year before sampling, and current smokers were those indicating that they were smokers at the time of sampling. The project has been approved by the Regional Committee for Medical and Health Research Ethics in Southern Norway in accordance with the WMA Declaration of Helsinki. The ethical approval covered access to the NSCLC databank. The Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial (PLCO) Study, a randomized trial aimed at evaluating the efficacy of screening in reducing cancer mortality, recruited approximately 155,000 men and women age 55 to 74 years from 1992 to 20014. Screening for lung cancer among participants in the intervention arm included a chest x-ray at baseline followed by either three annual x-rays (for current or former smokers at enrollment) or two annual x-rays (for never smokers); participants in the control arm received routine health care. Screening-arm participants provided data on sociodemographic factors, smoking behavior, anthropometric characteristics, medical history, and family history of cancer, as well as blood samples annually for the first 6 years of the study (baseline T0 and T1 through T5). Lung cancers were ascertained through annual questionnaires mailed to the participants, and positive reports were followed up by abstracting medical records or death certificates. Follow-up in the trial as of July 2009 was 96.7%. Patients were excluded because of missing baseline questionnaire, previous history of any cancer, diagnosis of multiple cancers during follow-up, missing smoking information at baseline, missing consent for utilization of biologic specimens for etiologic studies, or unavailability/insufficient quantity of serum or DNA specimens. The Resource for the Study of Lung Cancer Epidemiology in North Trent (ReSoLuCENT) is an ongoing study conducted in Sheffield from 2006 and due to complete recruitment in 2016. The study recruited pathologically confirmed lung cancer cases diagnosed at age 60 years or younger and family matched controls. Lung cancer cases diagnosed at ages older than 60 years were recruited if they reported a family history of lung cancer. The cases and matched controls were recruited through several major cancer treatment centers, however, the majority were recruited in North Trent. All participants completed a detailed lifestyle questionnaire which included questions about occupational exposures, education, medical history and family history of cancer and lung disease. Participants also donated blood samples for DNA extraction. The ReSoLuCENT study has been funded by the Sheffield Hospitals Charity, Sheffield ECMC and Weston Park Hospital Cancer Charity. First degree relatives were removed from the sample deposited to dbGaP. The Roy Castle Lung Study of Liverpool Lung Project (LLP) is a case-control and cohort study which has recruited over 11,500 individuals since 1996 from the Liverpool region in the UK. Detailed epidemiological and clinical data is collected with associated specimens (i.e. tumor tissue, blood, plasma, sputum, bronchial lavage and oral brushings). The participants have completed a detailed lifestyle questionnaire at recruitment, with repeat questionnaires at intervals; updated data on clinical outcome and hospital events are collected through the Health and Social Care Information Center (including Office of National Statistics mortality data, Cancer Registry and Health Episode Statistics). The project is registered on the UK National Institute for Health Research (NIHR) lung cancer portfolio and has all the required ethical approvals and sponsorship arrangements in place. The lung tumors were reviewed by the reference pathologist. The Seoul Bundang Lung Cancer Study was conducted between 2005 and 2010 to discover genetic and environmental factors related with lung cancer development. Lung cancer cases were recruited at the Seoul National University Hospital in Bundang. Controls were selected randomly from individuals participated in health check-up program and were frequency matched with cases on age and sex. All subjects were interviewed, and information on lifestyle risk factors, occupational history and medical and family history was collected using a standard questionnaire. Tumors were reviewed by the pathologists in the hospital. If the reviews conflicted, a consensus was arrived at after discussion. Coding of histology was based on 2001 WHO/IASLC. Genomic DNA was extracted based on standard protocol. The Shanghai Cohort Study (SCS) consisted of 18,244 men in Shanghai, China, who were 45-64 years old at the time of enrollment during 1986-1989. Approximately 80% of eligible men participated in the study. At the time of recruitment, each cohort subject was interviewed in-person by a trained nurse interviewer using a structured questionnaire that included background information, history of tobacco and alcohol use, current diet, and medical history. At the completion of the interview, the nurse collected a 10 ml blood and a single void urine specimen from the study participant. The buccal cell samples were collected from all surviving cohort members (~15,000) in the 2001-2002 follow-up interviews. The cohort has been followed for the occurrence of cancer and death through routine ascertainment of new cases from the population-based Shanghai Cancer Registry and Shanghai Vital Statistics Units. To maximize the cancer findings and minimize the loss of follow-up, we contacted each surviving cohort member annually. Retired nurses visit the last known address of each living cohort member and record details of the interim health history of the cohort member. As of December 31, 2014, cumulatively 612 (3.4%) original subjects were lost to follow-up, and 574 (3.1%) refused to our continued follow-up interview. A nested case-control study of incident lung cancer cases within the Shanghai Cohort Study was used to examine the association between serum levels of vitamin B6 and other compounds in the one-carbon metabolism pathway and risk of lung cancer. Briefly, 516 lung cancer cases were identified among cohort participants with available serum samples as of 12/31/2006. For each case, we randomly selected one control subject from all cohort members who were free of cancer and alive at the time of cancer diagnosis of the index case. Controls were matched to the index case by age at enrollment (±2 years), date of biospecimen collection (±1 month) and neighborhood of residence at recruitment, and smoking status (current, former and never smokers) as established previously for other studies. For former smokers, cases and controls were further matched by years since quitting smoking (<10 vs ≥10 years). One serum vial per subject was retrieved from biorepository and all serum samples were sent to the laboratory (B-vital) for measurements. DNA samples of 250 lung cancer cases and 250 matched controls were available for the present study. The Singapore Chinese Health Study (SCHS) cohort consisted of 63,257 Chinese men and women in Singapore when they were 45-74 years old at the time of enrollment between April 1993 and December 1998. At recruitment, each study subject was interviewed in person by a trained interviewer using a structured questionnaire that emphasized current diet assessed via a validated, 165-item food frequency questionnaire. The questionnaire also requested information on demographics, lifetime use of tobacco, incense use, current physical activity, usual sleep duration, reproductive history (women only), occupational exposure, medical history, and family history of cancer. Blood or buccal cell, and spot urine samples were collected first from a random 3% sample of cohort participants in April 1994, and extended to all surviving cohort participants starting in January 2000. Overall approximately 60% of eligible cohort participants donated biospecimens. The cohort has been passively followed for death and cancer occurrence through regular record linkage with the population-based Singapore Cancer Registry and the Singapore Registry of Births and Deaths. Migration out of Singapore, especially among housing estate residents, was negligible. As of latest update, only 55 individuals from this cohort were known to be lost to follow-up due to migration and other reason. A nested case-control study of incident lung cancer cases within the Singapore Chinese Health Study was used to examine the association between serum levels of vitamin B6 and other compounds in the one-carbon metabolism pathway and risk of lung cancer. As of 12/31/2011, 422 lung cancer cases were identified among cohort participants with available prediagnostic plasma samples. For each case, one control subject was randomly selected from all eligible cohort members who were alive and free of cancer on the date of cancer diagnosis of the index case. The control subject was individually matched to the index case by gender, dialect group (Hokkien, Cantonese), age at enrollment (±3 years), date of baseline interview (±2 year), date of biospecimen collection (±6 months), and smoking status (current, former, and never smokers). For current smokers, cases and controls were further matched by number of cigarettes per day (<15, ≥15 cigarettes/day). For former smokers, cases and controls were further matched by years since quitting smoking (<10, ≥10 years). One plasma aliquot per subject was retrieved from the biorepository and all plasma samples were sent to the laboratory (B-vital) for measurements, and one aliquot of DNA per subject for the present study. The International Agency for Research on Cancer (IARC) L2 Study. Lung cancer cases and controls were recruited through a multicentric case-control study coordinated by the IARC in Russia, Poland, Serbia, Czech Republic, and Romania from 2005 to 2013. Cases were incident cancer patients collected from general hospitals. Controls were recruited from individuals visiting general hospitals and out-patient clinics for disorders unrelated to lung cancer and/or its associated risk factors, or from the general population. Information on lifestyle risk factors, medical and family history was collected from subjects by interview using a standard questionnaire. All study participants provided written informed consent. The current study included 1,133 lung cancer cases and 1,117 controls genotyped on the Oncoarray. The Washington State University Lung Cancer Study is a hospital case-control study of 511 subjects with newly-diagnosed (within 1 year of diagnosis) lung cancer and 820 race-, sex- and age-matched controls. Lung cancer cases were recruited from lung cancer clinics within the H. Lee Moffitt Cancer Center while controls were recruited from the Lifetime Cancer Screening Center, a H. Lee Moffitt Cancer Center affiliate. None of the controls were diagnosed with any form of cancer at the time of screening. Detailed questionnaire data and oral buccal cells were collected for all subjects. The Total Lung Cancer (TLC) Study is a hospital-based study that included 458 lung cancer patients recruited for Moffitt Cancer Center's Total Cancer Care™ protocol between April 2006 and August 2010. Total Cancer Care™ is a multi-institutional observational study of cancer patients that prospectively collects self-reported demographic and clinical data, medical record information and blood samples for research purposes. All patients used in this cohort were recruited from the Thoracic Oncology Clinic at the Moffitt Cancer Center. The Vanderbilt Lung Cancer Study (BioVU) is a case-control study nested within the Vanderbilt University Medical Center biobank, BioVU. BioVU is a biorepository of DNA extracted from blood drawn from patients seeking routine clinical care at Vanderbilt University Medical Center and linked to de-identified electronic health records for research purposes. Lung cancer cases and controls were identified from BioVU participants in February 2014. Lung cancer cases were identified from the Vanderbilt tumor registry. All specimens undergo pathologic review for determination of morphology. Coding of histology was based on SEER Program Coding Guidelines. Controls were randomly selected from BioVU participants, excluding cancer patients, and were matched to cases on age (± 5 years), sex, and race. Relevant covariates were identified from electronic health records using natural language processing. Genomic DNA was extracted based on a standard protocol.
Programmatic submissions (XML based) For further information please check our Submission FAQs, submission quickguide as well as submission terms! Introduction Besides the Submitter Portal tool, EGA supports programmatic sequence and clinical data metadata submissions. If you are not sure what this means, you may want to explore our brief metadata introduction. Programmatic submissions are recommended for array-based submission. Moreove, it may be of help if your submission is recurrent or it is difficult to manage manually due to its sheer size. Otherwise, we highly recommend using the Submitter Portal to perform submissions. In this page we will guide you through the required steps to programmatically submit data to the EGA. Programmatic submissions require your metadata to be structured for an easy and straightforward validation and archival. It basically consists in formatting your metadata as Extensible markup language (XML) files and submitting them to the EGA using the WEBIN Before submitting metadata to the EGA, it is important to ensure that the information in your XML files is compliant with our standards. You can see further details on how these standards are maintained at EGA at our EGA Schemas documentation page. Using WEBIN, you can validate your XML files against EGA's schemas to ensure that your metadata is compliant before submission. WEBIN services WEBIN production service WEBIN test service We advise you to submit your metadata to the test service when submitting to the production service for the first time. The test service is identical to the production service except that all submissions will be discarded in the following 24 hours. This allows you to learn about the submission process without having to worry about data being submitted. Authentication Authentication is required each time a submission is made. The submission service uses HTTPS protocol for metadata encryption and identification to provide a secure submission environment. Data file upload Both Runs and Analyses reference files (e.g. FASTQ need to be uploaded to the EGA before these metadata objects are submitted. In other words, if you submit a Run that references a file that we cannot find associated with your account, the metadata submission will fail. See further details on how to upload your files in our File Upload documentation. Metadata model of the EGA Our metadata model is formed by multiple metadata objects. Check further details in our documentation at our EGA Schema documentation page. Working with EGA XMLs files Now that the basic concepts of the EGA metadata have been described, you can start preparing your programmatic submission through XML. Here you will find the guidance on how to prepare the XML files. Programmatic Submission Tutorial Video Take a look at the Programmatic Submission Tutorial Video, which explains the workflow of a programmatic submission and goes over an example metadata submission. Programmatic Submission Tutorial Video. When building your XML files, we recommend using text editors (e.g.Sublime Text or VisualStudio) that allow you to visualise the structure of the XML with ease. Furthermore, these editors constantly check the consistency of the XML structure. Alternatively, and if the submission consists of a big number of objects (specially analyses), you may find the tool star2xml handy. This tool allows for a direct conversion between metadata in a tabular format (e.g. a spreadsheet) into XMLs. Identifying objects: Aliases and center names Every EGA object must be uniquely identified within the submission account using their alias attribute. The aliases can be used in submissions to make references between EGA objects. Let us dig into EGA's use of aliases and center names: alias: every object should have a name that is unique within your submission account. Once submitted successfully, every alias will be assigned a unique and permanent accession (EGA ID). refname: when an object references another by its alias, the alias of the referenced object goes into the "refname" attribute of the referencing object. For example, if a sample has the alias "sample1", and an experiment uses this sample, then the experiment's "EXPERIMENT/SAMPLE/refname" attribute should be "sample1". center_name: The "center_name" attribute is required within the submission XML and, if not provided when the object is submitted, it will be automatically filled using your default EGA account center_name. This element is the "controlled vocabulary acronym or abbreviation that is provided to the account holder when the account is first generated". If the submitter is brokering a submission for another institute, the submitter should use their special broker account name in broker_name while the data centre acronym remains in center_name. Log-in details should have been provided when you requested a submission account. Please contact our Helpdesk team if you have any questions. run_center: Many submitting centers contract out the actual sample sequencing to another center. In these cases, the sequencing center should be acknowledged in the run_center attribute. Again, this is controlled vocabulary and the acronym should be sought from EGA helpdesk before submitting. Please contact our Helpdesk team if you have any questions. Prepare your XMLs The goal of this section is to provide sufficient information to be able to create the metadata XML documents required for programmatic submissions. Please note, the EGA utilises the XML schemas maintained at the European Nucleotide Archive (ENA). It is important due to the fact that by using a similar system, some pieces of documentation from the ENA's programmatic submission can also help you with your programmatic submission to the EGA. For example, you can submit programmatically without using a Submission XML by following the steps at Submission actions without submission XML. A submission does not have to contain all different types of XMLs. For example, it is possible to submit only a few samples; or a study that is later to be referenced. You can submit each object one by one, or submit all in a batch: you choose what method of submission works best for you. We do recommend, nevertheless, that you submit the objects to be referenced (e.g. samples or studies) first, and the objects that reference these (e.g. experiments or datasets) afterwards. You can see a graphical view of these objects and their relationships at our EGA Schemas page. Independently of the submission scenario, you will always require a Dataset XML. The entity of a dataset is what is used to control access to the given data, in the form of runs or analyses. In other words, when a requester is granted access, it is through the dataset and the objects (e.g. runs or analyses) that the dataset contains, granting access to them in one go. Given the nature of the EGA, a dataset XML will always be required for the data access. First, we will differentiate between submissions of "raw" and "processed" data: Runs and Analyses, respectively. Run data submissions Raw data derives from instruments "as is". For example, a plain sequence file (e.g. FASTQ or unaligned BAM files) would be considered raw data. A typical raw (unaligned) sequence read submission consists of 8 XMLs: Submission Study Sample Experiment Run DAC Policy Dataset When technical reads (e.g. barcodes, adaptors or linkers) are included in the submitted raw sequences, a spot descriptor must be submitted to describe the position of the technical reads so that they can be removed. The following data files can be submitted without providing spot descriptor information in the experiment/run XML: BAM files (single reads) SFF files (single reads without barcodes) FastQ files (single reads without any technical reads) Complete Genomics files Analysis data submissions Processed data is, in some way, refined raw data. This includes raw data that has been processed by some form of analysis method (e.g. alignment, noise reduction, etc.). For example, an aligned sequence (e.g. BAM file), that was created using raw FASTQ files, would be a processed file. This category includes most types of data: sequence alignment files (e.g. BAM or CRAM), clinical data (e.g. phenopackets), sequence variation files (e.g. VCF), sequence annotation, etc. A typical EGA analysis data submission consists of 7 EGA XML: Submission Study Sample Analysis DAC Policy Dataset We accept three different types of analysis data submissions: BAM files (for multiple read alignments) VCF files (for sequence variations) Phenotype files (in any format) In anycase, keep in mind that samples must be created in order to be referenced in the analyses. In other words, the provenance of the information within the BAM, VCF and phenotype files Example XMLs Below you can find a non-extensive list of example XMLs with descriptive fields (i.e. explaining what to provide in each field). Furthermore, you can also find real examples (i.e. the true value of the provided fields) in our GitHub repository. Submission XML The submission XML is used to validate, submit or update any number of other objects. The submission XML refers to other XMLs. New submissions use the ADD action to submit new objects. Object updates are done using the MODIFY action and objects can be validated using the VERIFY action. Descriptive submission XML example True values submission XML example Study XML The study XML is used to describe the study containing a title, a study type and abstract as it would appear in a publication. Descriptive study XML example True values study XML example Please use the following notation within the property "STUDY_LINKS" when including PubMed citations in the Study XML: <STUDY_LINKS> <STUDY_LINK> <XREF_LINK> <DB>PUBMED</DB> <ID>18987735</ID> </XREF_LINK> </STUDY_LINK> </STUDY_LINKS> Sample XML The sample XML is used to describe the samples used to obtain the data, whether they were sequenced, measured in any other way, or have an associated phenotype. The mandatory fields include information about the taxonomy of the sample, sex, subject ID and phenotype. For example, the mandatory attribute fields for each sample would look like these, within the array of "SAMPLE_ATTRIBUTES": <SAMPLE_ATTRIBUTES> <SAMPLE_ATTRIBUTE> <TAG>subject_id</TAG> <VALUE>free text!</VALUE> </SAMPLE_ATTRIBUTE> <SAMPLE_ATTRIBUTE> <TAG>sex</TAG> <VALUE>female/male/unknown</VALUE> </SAMPLE_ATTRIBUTE> <SAMPLE_ATTRIBUTE> <TAG>phenotype</TAG> <VALUE>Free text, EFO terms (e.g. EFO:0000574) are recommended</VALUE> </SAMPLE_ATTRIBUTE> </SAMPLE_ATTRIBUTES> Sample is one of the most important objects to be described biologically, it is highly recommended that “TAG-VALUE” pairs are generated as SAMPLE_ATTRIBUTES to describe the sample in as much detail as possible. For example, were we to give the population ancestry of the sample, we could add a new attribute to the array, in which, for example, we would indicate that the sample derives from an individual of "Mende in Sierra Leone" (MSL), with an african ancestry: <SAMPLE_ATTRIBUTE> <TAG>Population</TAG> <VALUE>MSL</VALUE> </SAMPLE_ATTRIBUTE> Given that VALUE and TAG are free text, the combinations are limitless in order to give you full flexibility on the information you want to provide. We recommend you use the Experimental Factor Ontology (EFO) to describe the phenotypes of your samples. You can provide more than one phenotype by adding more items to the array of SAMPLE_ATTRIBUTES. Phenotypes considered essential for understanding the data submission should be provided. Each phenotype described should be listed as a separate sample attribute <SAMPLE_ATTRIBUTE> </SAMPLE_ATTRIBUTE>. There is no limit to the number of phenotypes that can be submitted. If a suitable EFO accession cannot be found for your phenotype attribute, please consider using another controlled ontology database (e.g. HPO, MONDO, etc.) before using free text. Descriptive sample XML example True values sample XML example Experiment XML The experiment XML is used to describe the experimental setup, including instrument platform and model details, library preparation details, and any additional information required to correctly interpret the submitted data. Where any of these values differ between runs, a new experiment object must exist, since runs are grouped by experiments. Each experiment references a study and a sample by alias, or if previously-submitted, by accession. Pooled data must be demultiplexed by barcode for submission. Descriptive experiment ( Illumina paired read ) XML example True values experiment ( Illumina paired read ) XML example Run XML The run XML is used to associate data files with experiments and typically comprises a single data file (e.g. a FASTQ file). Please note that pooled samples should be de-multiplexed prior submission and submitted as different runs. Descriptive run XML example True values run XML example Analysis XML Given that an analysis can be used to submit any type of processed data to the EGA, we will list below an example of each of the three most common types of analysis XMLs submitted to the EGA: sequence alignments (e.g. BAM files); sequence variation (e.g. VCF files); and clinical metadata or phenotypes (e.g. phenopackets). Regardless of the type of processed data submitted in the analysis, the analysis must be associated with a Study and can reference multiple types of other objects, from samples to experiments, if they are available at the EGA. Just like with Runs, whenever a file is submitted to the EGA through an analysis object, the file MD5 checksums must be present, in order for the EGA to validate file integrity upon transfer. This also includes index files when applicable (e.g. .bai.md5 files). Ideally, any analysis that uses a reference sequence for some kind of alignment (e.g. BAM, CRAM or VCF files), would contain metadata about the alignment, such as INSDC reference assemblies and sequences, by either using accessions (e.g. CM000663.1) or common labels (e.g. GRCh37). Read alignment (BAM) Analysis XML The Analysis can be used to submit BAM alignments to EGA. Only one BAM file can be submitted in each analysis and the samples used within the BAM read groups must be associated with Samples. Descriptive bam alignments XML example True values bam alignments XML example Sequence variation (VCF) Analysis XML The Analysis can be used to submit VCF files to EGA. Only one VCF file can be submitted in each analysis and the samples used within the VCF files must be associated with Samples. Download analysis XML (VCF) Phenotype files The Analysis XML can be used to submit phenotype files to the EGA. Only one phenotype file can be submitted in each analysis and the samples used within the phenotype files must be associated with EGA Samples. Download analysis XML (Phenotype) DAC XML The DAC XML describes the Data Access Committee (DAC) affiliated to the data submission. The DAC may consist of a group or a single individual and is responsible for the data access decisions based on the application procedure described in the POLICY.XML. As with any other object, if it was already submitted to the EGA, there is no need to submit it again: you can reference an existing object within the EGA. Hence, A DAC XML does not need to be provided if your submission is affiliated to an existing EGA DAC.. Further information on DACs can be found here, and you can always contact our Helpdesk team if you have further inquiries. Descriptive dac XML example True values dac XML example Policy XML The Policy XML describes the Data Access Agreement (DAA) to be affiliated to the named Data Access Committee. Descriptive policy XML example True values study XML example Dataset XML The dataset XML describes the data files, defined by the Run.XML and Analysis.XML, that make up the dataset and links the collection of data files to a specified Policy. The dataset xml is commonly the last metadata object to be submitted, since it references multiple other entities. Please consider the number of datasets that your submission consists of. For example, a case-control study is likely to consist of at least two datasets. In addition, we suggest that multiple datasets should be described for studies using the same samples but different sequence technologies. Descriptive dataset XML example True values dataset XML example Validating and submitting your EGA Validating EGA's XMLs through Webin After you have ensured that the XMLs are properly formatted and contain all the required information. You can proceed to validate and submit your data. Use the curl command to validate your XML file: Once you have prepared your XML file and asserted you have access to Webin, you can validate your XML file programmatically against EGA's schemas using the curl command. There are multiple ways in which you can validate your XMLs. This variety has to do with the fact that: (1) there are 2 instances of Webin (test and production); and (2) that validation is a default step during submission. In other words, any time that you submit your data through Webin, it will be validated automatically before being accepted. This allows for 4 possible routes of validation, all having the same validation result: validating or submitting to either the production service or the test service of Webin. For example, directly validating a "study" object XML in the testing service (wwwdev…) would look like the following: curl -u <USERNAME>:<PASSWORD> -F "ACTION=VALIDATE" "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" In this command, you would need to replace <USERNAME> and <PASSWORD> with your EGA account username and password, respectively. You would also replace <INPUT_FILE> with the path to your XML file. A mock example would look like the following: curl -u ega-test-data@ebi.ac.uk:egarocks -F "ACTION=VALIDATE" "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" The validation attempt can have different results depending on the given arguments: If your XML file is valid according to EGA's schemas, you will see a message indicating that your XML file is compliant. For example, see below for our mock example, where the "success" was "true" (i.e. no validation errors found). Nevertheless, notice how the "<STUDY accession=" is empty: it is because we were simply validating, so the study did not get an accession or ID. <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="receipt.xsl"?> <RECEIPT receiptDate="2023-04-11T15:19:28.850+01:00" submissionFile="submission-EBI-TEST_1681222768850.xml" success="true"> <STUDY accession="" alias="Mock example" status="PRIVATE"/> <SUBMISSION accession="" alias="SUBMISSION-11-04-2023-15:19:28:840"/> <MESSAGES> <INFO>VALIDATE action has been specified.</INFO> <INFO>Submission has been rolled back.</INFO> <INFO>This submission is a TEST submission and will be discarded within 24 hours</INFO> </MESSAGES> <ACTIONS>VALIDATE</ACTIONS> <ACTIONS>PROTECT</ACTIONS> If there are any errors or warnings, the tool will display them, allowing you to correct them before submitting your data to EGA. For example, in the following response, it is said that the object we were trying to submit was already existing, and therefore the "success" was "false". <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="receipt.xsl"?> <RECEIPT receiptDate="2023-04-11T15:12:35.609+01:00" submissionFile="submission-EBI-TEST_1681222355609.xml" success="false"> <STUDY alias="Example!_Human Microbiome Project SP56J" status="PRIVATE" holdUntilDate="2023-03-11Z"/> <SUBMISSION alias="SUBMISSION-11-04-2023-15:12:35:576"/> <MESSAGES> <ERROR>In study, alias: "Example!_Human Microbiome Project SP56J". The object being added already exists in the submission account with accession: "ERP127584".</ERROR> <INFO>VALIDATE action has been specified.</INFO> <INFO>Submission has been rolled back.</INFO> <INFO>This submission is a TEST submission and will be discarded within 24 hours</INFO> </MESSAGES> <ACTIONS>VALIDATE</ACTIONS> <ACTIONS>PROTECT</ACTIONS> If the curl command retrieves no response at all, please double check if your username and password are correctly provided. Also notice the "ACTION=..." argument passed to the Curl command. This specifies the action to take during the call to Webin, so we do not need a "Submission" XML just for a validation attempt. See more at submission actions without submission XML. Furthermore, validation of multiple files or objects (e.g. sample, experiment, study…) can be done in a single command by adding more arguments (i.e. '-F'). For example: curl -u <USERNAME>:<PASSWORD> -F "ACTION=VALIDATE" "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" -F "SAMPLE=@sample.xml" -F "DATASET=@dataset.xml" As mentioned above, beside "validate" action in the test environment, you can also validate your metadata by three other methods: "Validate" in the production server. From our example above, you simply need to take the "dev" away from the URL. curl -u <USERNAME>:<PASSWORD> -F "ACTION=VALIDATE" "https://www.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" "Add" in the development server. From our example above, you would simply need to replace the action: from "validate" to "add". Whatever is submitted to this service will be discarded in 24h, so whether something gets submitted or not would not matter in the long run. curl -u <USERNAME>:<PASSWORD> -F "ACTION=ADD" "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" "Add" in the productionserver. A combination of the previous two methods, which would render this attempt into a submission. This path is just to be taken when you are sure your metadata is compliant and what you want to submit. curl -u <USERNAME>:<PASSWORD> -F "ACTION=ADD" "https://www.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" What happens after the submission of a dataset XML? Once you have completed the registration of your dataset/s please contact our Helpdesk Team to provide a release date for your study. Please note that all datasets affiliated to unreleased studies are automatically placed on hold until the authorised submitter or DAC contact contact the EGA Helpdesk for the study to be released. We strongly advise you not to delete your data until EGA Helpdesk confirms that your data has been successfully archived.
