464 results for "GitHub+primate+acoustics+baboon+Zenodo+10.5281+Gui*"

in 9.40 milliseconds.

• EGA QuickView

  EGA QuickView Secure and Remote Access to EGA Files What is EGA QuickView? EGA QuickView is a FUSE file system that allows secure and remote access to EGA files. It is a hybrid of sshfs and crypt4ghfs, which means it uses ssh to connect to the EGA distribution servers, downloads data in Crypt4GH format, and transparently decrypts those files. It is compatible with Linux and macOS 12. Why use EGA QuickView? EGA QuickView is a highly useful and convenient tool for anyone who needs to access EGA files remotely and securely. The software offers a quick and convenient way to browse through EGA files without the need for complete downloads, thus saving bandwidth and time. Whether you are a medical professional requiring access to sensitive patient data or a research scientist working on a project that involves EGA files, EGA QuickView provides a highly useful solution. The software boasts advanced security features, ensuring that your data is safe and protected at all times. Get started with EGA QuickView To get started with EGA QuickView, visit the EGA QuickView GitHub repository for more information and installation instructions.

  Documentation access/download/visualisation/ega-quickview

• EGA FUSE Client

  EGA FUSE Client What is EGA FUSE Client? The EGA FUSE Client is a Java Native Runtime (JNR) based Filesystem in Userspace (FUSE) client that allows users to access authorised EGA files by presenting them in a virtual directory. This means that users can use authorised EGA files like regular files without having to download them first. The EGA FUSE Client is a useful tool for researchers who need to access and work with large amounts of EGA data. Why use EGA FUSE Client? The EGA FUSE Client provides several benefits for researchers working with EGA data: Efficient use of storage space: With the EGA FUSE Client, users can access authorised EGA files without having to download them first. This means that users can save storage space on their local machines. Faster access to data: Since users can access authorised EGA files directly through the EGA FUSE Client, they can work with the data more quickly than if they had to download the files first. Simplified workflow: The EGA FUSE Client presents authorised EGA files in a virtual directory, allowing users to work with the files like regular files. This simplifies the workflow for researchers who need to access and work with large amounts of EGA data. Get started with EGA FUSE Client To get started with EGA FUSE Client visit the EGA FUSE Client GitHub repository for more information and installation instructions.

  Documentation access/download/visualisation/fuse-client

• PyEGA3 download client

  PyEGA3 Checks prior to download - Is the dataset available for download? Most datasets are available using the PyEGA3 Download client, except for specific legacy datasets. For these ones, refer to Live Outbox distribution, an alternative way to download these datasets.If the dataset you've been granted access to begins with the ID EGAD5XXXXXXX, please refer to the Live Outbox distribution. If the dataset you have been granted access to contains encrypted data (filename ending in .gpg), contact EGA Helpdesk for detailed information on how to download these files.General Information The PyEGA3 download client is available at its GitHub repository. In its README you will find a step by step guide on how to use it to download files you have been granted access to. PyEGA3 implements the GA4GH-compliant HTSGET protocol supporting requests over genomic ranges. This enables the download of specific regions of interest rather than the entire file. HTSGET is not possible for all the files held by EGA: if the dataset does not contain index files (e.g. filename ending in *.crai), then genomic range requests cannot be performed. The maximum number of connections allowed is 30. If more than 30 connections are used, the download client will encounter 500 errors. The recommended number of connections for maximum output is one. Check out our video tutorial! Frequently Asked Questions General questions How do I download the datasets to which I have been granted access? After setting up your EGA download account, you can proceed to download using the EGA download client – PyEGA3. The pyEGA3 download client is a python-based tool for viewing and downloading files from authorised EGA datasets. This download client is continuously being developed for more user-friendly download experiences. Can I download the dataset’s metadata via the download client? No, the download client cannot be used to download metadata. Registered EGA users can download metadata of an authorised dataset by logging into the EGA webpage and navigating to the dataset of choice. Approximately two thirds down the page you will find the option to download the metadata as a zip file. How do I download large datasets as quickly as possible? The main EGA download client is pyEGA3 download client, a python-based tool for viewing and downloading files from authorised EGA datasets.We could exceptionally consider the option of an Aspera download for users experiencing very slow download rates or facing particular issues.Contact the EGA Helpdesk for further information. Pre-requirement and Installation What are the major requirements for the PyEGA3 download client installation and usage? The PyEGA3 client is compatible with any OS with Python 3.6+ installed. The client requires a connection to the internet, sufficient space on the destination drive, and the EGA download account credentials. Where can I find the PyEGA3 download client installation and download instructions? We strongly recommend checking our video tutorial demonstrating the usage of PyEGA3 from installation to file download. More detailed information is available at PyEGA3 GitHub. What ports are required for the PyEGA3 download client to function? PyEGA3 makes HTTPS calls to the EGA Data API (https://ega.ebi.ac.uk:8443). This port 8443 must be reachable from the location where PyEGA3 is executed to avoid timeouts. PyEGA3 download client How to update the PyEGA3 download client to be on the latest version? Updating the client to the latest version can be achieved by running the following command: pip3 install pyega3 --upgrade I do not have an active EGA account. Can I test the download with the PyEGA3 download client? An EGA download test account (ega-test-data@ebi.ac.uk) has been created for troubleshooting and training purposes. The test account does not require an EGA username and password because it contains publicly accessible files from the 1000 Genomes Project. More information about the use of this account and log in details are available at pyEGA3. I lost my connection while the download was in progress. Does this mean that I have to download the file again from the start? The PyEGA3 download client supports automatic resumption of downloads. To enable quicker download speeds, the API breaks files into up to four segments and downloads them in parallel. With the resume feature, downloads will automatically resume if you encounter any errors or if the connection is interrupted. How can I improve the overall download speed of the PyEGA3 download client? Download speeds can be optimised using the --connections parameter which will parallelise download at the file level. If the --connections parameter is provided, all files >100Mb will be downloaded using the specified number of parallel connections. It is important to note that files are still downloaded sequentially, so using multiple connections does not mean downloading multiple files in parallel. The maximum number of connections allowed is 30. If more than 30 connections are used, the download client will encounter 500 errors. The recommended number of connections for maximum output is one. Why is my file taking a long time to be saved? Please note that when a file is being saved, it goes through two processes. First, the downloaded file "chunks" are pieced back together to reconstruct the original file. Secondly, PyEGA3 calculates the checksum of the file to confirm that the file was downloaded successfully. Larger files will take more time to reconstruct and validate the checksum. Why is the HTSGET protocol not working with my dataset of interest? PyEGA3 implements the GA4GH-compliant htsget protocol for supporting requests over genomic ranges. This exciting new feature means that for data files with accompanying index files (e.g. .crai for CRAM files) users can download specific regions of interest rather than the entire file, saving both time and storage space. Please note that in order for the genomic range requests to work for a BAM, CRAM or VCF file there must be an associated BAI, CRAI or TBI file, respectively. If the dataset does not contain these index files then genomic range requests cannot be performed. After download Do I need to decrypt data files downloaded through the pyEGA3 download client? Files are transferred over secure HTTPS connections and received unencrypted, thus there is no need for decryption after data download. What is the purpose of an MD5 file? After the download completes, file integrity is verified using checksums. The PyEGA3 download client automatically verifies each file’s unencrypted md5 after download to ensure that the file was downloaded correctly from the EGA. Errors Why do I get a “400 Client Error” while trying to access the download client? Ensure that your credentials are formatted correctly. Please contact the EGA Helpdesk if you are unsure about your account’s username (i.e. your email address). Why do I get a “500 Server Error”? It is important to note that files are still downloaded sequentially, so using multiple connections does not mean downloading multiple files in parallel. The maximum number of connections allowed is 30. If more than 30 connections are used, the download client will encounter 500 errors. The recommended number of connections for maximum output is one. Otherwise, 500 server errors are internal server errors from EGA’s end. Retry after some time and, if the 500 error persists: check our homepage or Twitter for any outage information or report it to EGA Helpdesk. Why do I get a “slice error”? This type of error occurs when one slice download was prematurely interrupted. Please re-start the same download. The download client will pick up any partial download and download any missing slices. Why do I get the error “Dataset' EGADXXXXXXXXXXX' is not in the list of your authorised datasets”? Please note that the data access was given to the email (username) provided to the Data Access Committee (DAC) in your official request. Therefore, you shall check that the account that you are using is the same one that you sent to the DAC. Why do I get “ERROR:root:Failed to obtain IP address”? This might mean that the connection ports are blocked. Check that the port 8443 is reachable.In case the port is reachable and the error persists, please try to modify the file pyega3.py on line 44 in the python module installation: Replace endpoint = 'https://ipinfo.io/json' with endpoint = 'https://api.myip.com' Why do I get the error “Invalid username, password or secret key - please check and retry”? Make sure that you are using the correct credentials (username and password) and there is no typo. Contact EGA Helpdesk if the problem persists.

  Documentation access/download/files/pyega3

• EGA metadata schema

  EGA Schemas In this page, you will find information on how the European Genome-phenome Archive (EGA) manages its metadata standards using both XML Schema Definition (XSD) and JavaScript Object Notation (JSON) formats. If you are not sure what this means, you may want to explore our brief metadata introduction. This information may be of your interest if you are planning to learn more about how the EGA is built and how to wrap around it for other processes. Nevertheless, if you are a common user (e.g. submitter or requester), you would not have to worry about these schemas nor their format, since they are implemented in user-friendly ways for you. Metadata standards are rules that define how to format and structure data of metadata objects (i.e. entities), like EGA's samples or experiments in a consistent manner. These objects are the nodes of the metadata model of the EGA (Figure 1). Figure 1. Diagram of EGA's metadata model. The model's building blocks are objects (e.g. sample), which can reference each other (e.g. an experiment referencing the used samples). Once your files are uploaded, they can also be referenced by Runs and Analyses. The submission object is an object itself that compiles many others. At EGA, we inherit our metadata schemas from the European Nucleotide Archive (ENA), and we have expanded them to include bespoke objects such as "Policy", "Dataset", and "DAC" (Data Access Committee) for our specific use-case: handling sensitive human data. See below a list of all our metadata objects and some context for each. Metadata Object EGA accession Description Examples of metadata fields Study EGAS… Information about the study Study type, study title, study abstract… Sample EGAN… Information about the used samples in the experiment or analysis Taxon ID, scientific name, biological sex, phenotype… Experiment EGAX… Information about the performed experiment Used libraries, sequencing platform, reference to the used samples… Analysis EGAZ… Contains information about the analysis Type of analysis, used assembly, reference sequence… Run EGAR… The run holds information about the files containing the raw reads generated in a run of sequencing Platform, spot descriptor, raw file references… DAC EGAC… Contains information about the Data Access Committee (DAC) DAC contacts, contact emails… Policy EGAP… Contains the Data Access Agreement (DAA) and policy which its usage complies with Policy text, data use ontologies (DUO) codes… Dataset EGAD… Contains the collection of runs/analysis to be subject to controlled access Dataset type, compilation of Run's and Analysis' IDs There are two different sets of schemas, based on their formats, in which the EGA accepts metadata: XSDs (for XML files) and JSON Schemas (for JSON files). EGA's XML Schema Definition . When programmatic submissions are pushed through the European Bioinformatics Institute (EBI) system, XML format is used. The schemas that are applied for this format are defined in XML Schema Definition (XSD) files, which can be found at ENA's GitHub repository. You can find more information on how to validate and submit your data programmatically in our programmatic submission documentation. Furthermore, see at our GitHub repository some XML examples with either made-up values (similar to what you would submit) or descriptive values for each field (just for documentation). EGA's JSON Metadata Schemas . When programmatic submissions are pushed through the Centre for Genomic Regulation (CRG) system, JSON format specifications are used instead. See the full JSON specifications for further details. In conclusion, the EGA metadata schemas are crucial for maintaining the quality and consistency of submitted data. By understanding and following the rules outlined in these schemas, you can ensure that your submissions comply with the EGA's standards and contribute to a valuable and accessible genomic resource. Sample checklists Besides the standards in our schemas, we have another layer called Sample checklists. These are another system that the EGA inherits from ENA, and specifies what attributes are required or allowed for a sample object. The EGA uses these checklists to enforce that, for example, a sample object has the three mandatory attributes: subject ID, sex and phenotype of the individual the sample was taken from. When you submit to EGA, our checklist is automatically selected by default.

  Documentation submission/metadata/ega-schema

• How to encrypt files with Crypt4gh

  File preparation Due to the processes used at the EGA for file archival the use of non-alphanumeric characters in a filename will cause issues in archival. By convention whitespaces in filenames are to be avoided and should be replaced with the underscore character (_). Before encrypting your files please make sure that any files that will be uploaded to EGA do not use special characters such as # ? ( ) [ ] / \ = + < > : ; " ' , * ^ | & Crypt4gh EGACryptor Files encrypted with Crypt4gh must be uploaded via INBOX Before uploading If you are not a registered EGA user, you will first need an EGA user account. Please note that it may take a few days for your account to be activated, as it needs to be vouched for by the EGA Helpdesk. Once your account is validated you will be able to request a submitter role. Meanwhile, you can create and add your public key to your EGA account profile. This option is not available for old submission accounts (e.g. ega-box-NNN). As soon as you have been granted a submitter role, you will be able to connect with your username and password to the EGA inbox using the SFTP protocol. If you have also registered a public key in your profile, you can also connect using this key. Encrypting your files Please note that you can also encrypt your files by uploading them directly to the "to-encrypt" folder in your upload area If your connection is unstable, please encrypt them using Crypt4gh The EGA encryption of this inbox is based on Crypt4GH. You can install a python implementation of it, with pip install crypt4gh or directly from the Github repository pip install git+https://github.com/EGA-archive/crypt4gh.git Save now the following Crypt4GH public key, into a file, say ingestion.pubkey. __EGA_INGESTION_PUBKEY__ Encrypt a given file with the following command: crypt4gh encrypt --recipient_pk ingestion.pubkey < file_to_encrypt > encrypted_file.c4gh The command reads the file from stdin (with < ) and output the encrypted version to stdout (with > ). Replace file_to_encrypt and encrypted_file.c4gh with the appropriate filenames but make sure to not use the same filename for both reading and writing because your SHELL would then truncate both files before you even read or write.

  Documentation submission/data/file-preparation/crypt4gh

• Distribution of data using Live Distribution

  Live Distribution Welcome to the documentation for using the Live Distribution feature for distributing data files securely through the EGA platform. This guide will walk you through the process of downloading encrypted files and decrypting them using Crypt4GH. Please follow the steps below to ensure a smooth experience. Before Downloading Create an EGA user. Make sure that you have the permissions to download the dataset of interest. In case you don’t have access, request access to the dataset. Add your Crypt4GH-compatible public key to your EGA account. Please allow a few hours for your public key to be synced with your profile. Afterwards, you will be able to connect to your EGA outbox using the SFTP protocol. Download Graphical User Interface (GUI) You can use any GUI that supports SFTP connections, such as FileZilla, an open-source FTP client. For Filezilla as your GUI, follow these steps to download files: Open FileZilla and access Site Manager (File > Site Manager). Create a new connection with the following settings (Figure 1): Protocol: SFTP - SSH File Transfer Protocol Host: __EGA_OUTBOX_DOMAIN__ Logon Type: Key file User: your EGA username Key file: path/to/your/private_key Figure 1: Process of establishing a new connection to __EGA_OUTBOX_DOMAIN__ using a key file as the logon method in FileZilla. The figure showcases the FileZilla version 3.52.2 operating on IOS v11.2.3. By following the depicted steps, users can create a secure and efficient connection to the EGA outbox, ensuring seamless data transfers. Click Connect to access your Outbox. This folder serves as your storage space within the EGA cloud, containing files accessible for download in a secure way. Browse the remote directory on the right side of the FileZilla screen. Select the files you wish to download, right-click, and choose Download (Figure 2). Figure 2: Step-by-step process of manually downñoad files from __EGA_OUTBOX_DOMAIN__ using FileZilla, with FileZilla version 3.52.2 operating on IOS v11.2.3. The figure demonstrates how users can downñoad data from the EGA outbox to their local storage by following the depicted steps SFTP command line You can also use the SFTP command line to securely download files from the EGA Outbox. Using SFTP command line client in Linux/Unix Open a terminal window Enter the following command to connect: sftp username@hostname Enter your EGA password To see a list of available sftp commands type help sftp> put – Upload file sftp> get – Download file sftp> cd path – Change remote directory to ‘path’ sftp> pwd – Display remote working directory sftp> lcd path – Change the local directory to ‘path’ sftp> lpwd – Display local working directory sftp> ls – Display the contents of the remote working directory sftp> lls – Display the contents of the local working directory Type get command to download files. For example: get encrypted_file.c4gh Use the bye command to close the connection (SFTP session). Convenient SSH settings Include the following settings in your SSH config file, located in ~/.ssh/config Host __EGA_OUTBOX_DOMAIN__ EGA-outbox hostname __EGA_OUTBOX_DOMAIN__ User username IdentityFile path/to/the/private/key Replace username and path/to/the/private/key with the appropriate settings, and you will be able to connect to the __EGA_OUTBOX_DOMAIN__ simply using sftp EGA-outbox. How to decrypt Files archived at the EGA are encrypted based on Crypt4GH. Hence, to decrypt the files you need to install Crypt4GH. You can install a python implementation of it, with pip install crypt4gh or directly from the Github repository pip install git+https://github.com/EGA-archive/crypt4gh.git After installing Crypt4GH, decrypt files using the following command: crypt4gh decrypt --sk /path/private/key < encrypted_file.c4gh > decrypted_filename The command reads the encrypted file from stdin (with <) and output the decrypted version to stdout (with >). Replace encrypted_file.c4gh and decrypted_filename with the appropriate filenames but make sure to not use the same filename for both reading and writing because your SHELL would then truncate both files before you even read or write. Frequently Asked Questions What username should I use to log in to my outbox? The authentication process for logging in to the EGA website, as well as accessing your inbox and outbox, requires the use of your username, not your email address. Therefore, if you registered a username different from your email address when creating your EGA account, you must use that username to log in. If you have forgotten your registered username, please, contact our Helpdesk team for assistance. I see that some files in my dataset have 'unavailable' as extension. What should I do? Within your Outbox, you'll find a list of all the datasets available for download. Occasionally, certain files may be marked as "unavailable". These unavailable files can be identified by the "unavailable" extension added to their filenames (e.g. filename.fastq.gz.unavailable.c4gh). If you encounter an unavailable file that you need, please reach out to our Helpdesk. We'll promptly work on making the file accessible for download as soon as possible. Specific to using keys Can I access one EGA account from different devices? Yes, you can access your account from different devices by linking several public keys to your EGA account. Each device can generate a unique public-private key pair, and the corresponding public keys can be linked to the same account. This way, you can use different public keys on different devices and still have access to the same account and data. I have several keys and I don't remember which one is which When generating SSH keys, it's a good practice to add a comment using the -C flag. This will allow you to add a descriptive tag to your key, making it easier to identify later on. Here's an example command that generates an SSH key with a comment: ssh-keygen -t ed25519 -C work-pass In this example, we're generating an ed25519 SSH key with the comment work-pass. Once you have multiple keys with different comments, you can use the comments to easily identify each key. To view the comments for your existing SSH keys, you can use the following command: ssh-keygen -l -f /path/to/key This will display the key fingerprint and the associated comment. By checking the comments, you should be able to identify which key is which. What if I can't find my SSH keys for uploading files with a key file, and how can I use new keys? If you can't find your SSH keys, don't worry - you can make new ones. To do this, open your terminal or command prompt and type a command to make a new SSH key. You can pick a name for the key, and choose a password to keep it safe. After making the key, you can add the new key to your account or server where you want to upload files using the key file. This usually involves copying and pasting the key's "public" (e.g. file.pub) part to the right place. If you lose track of the key again, just make a new one and add it again. Keep in mind that SSH keys belong to you and your computer, so if you switch computers or accounts, you'll need to make new keys. I don't want to type the passphrase every time I use the key. What can I do? You can use an ssh-agent to avoid typing the passphrase every time you use the key. An ssh-agent is a program that stores your private keys in memory and provides them to ssh when needed. You can add your key to the ssh-agent using the command ssh-add followed by the path to your key file.Here's an example of the steps to follow: Open a terminal window.Start the ssh-agent by typing the command eval $(ssh-agent).Add your key to the ssh-agent by typing the command ssh-add [key filepath]. For instance, if your key file is located in the home directory with the name mykey, the command will look like this: ssh-add ~/mykey After adding your, key to the ssh-agent, you should be able to use ssh without having to enter your passphrase every time. Can I use my password for authentication (without my private key)? If you prefer to use your username and password for authentication instead of your private key, you can still do so. When using a Graphical User Interface (GUI) such as FileZilla, you can select Ask for password as your Logon Type (Figure 3). This option will prompt you to enter your password when you click Connect, instead of using your private key. Figure 3: This option will prompt you to enter your password when you click "Connect", instead of using your private key. Figure 3: Process of establishing a new connection to __EGA_OUTBOX_DOMAIN__ using your password as the logon method in FileZilla. The figure showcases the FileZilla version 3.52.2 operating on IOS v11.2.3. By following the depicted steps, users can create a secure and efficient connection to the inbox, ensuring seamless data transfers. It's worth noting that using a password for authentication can be less secure than using an SSH key, as passwords can be more easily compromised through various means. However, if you choose to use your password for authentication, selecting "Ask for password" as your Logon Type is a good way to do so securely via a GUI. Why is it better to use my key and not my password? SSH keys for authentication is generally considered to be more secure and convenient than using passwords. SSH keys are more difficult to crack than passwords, and they can be restricted to specific users and machines, giving you more control over access. Once you set up your SSH keys, you can use them to authenticate quickly and easily, without having to enter a password every time. This makes automation of tasks, such as uploading encrypted files, much simpler. Additionally, SSH keys provide better logging, allowing you to keep track of who is accessing your systems and when. All in all, using SSH keys is a good practice for improving security and convenience in your authentication process.

  Documentation access/download/files/live-outbox

• Out with the Java, in with the Python - new EGA data download client unveiled

  We will shortly be deprecating the Java-based download client also known as API V2. The Java client began its life at the back end of 2014, and during its time has seen many developments that have served our users well. During its tenure, the Java client enabled distribution of over 6.5 million files totalling nearly 15 petabytes of data to researchers around the globe. Recent increases in genomic and phenotypic data file sizes, the number of datasets archived in the EGA, the number of users downloading data, and the diversity of download locations has meant that at times the Java client struggled to distribute data efficiently. In response, our team of developers have built a new download client with more robust features and better functionality and reliability. Enter stage left our shiny new API V3.1! The new API V3.1 will replace both API V2 and API V3. The download client for the new API V3.1 is written in Python, and the code is openly available on Github. This Python client makes https calls to the EGA AAI (https://ega.ebi.ac.uk:8443/) and to the EGA Data API (https://ega.ebi.ac.uk:8052), both of which must be accessible from the client location. In addition to general updates to improve performance, the API V3.1 has three new features enhancing the EGA distributions services. First, requested data are now downloaded over secure https connections rather than http, meaning that data can be delivered unencrypted. Instead of separately requesting, downloading, and decrypting data using the Java client, data can now be requested and downloaded using a single command without the need for decrypting. The API V3.1 also automatically verifies each file’s unencrypted md5 after download to ensure the file has downloaded correctly. Second, the new API supports segmenting and automatic resuming. To enable quicker download speeds, the API breaks files into up to four segments and downloads them in parallel. With the new resume feature, downloads - both segmented and continuous stream - will automatically resume if they encounter any errors or the connection is interrupted. These features result in both a faster and more reliable download experience. Finally, the new API V3.1 implements the GA4GH-compliant htsget protocol for supporting requests over genomic ranges. This exciting new feature means that for data files with accompanying index files (e.g. .crai for CRAM files) users can download specific regions of interest rather than the entire file saving both time and storage space. The new features of the Python-based API V3.1 represent significant improvements in security, speed, reliability, and flexibility that are sure to provide a better data download experience. To take full advantage of these features, users must update to Python 3.X. Considering the changes in the data structures with the new API, the upgrade to a new version of the Python client is also mandatory. Updating the client to the latest version can be achieved by running the following command: pip3 install pyega3 -U We are excited for users to try out the new features and take advantage of the multifariously growing EGA distribution services for human biomedical omics data. If you have any questions or feedback, please email the EGA Helpdesk team. We'd love to hear from you!

  Blog new-ega-data-download-client

• Programmatic submission based on XML

  Programmatic submissions (XML based) For further information please check our Submission FAQs, submission quickguide as well as submission terms! Introduction Besides the Submitter Portal tool, EGA supports programmatic sequence and clinical data metadata submissions. If you are not sure what this means, you may want to explore our brief metadata introduction. Programmatic submissions are recommended for array-based submission. Moreove, it may be of help if your submission is recurrent or it is difficult to manage manually due to its sheer size. Otherwise, we highly recommend using the Submitter Portal to perform submissions. In this page we will guide you through the required steps to programmatically submit data to the EGA. Programmatic submissions require your metadata to be structured for an easy and straightforward validation and archival. It basically consists in formatting your metadata as Extensible markup language (XML) files and submitting them to the EGA using the WEBIN Before submitting metadata to the EGA, it is important to ensure that the information in your XML files is compliant with our standards. You can see further details on how these standards are maintained at EGA at our EGA Schemas documentation page. Using WEBIN, you can validate your XML files against EGA's schemas to ensure that your metadata is compliant before submission. WEBIN services WEBIN production service WEBIN test service We advise you to submit your metadata to the test service when submitting to the production service for the first time. The test service is identical to the production service except that all submissions will be discarded in the following 24 hours. This allows you to learn about the submission process without having to worry about data being submitted. Authentication Authentication is required each time a submission is made. The submission service uses HTTPS protocol for metadata encryption and identification to provide a secure submission environment. Data file upload Both Runs and Analyses reference files (e.g. FASTQ need to be uploaded to the EGA before these metadata objects are submitted. In other words, if you submit a Run that references a file that we cannot find associated with your account, the metadata submission will fail. See further details on how to upload your files in our File Upload documentation. Metadata model of the EGA Our metadata model is formed by multiple metadata objects. Check further details in our documentation at our EGA Schema documentation page. Working with EGA XMLs files Now that the basic concepts of the EGA metadata have been described, you can start preparing your programmatic submission through XML. Here you will find the guidance on how to prepare the XML files. Programmatic Submission Tutorial Video Take a look at the Programmatic Submission Tutorial Video, which explains the workflow of a programmatic submission and goes over an example metadata submission. Programmatic Submission Tutorial Video. When building your XML files, we recommend using text editors (e.g.Sublime Text or VisualStudio) that allow you to visualise the structure of the XML with ease. Furthermore, these editors constantly check the consistency of the XML structure. Alternatively, and if the submission consists of a big number of objects (specially analyses), you may find the tool star2xml handy. This tool allows for a direct conversion between metadata in a tabular format (e.g. a spreadsheet) into XMLs. Identifying objects: Aliases and center names Every EGA object must be uniquely identified within the submission account using their alias attribute. The aliases can be used in submissions to make references between EGA objects. Let us dig into EGA's use of aliases and center names: alias: every object should have a name that is unique within your submission account. Once submitted successfully, every alias will be assigned a unique and permanent accession (EGA ID). refname: when an object references another by its alias, the alias of the referenced object goes into the "refname" attribute of the referencing object. For example, if a sample has the alias "sample1", and an experiment uses this sample, then the experiment's "EXPERIMENT/SAMPLE/refname" attribute should be "sample1". center_name: The "center_name" attribute is required within the submission XML and, if not provided when the object is submitted, it will be automatically filled using your default EGA account center_name. This element is the "controlled vocabulary acronym or abbreviation that is provided to the account holder when the account is first generated". If the submitter is brokering a submission for another institute, the submitter should use their special broker account name in broker_name while the data centre acronym remains in center_name. Log-in details should have been provided when you requested a submission account. Please contact our Helpdesk team if you have any questions. run_center: Many submitting centers contract out the actual sample sequencing to another center. In these cases, the sequencing center should be acknowledged in the run_center attribute. Again, this is controlled vocabulary and the acronym should be sought from EGA helpdesk before submitting. Please contact our Helpdesk team if you have any questions. Prepare your XMLs The goal of this section is to provide sufficient information to be able to create the metadata XML documents required for programmatic submissions. Please note, the EGA utilises the XML schemas maintained at the European Nucleotide Archive (ENA). It is important due to the fact that by using a similar system, some pieces of documentation from the ENA's programmatic submission can also help you with your programmatic submission to the EGA. For example, you can submit programmatically without using a Submission XML by following the steps at Submission actions without submission XML. A submission does not have to contain all different types of XMLs. For example, it is possible to submit only a few samples; or a study that is later to be referenced. You can submit each object one by one, or submit all in a batch: you choose what method of submission works best for you. We do recommend, nevertheless, that you submit the objects to be referenced (e.g. samples or studies) first, and the objects that reference these (e.g. experiments or datasets) afterwards. You can see a graphical view of these objects and their relationships at our EGA Schemas page. Independently of the submission scenario, you will always require a Dataset XML. The entity of a dataset is what is used to control access to the given data, in the form of runs or analyses. In other words, when a requester is granted access, it is through the dataset and the objects (e.g. runs or analyses) that the dataset contains, granting access to them in one go. Given the nature of the EGA, a dataset XML will always be required for the data access. First, we will differentiate between submissions of "raw" and "processed" data: Runs and Analyses, respectively. Run data submissions Raw data derives from instruments "as is". For example, a plain sequence file (e.g. FASTQ or unaligned BAM files) would be considered raw data. A typical raw (unaligned) sequence read submission consists of 8 XMLs: Submission Study Sample Experiment Run DAC Policy Dataset When technical reads (e.g. barcodes, adaptors or linkers) are included in the submitted raw sequences, a spot descriptor must be submitted to describe the position of the technical reads so that they can be removed. The following data files can be submitted without providing spot descriptor information in the experiment/run XML: BAM files (single reads) SFF files (single reads without barcodes) FastQ files (single reads without any technical reads) Complete Genomics files Analysis data submissions Processed data is, in some way, refined raw data. This includes raw data that has been processed by some form of analysis method (e.g. alignment, noise reduction, etc.). For example, an aligned sequence (e.g. BAM file), that was created using raw FASTQ files, would be a processed file. This category includes most types of data: sequence alignment files (e.g. BAM or CRAM), clinical data (e.g. phenopackets), sequence variation files (e.g. VCF), sequence annotation, etc. A typical EGA analysis data submission consists of 7 EGA XML: Submission Study Sample Analysis DAC Policy Dataset We accept three different types of analysis data submissions: BAM files (for multiple read alignments) VCF files (for sequence variations) Phenotype files (in any format) In anycase, keep in mind that samples must be created in order to be referenced in the analyses. In other words, the provenance of the information within the BAM, VCF and phenotype files Example XMLs Below you can find a non-extensive list of example XMLs with descriptive fields (i.e. explaining what to provide in each field). Furthermore, you can also find real examples (i.e. the true value of the provided fields) in our GitHub repository. Submission XML The submission XML is used to validate, submit or update any number of other objects. The submission XML refers to other XMLs. New submissions use the ADD action to submit new objects. Object updates are done using the MODIFY action and objects can be validated using the VERIFY action. Descriptive submission XML example True values submission XML example Study XML The study XML is used to describe the study containing a title, a study type and abstract as it would appear in a publication. Descriptive study XML example True values study XML example Please use the following notation within the property "STUDY_LINKS" when including PubMed citations in the Study XML: <STUDY_LINKS> <STUDY_LINK> <XREF_LINK> <DB>PUBMED</DB> <ID>18987735</ID> </XREF_LINK> </STUDY_LINK> </STUDY_LINKS> Sample XML The sample XML is used to describe the samples used to obtain the data, whether they were sequenced, measured in any other way, or have an associated phenotype. The mandatory fields include information about the taxonomy of the sample, sex, subject ID and phenotype. For example, the mandatory attribute fields for each sample would look like these, within the array of "SAMPLE_ATTRIBUTES": <SAMPLE_ATTRIBUTES> <SAMPLE_ATTRIBUTE> <TAG>subject_id</TAG> <VALUE>free text!</VALUE> </SAMPLE_ATTRIBUTE> <SAMPLE_ATTRIBUTE> <TAG>sex</TAG> <VALUE>female/male/unknown</VALUE> </SAMPLE_ATTRIBUTE> <SAMPLE_ATTRIBUTE> <TAG>phenotype</TAG> <VALUE>Free text, EFO terms (e.g. EFO:0000574) are recommended</VALUE> </SAMPLE_ATTRIBUTE> </SAMPLE_ATTRIBUTES> Sample is one of the most important objects to be described biologically, it is highly recommended that “TAG-VALUE” pairs are generated as SAMPLE_ATTRIBUTES to describe the sample in as much detail as possible. For example, were we to give the population ancestry of the sample, we could add a new attribute to the array, in which, for example, we would indicate that the sample derives from an individual of "Mende in Sierra Leone" (MSL), with an african ancestry: <SAMPLE_ATTRIBUTE> <TAG>Population</TAG> <VALUE>MSL</VALUE> </SAMPLE_ATTRIBUTE> Given that VALUE and TAG are free text, the combinations are limitless in order to give you full flexibility on the information you want to provide. We recommend you use the Experimental Factor Ontology (EFO) to describe the phenotypes of your samples. You can provide more than one phenotype by adding more items to the array of SAMPLE_ATTRIBUTES. Phenotypes considered essential for understanding the data submission should be provided. Each phenotype described should be listed as a separate sample attribute <SAMPLE_ATTRIBUTE> </SAMPLE_ATTRIBUTE>. There is no limit to the number of phenotypes that can be submitted. If a suitable EFO accession cannot be found for your phenotype attribute, please consider using another controlled ontology database (e.g. HPO, MONDO, etc.) before using free text. Descriptive sample XML example True values sample XML example Experiment XML The experiment XML is used to describe the experimental setup, including instrument platform and model details, library preparation details, and any additional information required to correctly interpret the submitted data. Where any of these values differ between runs, a new experiment object must exist, since runs are grouped by experiments. Each experiment references a study and a sample by alias, or if previously-submitted, by accession. Pooled data must be demultiplexed by barcode for submission. Descriptive experiment ( Illumina paired read ) XML example True values experiment ( Illumina paired read ) XML example Run XML The run XML is used to associate data files with experiments and typically comprises a single data file (e.g. a FASTQ file). Please note that pooled samples should be de-multiplexed prior submission and submitted as different runs. Descriptive run XML example True values run XML example Analysis XML Given that an analysis can be used to submit any type of processed data to the EGA, we will list below an example of each of the three most common types of analysis XMLs submitted to the EGA: sequence alignments (e.g. BAM files); sequence variation (e.g. VCF files); and clinical metadata or phenotypes (e.g. phenopackets). Regardless of the type of processed data submitted in the analysis, the analysis must be associated with a Study and can reference multiple types of other objects, from samples to experiments, if they are available at the EGA. Just like with Runs, whenever a file is submitted to the EGA through an analysis object, the file MD5 checksums must be present, in order for the EGA to validate file integrity upon transfer. This also includes index files when applicable (e.g. .bai.md5 files). Ideally, any analysis that uses a reference sequence for some kind of alignment (e.g. BAM, CRAM or VCF files), would contain metadata about the alignment, such as INSDC reference assemblies and sequences, by either using accessions (e.g. CM000663.1) or common labels (e.g. GRCh37). Read alignment (BAM) Analysis XML The Analysis can be used to submit BAM alignments to EGA. Only one BAM file can be submitted in each analysis and the samples used within the BAM read groups must be associated with Samples. Descriptive bam alignments XML example True values bam alignments XML example Sequence variation (VCF) Analysis XML The Analysis can be used to submit VCF files to EGA. Only one VCF file can be submitted in each analysis and the samples used within the VCF files must be associated with Samples. Download analysis XML (VCF) Phenotype files The Analysis XML can be used to submit phenotype files to the EGA. Only one phenotype file can be submitted in each analysis and the samples used within the phenotype files must be associated with EGA Samples. Download analysis XML (Phenotype) DAC XML The DAC XML describes the Data Access Committee (DAC) affiliated to the data submission. The DAC may consist of a group or a single individual and is responsible for the data access decisions based on the application procedure described in the POLICY.XML. As with any other object, if it was already submitted to the EGA, there is no need to submit it again: you can reference an existing object within the EGA. Hence, A DAC XML does not need to be provided if your submission is affiliated to an existing EGA DAC.. Further information on DACs can be found here, and you can always contact our Helpdesk team if you have further inquiries. Descriptive dac XML example True values dac XML example Policy XML The Policy XML describes the Data Access Agreement (DAA) to be affiliated to the named Data Access Committee. Descriptive policy XML example True values study XML example Dataset XML The dataset XML describes the data files, defined by the Run.XML and Analysis.XML, that make up the dataset and links the collection of data files to a specified Policy. The dataset xml is commonly the last metadata object to be submitted, since it references multiple other entities. Please consider the number of datasets that your submission consists of. For example, a case-control study is likely to consist of at least two datasets. In addition, we suggest that multiple datasets should be described for studies using the same samples but different sequence technologies. Descriptive dataset XML example True values dataset XML example Validating and submitting your EGA Validating EGA's XMLs through Webin After you have ensured that the XMLs are properly formatted and contain all the required information. You can proceed to validate and submit your data. Use the curl command to validate your XML file: Once you have prepared your XML file and asserted you have access to Webin, you can validate your XML file programmatically against EGA's schemas using the curl command. There are multiple ways in which you can validate your XMLs. This variety has to do with the fact that: (1) there are 2 instances of Webin (test and production); and (2) that validation is a default step during submission. In other words, any time that you submit your data through Webin, it will be validated automatically before being accepted. This allows for 4 possible routes of validation, all having the same validation result: validating or submitting to either the production service or the test service of Webin. For example, directly validating a "study" object XML in the testing service (wwwdev…) would look like the following: curl -u <USERNAME>:<PASSWORD> -F "ACTION=VALIDATE" "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" In this command, you would need to replace <USERNAME> and <PASSWORD> with your EGA account username and password, respectively. You would also replace <INPUT_FILE> with the path to your XML file. A mock example would look like the following: curl -u ega-test-data@ebi.ac.uk:egarocks -F "ACTION=VALIDATE" "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" The validation attempt can have different results depending on the given arguments: If your XML file is valid according to EGA's schemas, you will see a message indicating that your XML file is compliant. For example, see below for our mock example, where the "success" was "true" (i.e. no validation errors found). Nevertheless, notice how the "<STUDY accession=" is empty: it is because we were simply validating, so the study did not get an accession or ID. <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="receipt.xsl"?> <RECEIPT receiptDate="2023-04-11T15:19:28.850+01:00" submissionFile="submission-EBI-TEST_1681222768850.xml" success="true"> <STUDY accession="" alias="Mock example" status="PRIVATE"/> <SUBMISSION accession="" alias="SUBMISSION-11-04-2023-15:19:28:840"/> <MESSAGES> <INFO>VALIDATE action has been specified.</INFO> <INFO>Submission has been rolled back.</INFO> <INFO>This submission is a TEST submission and will be discarded within 24 hours</INFO> </MESSAGES> <ACTIONS>VALIDATE</ACTIONS> <ACTIONS>PROTECT</ACTIONS> If there are any errors or warnings, the tool will display them, allowing you to correct them before submitting your data to EGA. For example, in the following response, it is said that the object we were trying to submit was already existing, and therefore the "success" was "false". <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="receipt.xsl"?> <RECEIPT receiptDate="2023-04-11T15:12:35.609+01:00" submissionFile="submission-EBI-TEST_1681222355609.xml" success="false"> <STUDY alias="Example!_Human Microbiome Project SP56J" status="PRIVATE" holdUntilDate="2023-03-11Z"/> <SUBMISSION alias="SUBMISSION-11-04-2023-15:12:35:576"/> <MESSAGES> <ERROR>In study, alias: "Example!_Human Microbiome Project SP56J". The object being added already exists in the submission account with accession: "ERP127584".</ERROR> <INFO>VALIDATE action has been specified.</INFO> <INFO>Submission has been rolled back.</INFO> <INFO>This submission is a TEST submission and will be discarded within 24 hours</INFO> </MESSAGES> <ACTIONS>VALIDATE</ACTIONS> <ACTIONS>PROTECT</ACTIONS> If the curl command retrieves no response at all, please double check if your username and password are correctly provided. Also notice the "ACTION=..." argument passed to the Curl command. This specifies the action to take during the call to Webin, so we do not need a "Submission" XML just for a validation attempt. See more at submission actions without submission XML. Furthermore, validation of multiple files or objects (e.g. sample, experiment, study…) can be done in a single command by adding more arguments (i.e. '-F'). For example: curl -u <USERNAME>:<PASSWORD> -F "ACTION=VALIDATE" "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" -F "SAMPLE=@sample.xml" -F "DATASET=@dataset.xml" As mentioned above, beside "validate" action in the test environment, you can also validate your metadata by three other methods: "Validate" in the production server. From our example above, you simply need to take the "dev" away from the URL. curl -u <USERNAME>:<PASSWORD> -F "ACTION=VALIDATE" "https://www.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" "Add" in the development server. From our example above, you would simply need to replace the action: from "validate" to "add". Whatever is submitted to this service will be discarded in 24h, so whether something gets submitted or not would not matter in the long run. curl -u <USERNAME>:<PASSWORD> -F "ACTION=ADD" "https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" "Add" in the productionserver. A combination of the previous two methods, which would render this attempt into a submission. This path is just to be taken when you are sure your metadata is compliant and what you want to submit. curl -u <USERNAME>:<PASSWORD> -F "ACTION=ADD" "https://www.ebi.ac.uk/ena/submit/drop-box/submit/" -F "STUDY=@study.xml" What happens after the submission of a dataset XML? Once you have completed the registration of your dataset/s please contact our Helpdesk Team to provide a release date for your study. Please note that all datasets affiliated to unreleased studies are automatically placed on hold until the authorised submitter or DAC contact contact the EGA Helpdesk for the study to be released. We strongly advise you not to delete your data until EGA Helpdesk confirms that your data has been successfully archived.

  Documentation submission/metadata/submission/programmatic-submission-xml

• Papua New Guinean Genome Diversity Commitee

  Dac EGAC00001002177

• Papua New Guinean Y chromosome Diversity Commitee

  Dac EGAC00001002531