464 results for "GitHub+primate+acoustics+baboon+Zenodo+10.5281+Gui*"

in 120.42 milliseconds.

• What is a DAC?

  What is a DAC? Given the complexity, scale, and diversity of global submitters and studies, the EGA operates a distributed data access model in which requests are made to the data controller, not to the EGA. The European Commission defines a data controller, in the General Data Protection Regulation (GDPR), as the person that determines the purposes for which and the means by which personal data is processed. A Data Access Committees, commonly referred to as DACs, comprise on or more individuals (or data controllers) that review data access requests and make decisions on who can access personally identifiable genetic, phenotypic, and clinical data deposited at the EGA. Therefore, the members of a DAC should be individuals who have the authority to approve data access requests. The animation describes how you can authorise access to your sensitive data with the help of Data Access Committee and Authorisation tool. Acknowledgement to CSC - IT Center for Finland, Elixir Finland, Elixir Europe. Frequently Asked Questions How can I create a Data Access Committee? How can I create a Data Access Committee? The members of a DAC can come from different areas of expertise, such as data management, data analysis, information technology, legal and compliance, subject matter experts, privacy and security, and representatives from the organisations or individuals that provide data to the DAC. The specific members of a DAC can vary depending on the needs of the organisation and the type of data being managed. The EGA strongly suggests checking with your organisation to align with its regulations How should a DAC be named? The chosen name must be informative to the applicant.  For example, internal identifiers, such as grant numbers, should not be used. Individual PI names should also not be used. DAC's are often named after the organisation or department of the data source.  Browse the full list of DAC names currently in the EGA. How can I become an EGA DAC contact? To register a DAC at the EGA you must create first as an EGA user. Once your EGA user has been approved by the Helpdesk team, you will be able to log in to the DAC Portal. How can I register a DAC? To register a DAC, follow the DAC Portal instructions.  You will be required to provide a DAC name, name of the individual(s) that make up your DAC and contact details for your DAC including your Institutional email(s). Wherever possible, the DAC should make sure that all points of contact are readily available and able to answer any initial data requests/queries in < 2 weeks.  Once your DAC is registered, you will have to wait upon the validation from our Helpdesk team. As soon as all the validations have been completed, your DAC will be activated. Alternatively, you can also establish a DAC at the EGA during a programmatic submission through Webin API. Which are the possible roles of a DAC contact? There are two possible roles for DAC contacts: member and admin. An admin has additional privileges compared to a member: An “admin” can manage data requests, create and edit policies, edit the content of the DAC, add or remove contacts, and decide the role of each contact. A “member” can manage data requests and create policies. However, a member does not have permission to modify DAC details, edit information from policies where they are not admins, or add/remove contacts. There is no limit to the number of admins in a DAC, and each admin is responsible for deciding who should have editing privileges. This allows for a more decentralised and democratic approach to managing the DAC. How can I modify the information of a DAC? To modify a DAC, follow the instructions here. Keep in mind that only DAC contact with an admin role can modify the information of a DAC. If your DAC was registered before the lauch of the DAC Portal, and its ID is EGAC0 (not EGAC5), you must use the programmatic submission to modify it. Please, do not hesitate to contact our Helpdesk team if you need help with this! To prevent potential data breaches and ensure adherence to GDPR regulations, it is essential that the European Genome-Phenome Archive (EGA) is informed via the Helpdesk team of any changes to the Data Access Committee (DAC). This should be done in addition to any changes being made on the DAC portal. Data Controllers (as per the definition in the DPA) are also responsible for notifying the previous DAC of any modifications. Without proper notification, changes might not be automatically updated in our system, leading to the risk of incorrect permissions being applied and potential data access issues. Therefore, it is imperative that all Data Controllers follow this protocol to maintain data integrity and security. What’s the link between DAC, policy and dataset? A dataset is linked to one single policy. At the same time, one policy has a one to one relationship with a DAC. In this example, you can see that in this dataset page, we are only showing the information of one DAC (1 dataset - 1 DAC). However, the ratio of objects does not work the same in the other direction. One DAC can own multiple policy objects. And each policy object can be reused in several datasets. Thus, one DAC can manage one or more datasets. In this example, you can see that in this DAC page, we are showing all the datasets that are managed by one DAC (1 DAC - >400 datasets). EGA Data Access Committee Best Practices Which are the EGA DAC best practices? Refer to DAC Best Practices What happens if a DAC member changes institutions? EGA is committed to the protection and ownership of the data stored in our systems. We respect the institution's ownership of the data, and as such, if a DAC member changes institutions, the ownership of the data will not be transferred to the new institution. Therefore, before changing institutions, we request that the DAC contact add a new member who will replace them once they no longer work at the institution. This ensures that the data remains protected and is accessible to authorised personnel at the institution. To prevent potential data breaches and ensure adherence to GDPR regulations, it is essential that the European Genome-Phenome Archive (EGA) is informed via the Helpdesk team of any changes to the Data Access Committee (DAC). This should be done in addition to any changes being made on the DAC portal. Data Controllers (as per the definition in the DPA) are also responsible for notifying the previous DAC of any modifications. Without proper notification, changes might not be automatically updated in our system, leading to the risk of incorrect permissions being applied and potential data access issues. Therefore, it is imperative that all Data Controllers follow this protocol to maintain data integrity and security. What happens if EGA detects an unresponsive DAC? EGA defines an unresponsive DAC as a DAC with one or more contacts who do not respond to data access requests. EGA has procedures in place to identify these types of DACs, escalate the issue, and attempt to reassign the DAC to a responsive contact. This is a crucial step in ensuring that data can be accessed and utilised by researchers. If EGA identifies an unresponsive DAC, the organisation will first try to resolve the issue by escalating it to the appropriate parties. This may involve attempting to reassign the DAC to a more responsive contact. Unfortunately, in situations where we cannot reassign the DAC, the dataset will be withdrawn from the public website and the files will be removed from our system. If an EGA ID is referenced in a publication, the EGA will take extra steps to ensure that the public is made aware of the data's unavailability. I don't want to receive an email notification for pending requests. How can I do that? If you are an EGA DAC with pending requests, you will always receive emails for new data access requests. However, the EGA understands that a request resolution can take some time, for this reason, if you add a comment (make sure you save it by clicking the APPLY button!) we will filter those requests at the time of sending the notification! How can I manage data access requests? What documentation does the DAC need to provide? Each dataset that is submitted to the EGA must be linked to a policy object. The policy is a Data Access Agreement (DAA), which defines the terms and conditions of using the dataset, such as how the data files should be stored once downloaded or details of publication embargoes that should be observed by the approved user. As part of the Data Access Agreement, information regarding the application can be captured to help inform the DAC when making its decision. For example, requestors could be asked to provide a proposed title for their research and a proposal of how the data will be used. By asking for provision of such information the DAC can be assured that the requestor fully understands any consents associated with the data. It is important that accounts created at the EGA, are created solely for those individuals that will be downloading the data from the EGA. As part of the data access request, we strongly encourage you to identify individuals that will need an account at the EGA in order to prevent sharing of login details, which is strictly prohibited under EGA user account policy. Such information can easily be captured in the DAA. NOTICE The data access agreement template below is provided for guidance only and should be adapted as you see fit to suit your own purpose. In the interest of promoting data sharing, we suggest that if an agreement cannot be met around clause 19 in this example that both parties should agree to remain silent, and that the clause should be removed from the agreement. Example DAA How can the DAC provide the DAA? The DAC should provide their own DAA when registering a policy. Data requestors will download this document and should fill it in and send it back to the DAC. Data access decisions should be based on such documentation. The DAA can be downloaded through the request data webpage. Once it has been filled in, the signed copy of the DAA can be uploaded back to the request data webpage and sent to the DAC for review. How can I grant access to the data? Once you receive a data access request, you can login to the DAC Portal. In this portal you will see all your pending requests and will be able to grant or decline access to the requestors. I am a member of the Data Access Committee. Could I approve somebody else to deal with the requests on my behalf? If you want to delegate data access decisions to someone else, make sure that the individual's account is officially registered as a member of the DAC. Remember that a DAC contact with an "admin" role can always add new members to an existing DAC, remove members, and modify contact details through the DAC Portal. Can I automatise the process of managing data access requests? The answer is yes! You can use a programmatic approach using our DAC API! Check out the DAC API specification! Data Breach What should a DAC do if they suspect a breach? If a DAC suspects a data breach of one or more of their datasets, they should immediately contact the EGA Helpdesk team at this link. The DAC must provide the following information when contacting the EGA Helpdesk team: A list of affected datasets An estimated date of the data breach (or interval of dates) A list of unauthorised users who accessed the data (if available). Otherwise, they can provide a list of authorised users for the affected datasets Any observations they would like to raise to the EGA team Once the DAC has contacted the EGA team, we will respond within 48 hours (please allow some leeway during peak times) and activate our data breach protocol. What can I expect from the EGA if they detect a breach? Once the EGA determines that a security incident has occurred, we will notify all DAC members that a data breach has been detected, and take steps to contain the incident. Containment approaches may include: Revoking a data provider's access to the EGA resources, such as by changing passwords. Removing affected EGA datasets from distribution, such as by withdrawing a dataset. Disabling certain functions or services, such as the EGA ingestion pipeline. Shutting down the system or disconnecting it from the network. After the incident has been contained, the EGA will determine whether it is necessary to eradicate components related to the incident. Finally, the EGA will enable recovery of the service to normal operation and confirm that all services are functioning normally.

  Documentation access/data-access-committee/what-is-dac

• How to request data

  Request Data Access If you're looking to access data from the EGA for your research, you are in the right documentation page! In this page you will find a step by step guide on how to request data archived at the EGA, as well as some frequently asked questions you might find useful. When requesting data, you'll need to provide information about your research project and intended use of the data. Keep in mind that your data access request may require additional approvals or agreements depending on the datasets and data providers involved. Step by step guide Register yourself as an EGA user. Validate your account. A validation link will be sent to your email to activate your EGA user account. Log into the EGA page. Search the dataset of interest from the dataset catalogue or search bar. Click on “Request access” Add a comment requesting these files and send your request Your request is successfully registered, and the DAC will receive a notification about your request In addition to submitting your request, you should also contact the DAC directly via e-mail. You can access the DAC's contact information from the relevant dataset page. Frequently Asked Questions (FAQ) Which is the role of the EGA in the data access process? The European Genome-phenome Archive (EGA) facilitates the secure distribution under controlled access of personally identifiable genetic and phenotypic data. Data Access Committee (DAC) is a body of one or more individuals who are responsible for data release to external requestors, based on participant consent and/or National Research Ethics terms. How do I request access to multiple datasets? To request access for multiple datasets managed by the same Data Access Committee (DAC), please start by visiting the DAC catalogue to locate the DAC of your choice. Once you've found the relevant DAC, proceed to its dedicated page where you'll find a "Request access" button. After logging in to the EGA website, you'll be presented with a comprehensive list of datasets under the chosen DAC. Select the datasets you're interested in, and then compose a message indicating your access request. Click the "Send" button to submit your request, which will include the chosen datasets. Please be sure to review the data usage policies for the requested datasets, as policies can vary among datasets managed by the same DAC. I am a co-applicant. Will I be given a joint access account? Each applicant needs to create an EGA account themselves, with unique login details. User log-in details are for individual use only and should never be shared, under any circumstances. EGA services are only provided to EGA account holders. Therefore, the co-applicant will send a formal request to the DAC. What happens if a co-applicant leaves or a new member needs to be added to an existing DAC application? The appropriate Data Access Committee (DAC) should be contacted and informed about the required changes so that the application can be updated. I have changed my place of work since my EGA account was created. Can I still have access to data? You must contact the relevant Data Access Committee (DAC). The DAC did not reply to my data access request. How should I proceed? There is no estimated time for DAC response. Please allow up to four weeks for a response. In the meantime, please send them a weekly reminder of the status of your request by emailing them directly.If the DAC becomes unresponsive, meaning that they fail to communicate with a requestor within the specified time frame, please contact the EGA Helpdesk team. I have an account with the EGA and would like access to further datasets. How should I proceed? You need to reapply to the relevant Data Access Committee (DAC). Once the DAC has approved your application your account will be updated and an email will be sent to notify you of a change made to your account. How do I download files of the datasets I have been granted access to? The preferred method for data download and decryption is the EGA Secure Download Client. Please carefully review the provided documentation. How do I download metadata? Metadata is retrieved from the EGA webpage; you will not need to use any download client. Be informed that you will only be able to download metadata of your approved datasets. For such download, you should first log in the EGA webpage with your credentials. Then browse the metadata's dataset you have gained access and you will find a metadata download button. You may also want to have a look our metadata schema. I have forgotten or misplaced my user account password; how do I get a new one? Click on “Log in”, located in the top right corner of the EGA homepage. Then click “Forgot password”, type in your registered institutional email address, and instructions for resetting your password will be sent to you. Please check both regular and junk/spam folders. I was sent a link to set a password for my download account, but it seems like the link has expired. What should I do? The procedure is the same as the one for a forgotten password, explained in the previous question.

  Documentation access/request-data/how-to-request-data

• EGA account management

  How to manage your EGA account Welcome to the documentation page for creating an EGA user. The European Genome-Phenome Archive (EGA) is a platform for archiving, managing, and sharing human genomic and phenotypic data. To submit, access, and manage requests for the data hosted on the EGA platform, users are required to register for an account. This documentation page provides step-by-step instructions for creating an EGA user account, as well as information on the different types of EGA user accounts and their associated privileges. You can check the EGA data flow for a better understanding. Manage EGA account step-by-step guide Contact Information Public Keys Password Authenticator Federated Identity Sessions Request an EGA account step-by-step guide Fill in this form.You cannot use a generic email account (e.g., Gmail). Make sure you use your institutional email account.Your email is by default your username.The criteria for a strong password is: It should contain between 8 and 32 charactersIt should contain at least one uppercase letterIt should contain at least one numberIt should contain at least one special characterYou will need to wait for the Helpdesk team to check your identity. Please allow two business days for this step.After the Helpdesk validation step, you will receive an email with a link to validate your institutional email.Click the link to verify your email. It will redirect you to the EGA website.As soon as your account has been validated, you will be able to start adding more information to your EGA profile. For that, you first need to log in using your credentials from step 1. Manage EGA account step-by-step guide As a logged-in user, go to your profile by clicking on Welcome > My Profile. To modify it, click the Edit button.You will see the following options: Contact Information In this section, you can modify the personal information of your EGA account. Public Keys Adding a public key is necessary for downloading data via Live Outbox distribution. If you don’t know how to generate your key, you can follow the instructions available on How do I create a key? Password In the Password tab, you can change your password anytime to ensure the security of your account. However, if you receive an email notifying you that your password was modified and you did not initiate the change, it could be a sign that your account has been compromised. In this case, you should immediately contact our Helpdesk team to report the issue and receive assistance in restoring your account security. Authenticator You can set your two-factor authentication (2FA) system for your EGA account as an identity and access management security method that requires two forms of identification to access resources and data. To obtain your code, please follow the instructions below, also available in the Authenticator tab in your profile. Sessions The Sessions tab displays a list of all devices where you are currently logged in as an EGA user. This includes any open sessions you have on your computer, phone, or other devices. By reviewing the list of active sessions, you can monitor the devices that have access to your account and ensure that there are no unauthorised logins. If you notice any suspicious activity, you can close the sessions from this page to immediately terminate access to your account from the unauthorised device.

  Documentation how-to-manage-your-account

• Submitting array based metadata

  Submitting array based metadata For further information please check our Submission FAQs, submission quickguide as well as submission terms! The submission metadata required for Array-based submission must be submitted using EGA programmatic submission and by completing the Array-based format (AF) template. The guidelines for this workflow are described on this page. Please notice that all files should be encrypted and uploaded prior to the processing of your EGA Array-based-Format (AF) template. You can request a legacy EGA submission account (ega-box-XXXX) by populating this form. Please, allow two business days for our Helpdesk team to contact you after populating this form. Please notice that all files should be encrypted and uploaded prior to the processing of your EGA-Array-based-Format (AF) template. Metadata Model Registering Metadata Use the EGA programmatic submission to register your Study, Samples, Data Access Committee (DAC) and Policy. This online interface enables you to create new and edit existing submissions. Do not use the Submitter Portal to register the metadata objects for array submissions. Please, go to the EGA programmatic submission. Registering Study Go to the EGA programmatic submission Submit an study XML Take a note of your study accession number (EGASXXXXXXXXXXX) To use the study accession number in a publication, we suggest the following format: "Sequence data has been deposited at the European Genome-phenome Archive (EGA), which is hosted by the EBI and the CRG, under accession number EGASXXXXXXXXXXX. Further information about EGA can be found on https://ega-archive.org "The European Genome-phenome Archive of human data consented for biomedical research"(https://doi.org/10.1093/nar/gkab1059 ). Registering Samples Go to the EGA programmatic submission Submit a sample XML Save the list of accession numbers. These will be needed for the AF template. Registering Data Access Committee Further information on the role of your DAC. Go to the EGA programmatic submission Submit a DAC XML Take a note of the DAC accession number (EGACXXXXXXXXXXX) Registering Policy Your Data Access Policy provides the terms and conditions of data use. This is also referred to as the Data Access Agreement (DAA). Completion of a DAA by the applicant/s should form part of the application process to the Data Access Committee (DAA) Go to the EGA programmatic submission Submit a Policy XML Take a note of the Policy accession number (EGAPXXXXXXXXXXX) Complete the Array-based format (AF) spreadsheet Once you have completed the registration of your Study, DAC and Policy using the programmatix submission, you must then complete and return the AF spreadsheet The AF spreadsheet consists on four components: Do not use EGA IDs registered using the Submitter Portal. You can easily identify objects registered in the Submitter Portal by their EGA ID pattern: EGA[A-Z]5{10 more digits} (e.g. for a sample EGAN50000002506). For populating an AF spreadsheet, please exclusively use EGA IDs following this specific pattern: EGA[A-Z]0{10 more digits} (e.g. for a sample EGAN00001691542). Tab1) Webin accessions : Provide the accession numbers for your study, DAC and policy. Please, also add your ega-box number. Tab2) Sample & phenotypes: Please, leave this tab blank. Tab3) Dataset: Describe the dataset to be created Tab4) Data files: Define how your data is going to be organised into datasets and packets for distribution (linkage between samples and files). Should further assistance be required after going through the guide below; please do not hesitate to contact the EGA helpdesk Once the AF spreadsheet is populated, please send it to our EGA helpdesk for further validation. AF spreadsheet Should your submission require multiple DAC's or policies, use ' ; ' to separate the accession numbers. Accessions AF spreadsheet: Samples & phenotypes Samples and phenotypes AF spreadsheet: Datasets We suggest that each dataset consists of a common set of data. The example below consists of two datasets, grouped according to shared data type, technology and by case/control. We also like to capture the number of unique samples that make up the dataset and the Data Access Committee (DAC) responsible for providing the named dataset and their policy (EGAP). Datasets AF spreadsheet: Data files What follows is an example of how to map your samples to the array based files added to your upload account (4th tab). Data Files Please, find below some practical examples on how to register the linkage between samples-files Case 1) 1 sample or list of samples in different datasets: Data Files In case you have a list of samples that belong to different datasets, please, repeat the samples accession number/s in the first column and link the sample to the corresponding dataset each time (each row). Each row is one linkage between sample-file-dataset. Case 2) 1 sample links to several files: Data Files In order to add multiple files to one sample you MUST use “ ; “ between filenames. Example: file1.gpg;file2.gpg;file3.gpg In case that you want to add an extra file to the sample (phenotype or .Rdata), please use “Additional files” column. Important note: You MUST upload the encrypted and unencrypted md5sum values of all files uploaded to your submission account using the filename nomenclature (file.gpg, file.md5,file.md5.gpg). Your submission will not be processed without md5values supplied for all files in the CORRECT format. What happens after the submission of a dataset? All datasets affiliated to unreleased studies are automatically placed on hold until the authorised submitted or DAC contact instructs our EGA helpdesk for the study to be released. Finally, your data is archived within our databases and prepared for encrypted distribution upon the request of permitted EGA account holders. We strongly advise you NOT to delete your data until we confirm that your data has been successfully archived.

  Documentation submission/metadata/submission/array

• Submitter Portal API

  Submitter Portal API The metadata submission process can be difficult and time-consuming. For this reason, the EGA has developed the Submitter Portal, a tool that was created to offer a simplified and user-friendly method of registering metadata. Our portal provides features that are intended to make the input of metadata easier, ensuring that your data is registered correctly and effectively. Our page is divided into logical sections and includes a helpful video instruction to further assist you as you complete the submission process. With the Submitter Portal, you can rest assured that the submission of your data is in good hands. For those that want a more flexible and automated approach, we also provide a programmatic approach using the Submitter Portal AP in addition to the user interface of the Submitter Portal. With the help of our API, you can quickly include the submission of metadata into your own workflow for a more effective and individualised experience. Previous steps Create your EGA account To submit data you first need to create your EGA user. Then, once your account has been verified, you will have to request a submitter role and sign the EGA Data Processing Agreement (DPA). When the DPA is signed, you must send it to EGA Helpdesk for further validation. Please, note that if you already have an EGA account, or you have an ega-box (submission account), you can skip this step.  Upload your files Please note that all your files must be encrypted using the Crypt4GH tool before upload them.As soon as you are assigned with a submitter role, you will be able to connect to the EGA inbox and upload your files. Understand the EGA metadata schema It's crucial to comprehend the EGA metadata schema, a set of rules that specify how data is organised, described, and shared inside the EGA, in order to get the most out of this resource. Learn all about EGA metadata schema! Register your DAC and policy There are two objects in the EGA metadata schema that are registered in a separate portal. All DACs and policy objects are registered using the DAC Portal, a tool developed by EGA to help data controllers manage their data stored at the EGA. You can find the relevant information in the DAC Portal Guide. Programmatic submission All the calls to the API need to be Authenticated. We use the OpenID Connect protocol. API Usage Flow: Click here to check the API usage flow schema Obtain Access and Refresh Token: The first step to using the API is to obtain an access and refresh token. These tokens are required for authentication and authorisation of API requests. To obtain these tokens, you need to log in using your EGA credentials. Example: curl "https://idp.ega-archive.org/realms/EGA/protocol/openid-connect/token" \ -d "grant_type=password" \ -d "client_id=sp-api" \ -d "username=your-username" \ --data-urlencode "password=your-password" Use Access Token in API Calls: Once you have obtained the access token, you must use it in all the calls to the API. The access token is valid for a limited time period. When it expires, you will need to use the refresh token to obtain a new access token. Create Submission Object: After authentication, you can start creating a submission object. This object will be used to store all the metadata objects and files related to the submission. Example: curl "https://submission.ega-archive.org/api/submissions" \ -H "Authorization: Bearer your-access-token" \ -H "Content-Type: application/json" \ -d "{ \"title\":\"My submission title\", \"description\":\"My submission description\" }" Create EGA Metadata Objects: Within the submission object, you can create other metadata objects required for the submission. These objects may include information about the submitter, study, sample, experiment, and run. Example of study creation: curl "https://submission.ega-archive.org/api/submissions/{submisison-id}/studies" \ -H "Authorization: Bearer your-access-token" \ -H "Content-Type: application/json" \ -d "{ \"title\":\"My study title\", \"description\":\"My study description\", \"study_type\":\"Metagenomics\" }" If you would like to reuse already registered objects in new submissions, bear in mind that they must have an accession which you must use to reference them. You cannot reuse objects that have not been finalised and accepted yet.For example, when creating a new run which is reusing a previously submitted experiment, instead of using experiment_provisional_id, you must use experiment_accession_id. Enums There are some fields that only accept specific values, i.e. study_type. The API provides several enumerations to list these values. You can list all the enumerations, or a specific one, for example, study_types. Obtain IDs of files You can list the files available in your inbox and filter by a prefix by adding the prefix param, i.e. &prefix=/my_folder/abc Example: curl "https://submission.ega-archive.org/api/files?status=inbox&prefix=/my_folder/abc" \ -H "Authorization: Bearer your-access-token" \ -H "Content-Type: application/json" To improve performance and manageability, we have limited the number of files that the Submitter Portal API can return in a response. If a given call exceeds this limit, user will be asked to narrow down the search by using the query parameters already available in this endpoint. Finalise Submission: Once all the metadata objects are created and linked to the submission, you can finalise the submission. Finalising the submission sends it to the Helpdesk team for review. Please, note that all files linked to the submission must be ingested, and all objects created in a submission must be linked before you can finalise the submission. API Reference: For a full API reference, please check our specification documentation.

  Documentation submission/metadata/submission/sequencing-phenotype/submitter-portal-api

• Submitter Portal

  Submitter Portal The Submitter Portal does not support array submissions! Please refer to the array documentationAny new objects registered in the Submitter Portal will not be available for use in an array submission For further information please check our Submission FAQs, submission quickguide as well as submission terms! The metadata submission process can be difficult and time-consuming. For this reason, the EGA has developed the Submitter Portal, a tool that was created to offer a simplified and user-friendly method of registering metadata. Our portal provides features that are intended to make the input of metadata easier, ensuring that your data is registered correctly and effectively. Our page is divided into logical sections and includes a helpful video instruction to further assist you as you complete the submission process. With the Submitter Portal, you can rest assured that the submission of your data is in good hands. Submitter Portal Index Previous steps Access to Submitter Portal Start a Submission Take The Tour Tutorial Previous steps Create your EGA account To submit data, you first need to create your EGA user. Then, once your account has been verified by the Help Desk team, you will be able to request a submitter role and send the EGA Data Processing Agreement (DPA). In the case you already have an EGA account, or an ega-box (submission account), you can skip this step. Upload your files Please note that all your files must be encrypted using the Crypt4GH tool before upload them.As soon as you are assigned with a submitter role and added your public key to your profile, you will be able to connect to the EGA inbox and upload your files. Understand the EGA metadata schema It's crucial to comprehend the EGA metadata schema, a set of rules that specify how data is organised, described, and shared inside the EGA, in order to get the most out of this resource. Learn all about EGA metadata schema! Register your DAC and policy There are two objects in the EGA metadata schema that are registered in a separate portal. All DACs and policy objects are registered using the DAC Portal, a tool developed by EGA to help data controllers manage their data stored at the EGA. You can find relevant information in the DAC Portal Guide. Access to Submitter Portal 1. Request Submitter Role Once you have created your EGA user, request a Submitter Role and complete relevant information such as submission type and size. 2. Signature of the Data Processing Agreement The Data Processing Agreement (DPA) signed by CRG and EMBL-EBI (Joint Data Processors for the EGA) can be downloaded to the submitter for its signature. Further information related to the DPA can be in the Privacy Notice section. The DPA should be signed by a legal representative of the submitter with power of attorney/appointment duly authorised to sign contracts on behalf of the institution. The submitter will return a fully-executed copy of the DPA to us. Please note that due to the high volume of submissions to EGA the DPA is non-negotiable. Once you have completed all the information and attached the DPA signed, send your Submitter request and the EGA Helpdesk Team will validate it and send a confirmation to start submitting your metadata. Start a Submission The steps below will show you how to start a submission and are accompanied with a tutorial and the Take the tour of Submitter Portal. 1. Login Go to the Submitter Portal interface and log in using your EGA account credentials. 2. Create a submission The Submitter Portal interface will show you any already existing open submission. To start a new submission click on “Create a Submission”.In this phase you have the option to add any collaborator(s) as well as their permissions (“Read only” or “Read & Write”). If you want to invite all the collaborators from a previous submission, you can add them by selecting said submission.In the event that the submission has been closed, you will need to reopen it. To do this, you will need to modify one of the objects in the submission, for example, copy and paste the title. Once the submission is open, you can add a collaborator. Further information can be found in the Submission Tour 3. Create Study Click on add Study and complete the form with the relevant information. The fields with an asterisk (*) are mandatory. Further information can be found in the Submission Tour4. Create Sample Sample registration can be done either individually or in batch by using a template. Batch upload: download the template and complete it with information about the sequencing samples. Upload the samples.csv and the sample will be created.Add single sample: complete the mandatory fields with the metadata information of your sample.  Further information can be found in the Submission Tour5. Create Experiment Register an experiment with ​information about the sequencing methods, protocols and machines. Experiments generate the connection between samples and study. Further information can be found in the Submission Tour6. Create Run Samples, experiments and files are linked through runs. To register a run, you will be asked to select the appropriate experiment and the file format. Run registration can be done either individually or in batch by using a template. Batch upload: download the template and complete it with the sample - file linkage. Upload the run.csv and the runs will register.As the INBOX is a directory, bear in mind to include the file path (/). Example: /filename.c4gh. Add single run: select the file format, sample and experiment from the drop box. For re-using submitted samples, please use the EGA accession ID (EGANXXX). Please note that files must be uploaded in your inbox before linking your files and samples. Further information can be found in the Submission Tour7. Create Analysis Analysis should be only used for BAM/BAI or CRAM/CRAI pair, VCF and phenotype linkage to samples. The analysis is an EGA specific metadata object that links samples to files. This object also stores some metadata about your experiments, such as the experiment type, genome reference, or the platform used. **If only BAM or CRAM alignment files are submitted but not the original unaligned FASTQ files, then please make sure that the BAM or CRAM files also contain the unaligned reads. This is critical to enable primary re-analysis and re-alignment of the dataset using new tools or future genome assemblies.** Start by selecting the samples to be linked to the file, and then populate the required attribute fields. Please be aware of the existence of mandatory fields that must be populated.When populating the chromosome field (mandatory), please select the chromosomes from the checkbox. Please take in consideration that EGA allows for the re-use of any registered metadata. Therefore, a previously registered study or samples can be referenced for the analysis in the current data submission.Further information can be found in the Submission Tour 8. Create Dataset Dataset contains the collection of runs/analysis data files subjected to controlled access by being associated with a Policy. Complete the mandatory fields such as title, description, type and policy. Please, take in consideration that DAC and Policy must be registered in advance in the DAC Portal. Similar to other metadata objects, you can re-use previous runs or analyses, using their EGA accessions IDs (runs - EGAR, analyses - EGAZ). Further information can be found in the Submission Tour 9. Finalise submission Once the files are ingested please delete them from your SFTP INBOX before finalising your submission. This will prevent the files from reappearing in your Submitter Portal Inbox. Once you have completed your submission, click on the Finalise button, and schedule an estimated release date. Afterwards, your submission will be sent to Helpdesk for approval. Please note that once the Helpdesk has approved your submission, you will receive the identifiers for your data. See the Frequently Asked Questions (FAQ) for more information. 10. Update closed submission If you want to append more file to a closed submission, you will need to re-open the submission where the dataset was registered. Find the Submission: go to the Submitter Portal Menu -> Datasets section and find the submission id (EGA0XXX).  Reopen the submission where the dataset is located: to reopen a submission, you must edit any object within the submission. For example, you can copy and paste the title. Take The Tour The Submitter Portal Take The Tour is a visual guidance of all the steps described previously about how to start a submission. Tutorial In this tutorial we will show you how to use the Submitter Portal to register your metadata. Please note the video focuses on the run-based submission (for raw files - fastq - and aligned data - BAM/CRAM) and analysis-based submission (for your BAM/BAI pairs, variation -VCF - and phenotype files).

  Documentation submission/metadata/submission/sequencing-phenotype/submitter-portal

• EGA submission account terms

  Submission account terms Each EGA submission account has a dedicated submission box. To ensure continued service to all submitters EGA submission boxes should not exceed 8Tb in size, and cannot exceed 12Tb. Those that are near to or exceeding this value will be monitored on a weekly basis by the helpdesk team and the submitter will be contacted and guidance provided accordingly on completing their metadata registration. You will be able to register your metadata associated with the files in your submission account via EGA Submitter Portal, Webin or the EGA REST API. EGA also requests that metadata is submitted within 60 days of upload data to a submission box. When we detect files that are over 60 days old in a submission box we will notify the submitter requesting that metadata be submitted or the files removed from the submission box. It is important to note that data is not archived at the EGA until the metadata describing the files has been associated and the archival process has completed. It is only when associated metadata has been submitted that the archival process into EGA is fully triggered. EGA removes files from the submission box once they have been successfully archived, so no action is required by a submitter to remove successfully archived files. Files over 90 days old will be deleted from the submission box unless EGA Helpdesk has been contacted and confirmed an exemption with the submitter.

  Documentation submission/terms

• How to upload Crypt4GH files

  Uploading files Users who hold an ega-box-XXX account can upload files using either INBOX or FTP. Users who have a Submitter role associated with their email will only be able to upload files using INBOX. Before uploading your files, please make sure that any files that will be uploaded to EGA do not use special characters in their naming convention, such as # ? ( ) [ ] / \ = + < > : ; " ' , * ^ | &. This can cause issues with the archiving process, leading to problems for end users. The EGA is a shared, public service with limited storage. To manage the available resources, we enforce a limit of 10TB per submission account at any one time. If you exceed this limit, a “permission denied” message will be displayed. This will prevent you from uploading more files, but connecting to your inbox.For submissions larger than 10TB, please perform uploads in 10TB batches: register all the metadata and then finalise the submission. Upload the next batch of files and repeat the same metadata registration and finalisation process until you have completed the file upload. Further information can be found in the SP documentation. INBOX FTP The INBOX is only compatible with files encrypted using the Crypt4gh tool Before uploading If you are not a registered EGA user, you will first need an EGA user account. Please note that it may take a few days for your account to be activated, as it needs to be vouched for by the EGA Helpdesk. Once your account is validated, you will be able to request a submitter role. [Optional] Meanwhile, you can create and add your public key to your EGA account profile. This option is not available for old submission accounts (e.g., ega-box-NNN). As soon as you have been granted a submitter role, you will be able to connect with your username and password to the EGA inbox using the SFTP protocol. If you have also registered a public key in your profile, you can also connect using this key. To upload files to your account, you can use the graphical user interface (GUI) or the command line. Graphical User Interface (GUI)We recommend using FileZilla, a free, open-source FTP client. However, you can use any other GUI that allows connecting over the SFTP protocol. For FileZilla as your GUI, follow these steps to upload files: Create a new connection in Site Manager (File > Site Manager) and select the following options (Figure 1): Protocol: SFTP - SSH File Transfer ProtocolHost: __EGA_INBOX_DOMAIN__Logon Type: Key fileUser: your EGA usernameKey file: Path/to/your/private_keyFigure 1: Process of establishing a new connection to __EGA_INBOX_DOMAIN__ using a key file as the logon method in FileZilla. The figure showcases the FileZilla version 3.52.2 operating on IOS v11.2.3. By following the depicted steps, users can create a secure and efficient connection to the inbox, ensuring seamless data transfers.Click Connect, and you will log in remotely to your home directory. You can think of this folder as a storage "in the EGA cloud" in which you will add your files for the EGA. The uploading area has three folders:To-encrypt: Files uploaded in this folder will be encrypted automatically on the fly.Encrypted: Files uploaded in this folder must already be encrypted with Crypt4gh. Upload your files here if your connection is unstable or you have problems completing the upload into-encrypt.Etc: This folder contains two files that allow the server to show you your username and group instead of some internal numbers. Please do not upload files here; otherwise, you will obtain a permission denied error. Find the files you want to upload by browsing your local storage (left side of your screen in FileZilla). Select all the files you want to upload, then right-click on them and select Upload (Figure 2). Figure 2: Step-by-step process of manually uploading files to __EGA_INBOX_DOMAIN__ using FileZilla, with FileZilla version 3.52.2 operating on IOS v11.2.3. The figure demonstrates how users can transfer data from their local storage to the "EGA cloud" by following the depicted steps Please note that regardless of which folder you upload your files in, both folders (to-encrypt, encrypted) will point to the same path (/) (Figure 3). Therefore, you will see your files in both folders. Figure 3: Both folders, to-encrypt and encrypted, point to the same path (/)" If your connection is unstable, please encrypt your files first using Crypt4gh. Then upload them to the ‘encrypted’ folder. The example above shows how to connect to __EGA_INBOX_DOMAIN__ using the private key. However, if you prefer to log in using your credentials, you can do so. Please go to the Frequently Asked Questions (FAQs)  for more information. SFTP command line To upload files securely to your private area of the EGA, you can use SFTP(Secure File Transfer Protocol) with your favorite FTP client. Here's what you need to know to get started: Connect to the target host __EGA_INBOX_DOMAIN__. This is the new hostname for the EGA SFTP service. Log in with your EGA username and key files (or password). Upload files to your private EGA inbox to ensure that only you can access the files. By following these steps, you can securely upload your files to the EGA for safe storage and sharing. Using the SFTP command line client in Linux/Unix Open a terminal and type sftp username@hostnameEnter your EGA passwordTo see a list of available SFTP commands, type helpsftp> put – Upload filesftp> get – Download filesftp> cd path – Change remote directory to ‘path’sftp> pwd – Display remote working directorysftp> lcd path – Change the local directory to ‘path’sftp> lpwd – Display local working directorysftp> ls – Display the contents of the remote working directorysftp> lls – Display the contents of the local working directoryType the "put" command to upload files. For example: put *.bamUse the bye command to close the connection (SFTP session). After uploading- Once you have uploaded files to the inbox, please bear in mind that the checksum needs to be calculated, which can take up to two days. You will only be able to link your files to a run/analysis once the encrypted checksum has been calculated.- When linking your files to the 'Run' or 'Analysis', ensure that the file name matches the file path '/name' in the INBOX folder.- Please delete the files from your SFTP INBOX after all the runs/analyses have been registered and files are ingested (SP > Files > Files ingested). This will clear your inbox space an allow you to upload more files. This will also prevent the files from reappearing in your Submitter Portal inbox. Frequently Asked Questions Specific to the inbox What username should I use to log in to my inbox? The authentication process for logging in to the EGA website, as well as accessing your inbox and outbox, requires the use of your username.  If you have forgotten your registered username, please contact our Helpdesk team for assistance. How are checksums calculated in your inbox? If you encrypt the file beforehand and upload it to the "encrypted" folder, the unencrypted checksum will not be calculated until the file is ingested (i.e., until it is used in a run/analysis). If the file is uploaded to the "to-encrypt" folder, then both checksums are calculated.Please bear in mind that after files have been uploaded to the inbox, the checksum must be calculated, which can take from a few hours to two days.  Specific to using keys to authenticate Can I access one EGA account from different devices? Yes, you can access your account from different devices by linking several public keys to your EGA account. Each device can generate a unique public-private key pair, and the corresponding public keys can be linked to the same account. This way, you can use different public keys on different devices and still have access to the same account and data. I have several keys and I don't remember which one is which When generating SSH keys, it's a good practice to add a comment using the -C flag. This will allow you to add a descriptive tag to your key, making it easier to identify later on. Here's an example command that generates an SSH key with a comment: ssh-keygen -t ed25519 -C work-pass In this example, we're generating an ed25519 SSH key with the comment work-pass. Once you have multiple keys with different comments, you can use the comments to easily identify each key. To view the comments for your existing SSH keys, you can use the following command: ssh-keygen -l -f /path/to/key This will display the key fingerprint and the associated comment. By checking the comments, you should be able to identify which key is which. What if I can't find my SSH keys for uploading files with a key file, and how can I use new keys? If you can't find your SSH keys, don't worry - you can make new ones. To do this, open your terminal or command prompt and type a command to make a new SSH key. You can pick a name for the key, and choose a password to keep it safe. After making the key, you can add the new key to your account or server where you want to upload files using the key file. This usually involves copying and pasting the key's "public" (e.g. file.pub) part to the right place. If you lose track of the key again, just make a new one and add it again. Keep in mind that SSH keys belong to you and your computer, so if you switch computers or accounts, you'll need to make new keys. I don't want to type the passphrase every time I use the key. What can I do? You can use an ssh-agent to avoid typing the passphrase every time you use the key. An ssh-agent is a program that stores your private keys in memory and provides them to ssh when needed. You can add your key to the ssh-agent using the command ssh-add followed by the path to your key file.Here's an example of the steps to follow: Open a terminal window.Start the ssh-agent by typing the command eval $(ssh-agent).Add your key to the ssh-agent by typing the command ssh-add [key filepath]. For instance, if your key file is located in the home directory with the name mykey, the command will look like this: ssh-add ~/mykey After adding your, key to the ssh-agent, you should be able to use ssh without having to enter your passphrase every time. Can I use my password for authentication (without my private key)? If you prefer to use your username and password for authentication instead of your private key, you can still do so. When using a Graphical User Interface (GUI) such as FileZilla, you can select Ask for password as your Logon Type (Figure 3). This option will prompt you to enter your password when you click Connect, instead of using your private key. Figure 3: This option will prompt you to enter your password when you click "Connect", instead of using your private key. Figure 3: Process of establishing a new connection to __EGA_INBOX_DOMAIN__ using your password as the logon method in FileZilla. The figure showcases the FileZilla version 3.52.2 operating on IOS v11.2.3. By following the depicted steps, users can create a secure and efficient connection to the inbox, ensuring seamless data transfers. It's worth noting that using a password for authentication can be less secure than using an SSH key, as passwords can be more easily compromised through various means. However, if you choose to use your password for authentication, selecting "Ask for password" as your Logon Type is a good way to do so securely via a GUI. Why is it better to use my key and not my password? SSH keys for authentication is generally considered to be more secure and convenient than using passwords. SSH keys are more difficult to crack than passwords, and they can be restricted to specific users and machines, giving you more control over access. Once you set up your SSH keys, you can use them to authenticate quickly and easily, without having to enter a password every time. This makes automation of tasks, such as uploading encrypted files, much simpler. Additionally, SSH keys provide better logging, allowing you to keep track of who is accessing your systems and when. All in all, using SSH keys is a good practice for improving security and convenience in your authentication process.

  Documentation submission/data/uploading-files/inbox

• How to encrypt files with EGACryptor

  File preparation Due to the processes used at the EGA for file archival the use of non-alphanumeric characters in a filename will cause issues in archival. By convention whitespaces in filenames are to be avoided and should be replaced with the underscore character (_). Before encrypting your files please make sure that any files that will be uploaded to EGA do not use special characters such as # ? ( ) [ ] / \ = + < > : ; " ' , * ^ | &  Crypt4gh EGACryptor Files encrypted with EGACryptor must be uploaded via FTP EGACryptor The EGACryptor v.2.0.0 is a JAVA-based application which enables submitters to produce EGA compliant encrypted files along with files for the encrypted and unencrypted md5sum for each file to be submitted. The application will generate an output folder that will by default mirror the directory structure containing the original files. This output folder can subsequently be uploaded to the EGA FTP staging area via an FTP or Aspera client. Download EgaCryptor Download EgaCryptor Using the EgaCryptor Using the EgaCryptor Encrypting single file Encrypting multiple files Encrypting all files in folder Points to note Troubleshooting Troubleshooting Download EGACryptor The required jar files can be obtained by downloading EgaCrytptor jar file After the file has been downloaded, extraction of the zipped archive is required. The EGACryptor has been built to work with Java Runtime Environments from version 6 and above and with the OpenJDK Environment. Please refer to the relevant resources for installation guidance. Installing the latest version of the OpenJDK will include the JCE files. If your installation of Java JRE is less than 1.8.0_151 will require the manual installation of the JCE Policy Files. You can verify the version of the Java SE Runtime Environment (JRE) installed by using the command: $ java -version If you need to install the JCE please follow the instructions below: Installing the JCE policy files (due to licensing terms and conditions the required policy files must be downloaded direct from the ORACLE website) : Download the unlimited strength JCE policy files (JRE 6 / JRE 7/ JRE 8) Uncompress and extract the downloaded file. This will create a subdirectory called JCE. This directory contains the following files: README.txt, COPYRIGHT.html, local_policy.jar and US_export_policy.jar Install the two policy JAR files by replacing the existing ones in your java home directory. Install the two policy JAR files by replacing the existing ones in your directory. The standard place for JCE jurisdiction policy JAR files is: /lib/security [Unix] or \lib\security [Win32] Notes: refers to the directory where the Java SE Runtime Environment (JRE) was installed. Additional performance enhancements that have been included in the EGACryptor V2.0.0: The ability to parallelise the processing of datasets through the use of the resources on a system. Multicore systems will allow the user to specify n-1 cores for an n-core system. The use of this feature on clusters may speed up the processing of datasets that have large file numbers but consult your local cluster guide to ensure that there are not monopolising resources that are needed by other system users. The default for this process remains single threaded. 3 levels of system usage can be specified. Full usage within the limits detailed above. A limited mode that will ensure that 50% of the system resources are available for other tasks. Maximum mode is limited to 75% of system resources, this allows encryption to be prioritised but allows for the system to be usable for light alternate tasks. Finally there is a throttling mode that allows you to specify the exact number of computational threads to be used. the EGACryptor is able to ingest a structured directory and will output a directory with the same structure containing the encrypted files along with the md5checksums for the plain and encrypted files. The entire output directory can then be uploaded to the EGA for archival. as with the input path, it is now possible to specify the output path. the options have been updated inline with the upgraded functionality. The tool can only be used via the command line. The EGACryptor is designed to perform a single task, encrypting your data, for upload of these files please refer to our uploading guide Using the EgaCryptor Below are the three ways on how the EGACryptor tool can be used: Encrypting a single file : java -jar ../EGA-Cryptor_2_0_0.jar -i example1.bam Encrypting multiple files : java -jar ../EGA-Cryptor_2_0_0.jar -i "example1.bam,example2.bam" Encrypting all the files within a folder java -jar ../EGA-Cryptor_2_0_0.jar -i path/to/target/directory By default the EGACryptor v2 will create a new output directory containing all encrypted files and the relevant checksums within the target directory. If a specific directory is desired this can be specified by using the -o flag. This can be achieved in a similar manner to the following example: java -jar ../EGA-Cryptor_2_0_0.jar -i /path/to/target/directory -o /path/to/output/directory The tool will output three files per input file: file.gpg ( encrypted file ) file.md5( file md5 sum value file ) file.gpg.md5 ( encrypted file md5 sum value file ) All output files must then be uploaded to your submission account using Aspera or FTP. Further documentation on how to upload files: FTP and Aspera. Points to note Remember to provide the path to EgaCryptor.jar and run the command from within the directory the file/s are located. ECryptor writes files to the source directory of your local file system, as a result you must have write-access permissions for the source directory. Troubleshooting If in doubt about the function of the EGACryptor it is recommended to first consult the built-in documentation. This can be accessessed by using the -h flag as stated in the following table. Built-in Commands Table: list of the command line options built into EGA-Cryptor v2.0.0. Command line Option Action --------------------- ----------- -f Set this option to allow application to create maximum threads to utilise full capacity of cores/processors available on machine -h Use this option to view the bult-in help menu * -i File(s) to encrypt. Provide file/folder path or comma separated file path if multiple files in double quotes -l Set this option to allow application to create maximum threads equals to 50% capacity of cores/processors available on machine -m Set this option to allow application to create maximum threads equals to 75% capacity of cores/processors available on machine -o Path of the output file. This is optional. If not provided then output files will be generated in the same path as that of source file (default: output- files) -t Set this option if user wants to control application to create maximum threads as specified. Application will calculate no. of cores/processors available on machine & will create threads accordingly Encryption Errors UnixFileSystem.createFileExclusively (Native Method) The error is thrown by UNIX ("UnixFileSystem.createFileExclusively(Native Method)"). It appears that the user does not have write-access to the file system where the file to be uploaded is located. EGACryptor always writes MD5 checksums into files before uploading them to the server, and these files are created in the same location where the uploaded file itself resides.Solution: address your directory permission issue and re-run the command. Install JCE Unlimited Strength Jurisdiction Policy files The JCE policy unlimited strength jurisdiction files should be installed according to your current java version If you are still facing difficulty with the EGACryptor v.2 after having consulted the documentation please contact the EGA Helpdesk.

  Documentation submission/data/file-preparation/egacryptor

• Quick Guide for data submission

  Quick Guide This is a quick guide to submit data to the EGA. Please select data type to display the right detailed instructions. Sequence data Array-based data Get your submission account You first need to create your EGA user. Then, once your account has been verified by the Help Desk team, request your submission account and provide details of the data type and platform(s) in your submission. Register Study/DAC Use Submitter Portal to register your study, samples, Data Access Committee (DAC) and Policy. Upload data Encrypt your data files using the and upload it to your inbox using SFTP. Register experiments and runs Associate each data file to a registered sample and study by Linking files to samples. Details of the experimental procedure you followed must be provided. Finalise your submission Group your runs/analysis into datasets and link them to your new or existing DAC and policy using DAC Portal. Data request are done at a dataset level, thus files within a datasets must share release conditions. Release and admin When finalising the submission, set a released date to instruct our Helpdesk to release your study. All registered studies are automatically placed on hold until the named submission or DAC contact instructs our Helpdesk for the study to be released. Get your submission account Fill the submission form and provide details of the data type and platform(s) in your submission. Register Study/DAC Use EGA Programmatic submission to register your study, samples, Data Access Committee (DAC) and Policy. Upload data Encrypt your data files using the EgaCryptor and upload it using default FTP clients or Aspera. Complete the Array-Format (AF) spreadsheet Download the AF spreadsheet and complete all four sections. Return the spreadsheet to the EGA helpdesk. Release and admin Instruct our Helpdesk to release your study. All registered studies are automatically placed on hold until the named submission or DAC contact instructs our Helpdesk for the study to be released.

  Documentation submission/quickguide