Gateway Releases
 

The Innovation Gateway requires a significant volume of design and development work to deliver our vision and ambition. To achieve this our teams are continually working on the Gateway delivering major software releases approximately every 4 weeks.

On this page we explain what new developments we are currently working on and list the major releases made since the Innovation Gateway was released in its current format in June 2020. Clicking on an entry provides further detail about the functionality that the particular release delivered.

 

This month we are pleased to announce a new, automated approach to onboarding datasets to the Gateway. This feature enables data custodians to have a more straightforward way of updating the metadata made available on the Gateway.

Federated Metadata Automation

Prior to this release, adding a dataset to the Gateway was a manual process. Data custodians were required to enter the information held in their metadata catalogues, including descriptive and technical metadata, onto the Gateway via the dataset onboarding form, working through several sections before submitting a dataset for review.

Now, a data custodian has the option to synchronise their metadata catalogue directly with the Gateway so that data descriptions are automatically pulled through when onboarding datasets, reducing manual work. What’s more, a change to a custodian’s catalogue updates the Gateway record too, so researchers will always have the most up to date information possible when searching for datasets published on the Gateway.

We hope that this work will enable us to further scale the number of datasets available on the Gateway and allow other data providers to make use of the resource to build further tools and services of benefit to the wider ecosystem.

Other updates in this release

We have updated the custodian team management dashboard so that a group email address can be used to send and receive notifications as part of the data access request application process (under ‘Notifications’ the toggle should be switched on and a team email entered to activate). Similarly, data applicants making an initial enquiry about a dataset via the Gateway (by clicking ‘How to request access’ at the top of the dataset page followed by the ‘Make an enquiry’ button) now receive an email receipt of the message sent to the data custodian.

This release we also expanded the events (or actions) that we track on the Gateway homepage, improved the meta description of some of our site pages, and fixed several bugs including a correction to the dataset onboarding form, restoration of the character count (0/1500) across all text box sections of the data access application process, and repaired the ‘Report a bug’ button!

This release concentrated on enhancing the Gateway Data Use Register – a big piece of functionality we released earlier this year (see Jan release notes). We also made improvements to the data access application process and made form-filling in general across the Gateway more streamlined for users.

Data Use Register

We’ve enhanced the data use upload process so that users can dismiss warning and confirmation messages whenever they like, and if the upload is taking longer than usual then a loading gif and message will indicate this to the uploader. The analytics tags we use across the data use pages have been updated so that we can better understand how the data use register is being used, for example, to find out which options and filters are most frequently selected by Gateway users. This will allow us to focus our future development efforts on the most used functions.

We also fixed several bugs connected to the data use register:

  • Data uses generated automatically by Gateway data access requests and are no longer duplicated on the ‘Active’ tab and can now be archived.
  • An organisation added to a data use is now properly listed under the ‘Safe people’ section.
  • The download results button on the data use register has been fixed.
  • Edits to a data use when adding a non-Gateway dataset can now be saved.

Data access requests

The Five Safes application form for data access requests managed via the Gateway has improved so that it is now faster to complete – applicants will see an ORCID iD field under ‘Other Individuals’ in the ‘Safe people’ section of the form which can be used to prepopulate information on behalf of co-applicants. Note that applicants must first be added to the ‘Invite Contributors’ on the ‘Before you begin’ section for the prepopulate to work.

Also new to the Gateway data access request form is the ability for applicants to indicate the type of research to be conducted under ‘Project details’ in the ‘Safe project’ section of the form. When ‘research’ is selected as the answer to ‘What is the type of project?’ the user is presented with an expandable dropdown of selectable options and sub-options based on the Ontology Lookup Service, allowing applicants to be more transparent with their data access applications.

Faster form filling with autosuggest

To save time when adding people to multiple Gateway forms, for example ‘collaborators’ to a collection page or ‘contributors’ to a data access request, we have developed a ‘Recently added’ dropdown box that pulls a list of people previously added to any form across the Gateway by the current user.

If the current user has never added someone to a form before, the dropdown only becomes visible to the user once they start typing a name into the search box. The correct person can be selected from the results list and added to the current form – they will then appear in the current user’s ‘Recently added’ dropdown box ready to be autosuggested for next time.

Other

This release, we also fixed the ‘Browse by what’s new’ carousel on the homepage, updated the dataset page to include associated documents, and added the ability for a data custodian to disable the ‘Add new dataset’ button if they have a federated Gateway account (part of our federated metadata automation work).

Our March release focussed on improving the spatial field search filter.

We’ve updated each field in the spatial search filter to display a more readable location name, for example London or Wales, replacing the URL format from before. This makes it easier for users to apply the spatial filter when performing a search in the Gateway as they can see all the place name fields in hierarchical order (South London under England, for example). It’s also easier for custodians when onboarding datasets as they can add the location simply by typing its name into a search box when completing the spatial fields.

Please note a potential action for users - a Gateway search containing a spatial filter and bookmarked prior to this release will need to be updated by the user as the original URL will no longer be valid.

This release we also:

  • Implemented changes to improve the Data Access Request experience, including larger free-text boxes and pre-population of profile fields.
  • Fixed a few bugs related to our new Data Use Register functionality (see Jan release).
  • Improved the dataset management process by adding search and sort functions to the dataset dashboard.
  • Modified overall search behaviour to improve accuracy of search results containing multiple search items.

In January we released the first version of our Data Use Register (DUR) functionality.

This new feature means that data custodians can publicly show how their Gateway datasets are being used, by who and for what purpose – custodians can upload their data uses manually but this process is automatic for those using the Gateway data access request management system. We have already published over 600 data uses.

Users can search the Gateway data use register and view all data uses approved by data custodians, and these can be edited, updated and archived as a project progresses. Researchers might be interested to see how datasets listed on the Gateway are already being used to help inform their data request choices, whilst the public and funding bodies might search the register to understand how researchers are using the data.

The Gateway Data Use Register aims to improve transparency in the use of health data for research and innovation and provide a best practice example of how information about approved data uses is shared and managed. Its development is part of an ongoing project – led by the UK Health Data Research Alliance – to develop a core set of standards on data use registries.

Our December release focussed on updating dataset citations and login abilities.

The ‘export citation’ button makes it easier for users to export an automatically generated citation from the Gateway dataset page – clicking on the button copies the dataset citation to clipboard, confirmed by a pop-up message at the top of the page.

Users are now able to login to the Gateway with their ORCiD ID or via Office 365, increasing the number of signup and sign-in options to five – there’s now an option to recommend a new identity provider via the login modal too. Plus, when a user returns to the Gateway after logging out, they will see an orange dot next to their last preferred sign-in option.

We also improved a lot of our internal admin functions related to the approval of datasets listed on the Gateway (which means a smoother discoverability process, what we term on-boarding, for data custodians) and fixed some general bugs.

Our October release included two new large features- Activity Log for the data access request process and the Data Utility Wizard search tool.

The Activity Log now allows a complete log of all activities within a data access request application on the Gateway. This allows Gateway users making a request to see a history of updates for a better experience to manage their application and also benefits data custodians using the Gateway- the complete log reduces paperwork and offline folders/archives.

The Data Utility Wizard tool helps to refine a users search to only datasets that meet their data utility requirements- based on the Data Utility Framework we previously developed. The tools directs users to datasets that are mostly likely to meet their needs through an easy to understand interface.

We also pushed some developments for our internal use, such as integrating our CRM, Hubspot, with the Gateway and improving our Google analytics.

Our September release updated our metadata onboarding process- allowing custodians to apply for a dataset to be submitted even if there is a mandatory field missing and allowing the HDR internal team to create new Gateway 'teams' for custodians so they can create metadata for their datasets faster.

We also have been working on data profiling for our Cohort Discovery Tool so researchers will get a better look into the data that would be available to them when they request access to that dataset.

Our August release was mainly Data Access Request (DAR) process focused with 2 new bits of functionality released:

  • The Amend functionality which enables researchers to submit amendments to previously approved applications (researchers can now re-open a previously approved application and submit specific updates like adding a new dataset or a new collaborator).
  • The Contextual messaging functionality which enables exchange of messages between custodian and researchers. This facilitates communication between researchers and data custodians, making it easier to complete a data access request form.

We also onboarded a new data custodian to the Gateway- the Neonatal Data Analysis Unit from Imperial who will be using the end to end Five Safes data access process on the Gateway for all their data requests going forward.

Our July release updated 4 different parts of the Gateway:

  • Users can now browse the most popular (viewed) resources on the homepage via a carousel
  • We updated the stats dashboard with separate sections for monthly stats and cumulative stats
  • Our data access request process has changed for the datasets that use our 'Five safes' form- we now prompt users to make an enquiry to the data custodians first, before submitting the full form. This allows users to connect with the data custodian to ask any pre-submission questions and discuss their requirements.
  • We have added a section to the papers listed on the Gateway- users can now input authors in a separate section when uploading these.

Our June release allows users to now search within collections and sort the resources tagged in the collection by views, date and more, which will be particularly helpful with some of the larger collections. Across tools, projects, papers, collections and people, the new 'sort by' option has also been implemented allowing users more options to drill down and find relevant resources to them.

We have also implemented a new filter on the datasets page to allow filtering of datasets on whether they have consented for commercial use- particularly relevant for our industry users.

Finally, we updated our navigation bar across the site to match the new bar we implemented on the new homepage previously.

May has been a busy month for the Gateway. For our second May release we are very excited to reveal to you our new homepage design! It's brighter and fresher, with careful consideration paid to providing you the most information in the clearest possible way.

As if that wasn't enough, this release also contained improvements to the way a user is able to sort their search results, the ability for data custodians to have data access request forms sent to a generic email inbox and we have updated our T&Cs to to offer our audience the opportunity to stay in touch.

Our first May release delivered a major step forward for the 5 Safes Data Access Request forms (DAR), with applicants now able to create new DARs quickly by cloning an existing one. That means answers can now be copied from one application form into another. This will see a significant reduction in re-keying and data entry time for completing separate submissions to other custodians using 5 Safes.

This new functionality also paves the way for future developments on the DAR including the ability to amend, renew and extend applications, as well as applying for datasets across multiple custodians with a single form, and question bank.

We have now completed the build of HDR UK's own metadata catalogue and successfully transitioned data on all of the datasets you could previously find on the Gateway into the new catalogue. This improves stability and offers more control to improve the quality and completeness of metadata on the Gateway. It will provide you, and other researchers and innovators, with an improved experience when discovering healthcare data.

For updating or adding additional metadata, the metadata interface and workflow are now fully integrated into the Gateway. This provides a single, consistent user experience for data custodians, with native support for dataset versioning, making it easier for custodians to onboard/update information about their datasets onto the Gateway. And for those of you with your own in-house metadata catalogue, or if you are considering implementing one, our data model also supports interoperability with these. Please contact Alison.Elderfield@hdruk.ac.uk for more information.

Our April 2021 release includes some exciting new functionality - Cohort Discovery - as well as a significant improvement and extension to the dataset discovery filtering functionality within the Gateway.

Cohort Discovery: An innovative and exciting new Gateway feature, Cohort Discovery will allow you to search across many datasets to find those that include groups, or cohorts, of patients with specific characteristics that you require to support your research. For example, you could use the Cohort Discovery functionality to look for datasets that contains patients that don't smoke aged between 18-30 and who live in London. You can access the Cohort Discovery feature within the Gateway here.

Advanced dataset discovery filtering: The HDR UK Improving and Uniting Data teams have been working with data custodians to increase the breadth of the data describing each of the datasets - called metadata - listed on the Gateway.

To allow you to filter your search results during dataset discovery using this broader set of metadata, we have completely reengineered the way the filters work to use this broader set of metadata to create a much more compelling experience for you.

  • New filter categories: We have introduced new filter categories (and sub-categories) for Coverage; Provenance; Access; Format and standards; Data utility; and Technical Metadata. In addition to the existing filter categories of Keywords; Phenotype; Geographic coverage these provide you with a more powerful and flexible way to focus on the datasets that meet your research needs.
  • Advanced filtering functionality: In extending the filters we have also integrated search functionality into the filters that contain larger numbers of filter options e.g. the Keywords; Phenotype and Publisher filters, helping you find relevant filter options.

Data Access Requests: You can now delete Data Access Requests that are in a pre-submission state, enabling you to remove those draft access requests that are no longer needed. We have also introduced system validation of your DAR and a confirmation request from you before it is submitted. This will help you to avoid submitting applications prematurely or by mistake.

User support documentation: We have expanded, re-categorised and simplified the support pages for the Gateway to make it easier for you to find information to answer any questions you have about Gateway functionality or how you complete a task.

Performance and stability improvements

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Improving support for Internet Explorer 11.
  • Improving performance and stability of the metadata utility calculations.
  • Making enhancements to the Gateway database schema

Our March 2021 release continued to add further functionality to the 5 Safes Data Access Request forms, greater flexibility to Gateway collections and enhancements to the Gateway APIs.

Data Access Requests: For a data access request you have yet to submit (pre-submission DAR), you can export the questions and any answers you have completed into an email to share or print. You can also now delete any documents you've uploaded to a pre-submission DAR. Finally, we have added a scan indicator to let you know when documents you've uploaded are being checked for viruses.

Collections: You can now make your own Collections public for other Gateway users to see. At the same time we have also improved the look-and-feel of the collection cards and the keyword functionality making it easier for all users to discover and explore collections on the Gateway.

Searchable collections: When you run a search this now includes the Collections published on the website ensuring any Gateway resources that are relevant to your search are returned.

Gateway APIs: We have added further functionality to the Gateway's APIs making it easier for Custodian's access management systems to interoperate with the Gateway.

Performance and stability improvements

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Notifying you at login about about new mandatory user profile fields
  • Display of linked gateway resources has been improved to ensure they are listed for both linked resources.
  • Website improvements for mobile/tablet users.

Our February 2021 release focused on adding further functionality to the 5 Safes Data Access Request forms, onboarding a new data custodian to using the form and improvements to papers and collections.

Data Access Requests: As you complete a data access request an application question counter will now show how far you are through the questions for each section. The forms have also been extended so you can now add multiple people, organisations, and funders to a data access request. And finally, you can no longer submit a data access request for an old version of a dataset.

ISARIC now using 5 Safes forms: ISARIC are now using the enhanced 5 Safes data access forms for the data sets they have listed on the Gateway.

Papers: You are now able record multiple Digital Object Identifiers (DOIs), URLs or PDF links for each paper you add to the Gateway.

Collections: We have improved the layouts to collections and made featured collections more prominent to make them easier to identify.

Performance and stability improvements

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Where a request results in a large number of results we have now split these over multiple pages to keep page loading times to a minimum.
  • Duplicating abstract for Papers has been removed.
  • Website improvements for mobile/tablet users.
  • Accessibility improvements.
  • Ensuring IE/Safari support for user registration

Following a slightly longer sprint due to Christmas and New Year, our January 2021 release focused on adding further functionality to the 5 Safes Data Access Request forms along with some additional improvements in response to your feedback.

Faster results filtering by custodian: You can now quickly filter your dataset search results for a single custodian by clicking on the custodian's name within the search results which is beneath the dataset name.

User feedback buttons: We encourage user feedback and you can now reports bugs and suggest feedback from the footer of every page within the Gateway.

Data access requests: Where you don't complete an answer to a question, these are now highlighted to you making it easier to ensure you complete your access requests correctly.

Collection navigation improvements: We have removed the All tab from collections making it easier for you to find the resources that you need within each collection.

Performance and stability improvements

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Ensuring IE/Safari support for user registration.
  • Migrating 190 Courses to support the new courses entity.

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Support for IE user registration.
  • Fixing a small number of redirect links

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Update to metadata quality to use weighted scoring.

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Improvements to metadata catalogue caching performance and stability.
  • Fixed multiple minor issues with 5 safes data access request forms.
  • Fixed issue with National Core Studies data access request schema.

Our November 2020 release - the first of Milestone 2 - focused on adding further functionality to the 5Safes data access request forms, onboarding a number of custodians to use the new form along with a number of other improvements in response to your feedback.

National Core Studies & Early Adopters* now using 5 Safes forms: NCS's and Early Adopter organisations are now using the enhanced 5 Safes data access forms for the data sets they have listed on the Gateway.

Persistent dataset IDs: Datasets within the Gateway now have persistent IDs rather than the ID changing with each new version of the same dataset as provided by the metadata catalogue supplier. This will make it much easier for you to reference datasets in your resources helping to avoid using links that are out of date.

Dataset v2 updates: Dataset metadata has been transitioned to the new, more advanced, version 2 schema. This provides the foundation for you to undertake more advanced searches and results filtering in the future.


* National Core Studies & Early Adopters are: NHS Digital; Office for National Statistics; NHS Scotland; SAIL Databank; NI Honest Broker Service; INSIGHT HDR UK Hub.

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Fixed issue where search figures were not showing correctly on the dashboard page.

Our early November 2020 release focused on 5 Safes Data Access Requests (DARs), Data Custodian workflows and adding educational materials to the Gateway along with a number of other improvements in response to your feedback.

5 Safes DARs: In this release we have deployed a new standard, yet configurable, DAR form with the objective of making it easier for you to submit compliant DARs that were "right first time" enabling data custodians to effectively process them without having to seek clarification from requestors. We wanted to build this around a recognised and respected framework for considering access to confidential or sensitive data and an obvious choice was the Office of National Statistics' (ONS) Five Safes model which considers each request across five dimensions:

  • Safe projects
  • Safe People
  • Safe settings
  • Safe data
  • Safe outputs

Working with custodians we have defined the common, and custodian specific questions and guidance that would be required under each dimension for them to be able to process a DAR. We are now working with an initial set of custodians for them to adopt the new 5 safes DARs.

Educational materials (Courses): You can now add, edit and remove educational materials to the Gateway.

Data custodian workflows: You can now create multiple workflows and customise them by assigning people from your team to each step of a workflow. Every time a new DAR is submitted, you can also choose which of your workflow processes that you would like to assign to each DAR.

Uploading and downloading files: You can now upload files to the Gateway when adding resources to the Gateway. Any user is then able to download them. files are virus scanned as part of the upload process to protect the Gateway and other users' devices.

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Fixed an issue where submitting a request to access the HDR UK Papers project fails.

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Fixed issue where technical metadata details were blank for some datasets.
  • Fixed issue where datasets with over 100 elements of technical metadata were only showing the first 100 elements to users.
  • Fixed minor issue with edit button labelling on Gateway entities.

Our October 2020 release focused on a new design for Gateway Collections and the introduction of Collection categories. Further work has also taken place on Data Access Requests (DARs) as well as the release of a new dashboard reporting Data Access Requests (DARs). We have also progressed a number of other improvements in response to your feedback.

Collections & Collection categories: In response to your feedback we have refreshed the design for Collections on the Gateway. This will make it easier for you to discover collections and their content. We have also added functionality to group Collections within a Category. This will allow Collections with a common theme to be bundled together making it easier for you to see related Collections all in one location on the Gateway.

DAR Activity dashboard: A dashboard is now available on the Gateway where you can explore at-a-glance information about the Gateway including: monthly statistics; the datasets that have received the most access requests; top searches; and unmet demand i.e. searches that yielded no results.

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Fixed issue with Single Sign On (SSO) for some users.

Our August 2020 release focused on improving search and filtering functionality, making Gateway statistics available through a new analytics dashboard and providing additional functionality to help manage Collections. We have also progressed a number of other improvements in response to your feedback.

Search improvements & Advanced Filtering: We have improved the performance of the Gateway's dataset search functionality so that your search results are more accurate. In addition, advanced filtering functionality provides you with more options to reduce your search results based on various criteria to help you discover the resources that best meet your needs.

Analytics dashboard: We have released a dashboard page listing a collection of statistics, metrics and analytics to provide you with an overview of the Gateway's data and performance.

Archiving Collections: You can now archive and unarchive Collections that you have created on the Gateway.

Bulk load of Papers: 400 papers have been loaded into the Gateway significantly improving the richness of resources available to you as a user.

Unsubscribing from Latest news emails: You can now unsubscribe from the latest news emails using the link at the bottom of the email message.

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Fixed an issue where a blank screen appeared when trying to upload a project.
  • Fixed an issue with the metrics displayed on the homepage.
  • Fixed an issue where part of the homepage does not display correctly on some browsers.

Our July 2020 release focused on the first version of our Data Access Request (DAR) form as well as introducing Collections to the Gateway.

Data Access Requests (DARs): You can now submit an enquiry to data custodians to request access to one of their datasets listed on the Gateway.

Collections: You can now use Collections to group Gateway resources together for future reference and quick access. These could be based on a theme or just a group of favourite resources from across the Gateway.

Our July 2020 release focused on the first version of our Data Access Request (DAR) form as well as introducing Collections to the Gateway.

Data Access Requests (DARs): You can now submit an enquiry to data custodians to request access to one of their datasets listed on the Gateway.

Collections: You can now use Collections to group Gateway resources together for future reference and quick access. These could be based on a theme or just a group of favourite resources from across the Gateway.

We have made the following performance and stability improvements as part of this release. These may not be obvious to all users.

  • Provided functionality to enable users to opt-out of Google Analytics cookies
  • Transferred the domain to https://healthdatagateway.org
  • Updated our About page to list the browsers we support
  • Fixed an issue when logging in with LinkedIn
  • Fixed an issue with word-wrapping on the landing page

Our 1 June 2020 release marks the start of the next stage in our journey to deliver a tool to make it easier for you - researchers and innovators - to discover and request access to UK Health datasets.

In this first release we have focused on providing you with key functionality to enable dataset discovery. You can now search and sort across an initial series of UK health datasets from a number of UK data custodians helping you to discover resources to help you with your research or business. We have also released functionality for you to record and share the research projects that you have worked on, or are working on, as well as the tools you use to conduct your research.