Tools

Learn about the different tools available on the UKB-RAP.

Docker

The Research Analysis Platform allows users to bring in their own application through executing Docker Images. Join DNAnexus experts as they detail how to package and run your own application with Docker.

The session will cover how to use Docker on the command line interface (CLI), commands necessary for importing your registries, and how to run your package via the Platform GUI or CLI.

Video chapters

• 0:00 Introduction

• 2:43 Why Docker/Containers?

• 4:42 Docker Registries

• 6:05 Terminology

• 8:57 Docker and Security

• 12:03 Snapshot Workflow

• 12:59 Creating a bioinformatics snapshot on RAP

• 14:36 docker pull an existing base image from registry

• 16:02 docker save container to snapshot (tar.gz file)

• 17:29 Batch Processing with Swiss Army Knife

• 18:54 Using Docker Snaphot File in dx run SAK

• 23:24 Batch processing: dx find data | xargs

• 25:52 dx terminate

• 28:01 Basic Applet Structure

• 32:18 Use Docker directly inside an Applet

• 35:13 Using Docker Images

• 41:33 docker pull an existing base image from repository

• 42:36 Open a bash shell on container

• 44:21 EMBOSS

• 45:16 Install additional software

• 46:40 Use docker commit to save changes to image

• 48:34 Dockerfile: A Recipe to install software/dependencies

WDL

Introduction to WDL

DNAnexus experts take you through creating a starter WDL script and explain the basics of WDL syntax and how to use it on UKB RAP. They then compare it with native DNAnexus workflows and provide you tips on how to decide when to use WDL.

Video chapters

• 00:00 Introduction

• 05:11 General Workflows on RAP

• 08:10 Defining WDL & WDL Workflows

• 15:10 Running WDL Workflows

• 22:36 Writing WDL

• 28:06 WDL Script Example

• 35:04 WDL Syntax

• 45:47 Helpful Resources

• 47:00 Q&A

Advanced WDL concepts and docker

DNAnexus Experts elaborate on the starter example from our introduction to WDL webinar and add more complexity to our WDL script. Then they introduce more syntax (e.g. WDL expressions, dynamic resources, etc.) and discuss how to use WDL with the Docker environment.

Video chapters

• 00:00 Introduction

• 01:10 Recap of Previous WDL Webinar

• 02:46 Agenda

• 03:34 Specifying Inputs for a Workflow Run

• 06:30 Meta and Parameter Meta

• 09:14 Reusing Native Apps & Modularizing Workflows

• 13:35 Subworkflows

• 16:20 Standard Library

• 19:40 Expressions

• 25:31 Dynamic Resource Allocation

• 29:09 WDL Best Practices

• 34:10 Project Management

• 38:50 Docker & WDL Workflows

• 49:42 Q&A

Rstudio

Analysing UKB data using Rstudio workbench

The UK Biobank dataset is a uniquely rich resource containing over 10 petabytes of genetic and health data from over 500,000 volunteer participants. With the launch of RStudio Workbench Trial Version on the Research Analysis Platform, approved UK Biobank researchers can now analyze this extensive dataset using their programming language of choice.

Experts from RStudio, UK Biobank and DNAnexus walk through using RStudio Workbench to analyze the extensive UK Biobank dataset, available to all UK Biobank researchers free of charge until August 31. This webinar provides an overview on how to create notebooks, dashboards and incorporate Shiny apps all in the cloud on the Research Analysis Platform.

Video chapters

• 00:00 Webinar Introduction

• 04:05 UK Biobank Dataset & RStudio Workbench

• 18:38 RStudio Workbench Demo

• 40:48 RStudio Workbench on RAP

• 55:42 Q&A

Using R and RMarkdown

RStudio Workbench Trial Version is now available on the UK Biobank Research Analysis Platform, allowing users to use RStudio in their analysis. This session covers everything you’ll need to know to start working with the application and use R & R Markdown on the Platform.

Video chapters

• 0:00 Introduction

• 2:23 Local Analysis

• 3:07 Cloud-based Analysis

• 4:29 UKB RAP Project Storage and RStudio Instance Storage

• 6:57 Launching RStudio Workbench Trial Version

• 9:51 Terminating RStudio Workbench

• 11:05 Overview of RStudio Workbench layout

• 11:23 What is dx-toolkit?

• 12:59 Utilize Bulk Data Files

• 13:57 Bulk per Sample File Folder Structure

• 14:24 Transfer Bulk Files/Use dxFUSE

• 15:57 Utilize Pheno Data

• 17:14 Field title vs field name vs field UK Biobank format

• 18:31 Table exporter inside R script

• 20:38 Example phenotypic data

• 22:57 RStudio project versus UKB RAP project

• 26:19 Creating RStudio project

• 28:51 What is renv?

• 30:45 Save project using dx-backup-folder

• 31:52 Restore project using dx-restore-folder

• 32:53 dx upload versus dx-backup-folder

• 33:55 Restoring a RStudio Project from UKB RAP Project

• 34:32 Install packages after restoring

• 35:43 UKB RAP Project Storage and RStudio Storage

• 38:27 Launching Studio using ttyd with Docker container

• 40:12 Disadvantages

• 42:41 Access the ttyd instance

• 42:57 Running Bioconductor Docker in ttyd docker run

• 47:27 Q&A

Jupyter Lab

Introduction to Jupyter notebooks

An introduction to using Jupyter Notebooks on the Research Analysis Platform, to explore and analyze the uniquely rich UK Biobank dataset.

Learn how to:

• Understand the features of Jupyter notebooks and how to use them, based on your research interests and preferred computing language (Python, R, or Stata)

• Utilize Table Exporter to export cohorts as TSV files

• Utilize JupyterLab to load and analyze bulk data files on RAP

Video chapters

• 00:00 Introduction to JupyterLab on RAP

• 14:16 Starting Cohort Browser from a Dataset

• 19:52 Running Local Analysis vs. Cloud Analysis

• 21:41 Launching JupyterLab Instances

• 25:52 Working with JupyterLab

• 41:05 Demo

• 50:04 Q&A

Exploring and analyzing with Jupyter notebooks

Go in-depth analyzing the uniquely rich UK Biobank dataset with Jupyter notebooks. Learn how to utilize Jupyter Notebooks and Apache Spark to explore and analyze UK Biobank’s trove of phenotypic and health record data, covering 500,000 volunteer participants.

Video chapters

• 0:00 Introduction

• 3:27 Understanding Spark

• 13:00 Review of UKB Data

• 18:22 DataFrames

• 21:36 Workflow for Extracting Info from a Cohort

• 34:49 Tips on Working with DataFrames

• 35:37 More Information on JupyterLab Notebook

• 37:17 Demo: Access and Use Apollo Datasets in Spark Enabled JupyterLab

• 47:01 Summary of Spark JupyterLab

• 48:44 Q&A

Apps

Building Applications for Efficient Analysis on UKB-RAP

Learn the tips and tricks to faster, more efficient, and cost effective analysis by building applications on UKB-RAP. This webinar demystifies the app building process and show users how easy it is to build applications using pre-existing tools on the platform.

This webinar covers:

• The basics of app building to suit your analysis projects

• How to choose the right tools

• How to determine how many spot instances

• Real-world examples of how UKB-RAP users built their workflows

• Integrating containers into app building

• Leveraging spark when and where necessary

Video chapters

• 00:00 Housekeeping & Speaker Introductions

• 04:51 Introduction to Building Apps

• 15:55 Tips & Tricks for Building Python Apps

• 28:52 Using UKB-RAP like an on-prem HPC Cluster

• 49:31 Q&A