Posts Tagged ‘Object Storage’

Galaxy: A Workflow Management System for Modern Life Sciences Research

Nathan Bott

Healthcare Solutions Architect at EMC

Am I a life scientist or an IT data manager? That’s the question many researchers are asking themselves in today’s data-driven life sciences organizations.

Whether it is a bench scientist analyzing a genomic sequence or an M.D. exploring biomarkers and a patient’s genomic variants to develop a personalized treatment, researchers are spending a great amount of time searching for, accessing, manipulating, analyzing, and visualizing data.

Organizations supporting such research efforts are trying to make it easier to perform these tasks without the user needing extensive IT expertise and skills. This mission is not easy.

Focus on the data

Modern life sciences data analysis requirements are vastly different than they were just a handful of years ago.

In the past, once data was created, it was stored, analyzed soon after, and then archived to tape or another long-term medium. Today, not only is more data is being generated, but also the need to re-analyze that data means that it must be retained where it can be easily accessed for longer periods.

Additionally, today’s research is much more collaborative and multi-disciplinary. As a result, organizations must provide an easy way for researchers to access data, ensure that results are reproducible, and provide transparency to ensure best practices are used and that procedures adhere to regulatory mandates.

More analytics and collaboration represent areas where The Galaxy Project (also known as just Galaxy) can help. Galaxy is a scientific workflow, data integration, and data and analysis persistence and publishing platform designed to help make computational biology accessible to research scientists that do not have computer programming experience.

Galaxy is generally used as a general bioinformatics workflow management system that automatically tracks and manages data while providing support for capturing the context and intent of computational methods.

Organizations have several ways to make use of Galaxy. They include:

Free public instance: The Galaxy Main instance is available as a free public service at UseGalaxy.org. This is the Galaxy Project’s primary production Galaxy instance and is useful for sharing or publishing data and methods with colleagues for routine analysis or with the larger scientific community for publications.

Anyone can use the public servers, with or without an account. (With an account, data quotas are increased and full functionality across sessions opens up, such as naming, saving, sharing, and publishing Galaxy-defined objects).

Publicly available instances: Many other Galaxy servers besides Main have been made publicly available by the Galaxy community. Specifically, a number of institutions have installed Galaxy and have made those installations either accessible to individual researchers or open to certain organizations or communities.

For example, the Centre de Bioinformatique de Bordeaux offers a general purpose Galaxy instance that includes EMBOSS (a software analysis package for molecular biology) and fibronectin (diversity analysis of synthetic libraries of a Fibronectin domain). Biomina offers a general purpose Galaxy instance that includes most standard tools for DNA/RNA sequencing, plus extra tools for panel resequencing, variant annotation, and some tools for Illumina SNP array analysis.

A list of the publically available installations of Galaxy can be found here.

Do-it-yourself: Organizations also have the choice of deploying their own Galaxy installations. There are two options: an organization can install a local instance of Galaxy (more information on setting up a local instance of Galaxy can be found here), or Galaxy can be deployed to the cloud. The Galaxy Project supports CloudMan, a software package that provides a common interface to different cloud infrastructures.

How it works

Architecturally, Galaxy is a modular python-based web application that provides a data abstracting layer to integrate with various storage platforms. This allows researchers to access data on a variety of storage back-ends like standard direct attached storage, S3 object-based cloud storage, storage management systems like iRODs (the Integrated Rule-Oriented Data System), or a distributed file system.

For example, a Galaxy implementation might use object-based storage such as that provided by Dell EMC Elastic Cloud Storage (ECS). ECS is a software-defined, cloud-scale, object storage platform that combines that cost advantages of commodity infrastructure with the reliability, availability, and serviceability of traditional storage arrays.

With ECS, any organization can deliver scalable and simple public cloud services with the reliability and control of a private-cloud infrastructure.

ECS provides comprehensive protocol support, like S3 or Swift, for unstructured workloads on a single, cloud-scale storage platform. This would allow the user of a Galaxy implementation to easily access data stored on such cloud storage platforms.

With ECS, organizations can easily manage a globally distributed storage infrastructure under a single global namespace with anywhere access to content. ECS features a flexible software-defined architecture that is layered to promote limitless scalability. Each layer is completely abstracted and independently scalable with high availability and no single points of failure.

Get first access to our Life Sciences Solutions

You can test drive Dell EMC ECS by registering for an account and getting access to our APIs by visiting https://portal.ecstestdrive.com/

Or you can download the Dell EMC ECS Community Edition here and try it for FREE in your own environment with no time limit for non-production use

Examining TCO for Object Storage in the Media and Entertainment Industry

The cloud has changed everything for the media and entertainment industry when it comes to storage. The economies of scale that cloud-based storage can support has transformed the way that media organizations archive multi-petabyte amounts of media.

Tape-based multi-petabyte archives present a number of challenges, including a host of implementation of maintenance issues. Data stored on tape is not accessible until the specific tape is located, loaded onto a tape drive, and then positioned to the proper location on the tape. Then there is the factor of the physical footprint of the library frame, and real estate required for frame expansions – tape libraries are huge. This becomes all the more problematic in densely populated, major media hubs such as Hollywood, Vancouver and New York.

At first, the public cloud seemed like a good alternative to tape, providing lower storage costs. But while it’s cheaper to store content in the public cloud, you must also factor in the high costs associated with data retrieval, which can be prohibitive given data egress fees. The public cloud also requires moving your entire media archive library to the cloud and giving up the freedom to use the applications of your choice. Suddenly the lower initial costs of the public cloud can be wrapped up in a significantly larger price to pay.

Object storage is emerging as a viable option that offers media companies a number of benefits and efficiencies that the public cloud and tape-based archives simply cannot provide. In fact, object storage is rapidly becoming mandatory for applications that must manage large, constantly growing repositories of media for long-term retention.

Dell EMC Elastic Cloud Storage (ECS) blends next-generation object storage with traditional storage features that offer the media and entertainment world an on-premises cloud storage platform that is cost-competitive with multi-petabyte type libraries. ECS not only simplifies the archive infrastructure, it enables critical new cloud-enabled workflows not possible with a legacy tape library.

Instant Availability of Content

The greatest benefit of object storage for media and entertainment companies is the instant availability of their media content – you can’t access media on tape without a planned and scheduled retrieval from a robotic tape library. For a broadcast company, the delay in data availability could result in a missed air date, advertiser revenue loss, and legal fees.

With instant access to their entire archives, a whole new world of possibilities opens up for content creators. Archives aren’t often considered when it comes to content creation – the process of accessing media content has historically been difficult and the process of obtaining data often takes far too long. However, with instant access to archived media, archives can effectively become monetized, rather than just sitting around on tape in a dark closet gathering dust and being wasted. Being able to access all of your media content at any time allows rapid deployment of new workflows and new revenue opportunities. Further, with object storage, engineering resources that were focused on tape library maintenance can be re-focused on new projects.

Operational Efficiencies

Object storage can also offer increased operational efficiencies – eliminating annual maintenance costs, as one example. One of the biggest – and least predictable – expenses with operating a tape library is maintenance. Errors on a tape library are commonplace, drive failures and downtime to fix issues can impact deadlines and cause data availability issues that can require valuable engineering time and result in lost revenue.

Going Hot and Cold: Consolidation and Prioritization

Public cloud storage services can enable users to move cold or inactive content off of tier 1 storage for archiving, but concerns around security, compliance, vendor-lock and unpredictable costs still remain a concern.  Cold content can still deliver value and ESC allows organizations to monetize this data and provide an active-archive with the same scalability and low costs benefits, but without the lack of IT agility and reliability concerns.

ECS allows organizations to consolidate their backup and archive storage requirements into a single platform. It can replace tape archives for long-term retention and near-line purposes, and surpass public cloud service for backup.

In the video below, Dell EMC’s Tom Burns and Manuvir Das offer some additional perspective on how the media and entertainment industry can benefit from object storage: 

Stay current with Media & Entertainment industry trends

Digital Transformation with Radical Simplicity

Corey O'Connor

Senior Product Marketing Manager at Dell EMC² ETD

Digital Transformation with Radical Simplicity

Welcome to another edition of the Emerging Technologies ECS blog series, where we discuss topics related to cloud storage and ECS (Elastic Cloud Storage), Dell EMC’s cloud-scale storage platform.

The Inflection Point

It’s no surprise that unstructured data continues to grow exponentially year over year and doesn’t show signs of slowing down anytime soon. Some organizations are left managing this data with traditional storage infrastructure which is not only very expensive, but does not scale at the rate in which the data is growing.  IT budgets continue to remain flat or grow at an ungenerous rate of about 5% annually and capital expenses tend to double almost every year for most organizations.  The other pressing issue is the requirement of somehow maintaining the same (if not better) level of service with fewer resources as data growth continues to strain storage infrastructure.  This type of trend is not sustainable and if certain organizations do not transform their business then they will struggle without question.  We know what you’re thinking – wouldn’t it be great if the world’s largest provider of data storage systems created a cost effective, cloud-scale solution to solve this enterprise level challenge?

Dell EMC’s Elastic Cloud Storage (ECS)

Elastic Cloud Storage (ECS) is Dell EMC’s 3rd generation object-based storage system that provides the ability to:

  • Consolidate primary storage resources and elevate ROI
  • Modernize traditional and legacy applicationsfor better storage utilization
  • Accelerate cloud native applications to deliver new business value

ECS delivers a multipurpose platform that satisfies a variety of different use cases and plugs in perfectly to almost any existing Dell EMC investment(s). ECS singlehandedly simplifies management, increases agility, and most importantly – lowers costs.  At scale, ECS is undoubtedly one of the most cost effective solutions available in the market today.  In fact, analyst Enterprise Strategy Group (ESG) recently conducted a survey that shows ECS provides a 60% or greater cost advantage compared to other leading public cloud providers.   

ECS extends the cloud to primary storage and allows you to free up your infrastructure through Dell EMC cloud-enabled solutions (e.g. CloudPools, CloudBoost, CloudArray, etc.).  Customers have the ability to seamlessly tier colder, inactive workloads from existing primary storage investments (e.g. Isilon, VMAX Series, VPLEX, VNX Series, Vx Series, Data Domain, Data Protection Suite, etc.) to ECS.  This resource consolidation eliminates the need to purchase additional, more expensive platforms and better utilizes the infrastructure you have in your storage environment today.

An object-based platform like ECS can drastically increase responsiveness and better secure data when you compare it to that of a traditional NAS system.  Data is protected using erasure coding and the chunks of data are then geo-distributed across all nodes within the system providing instant read/write access from any location.  Strong consistency semantics ensures only the most recent copy of data is accessed simplifying application development efforts.  A geo-caching capability further enhances responsiveness through intelligent system recognition of access patterns which minimizes WAN traffic and improves system latency.

ECS provides simple and easy access to applications through a single global namespace.  This makes it easy for developers not having to deal with complex NFS file systems – they can focus on app development and not the operations and implementation details behind it.   By modernizing traditional applications into an object store, users get fast and easy provisioning, direct access to content over the web via HTTP, global accessibility through a single namespace, and the absolute best utilization of storage resources in the datacenter.

Cloud-native applications take full advantage of a cloud system framework.  ECS’ architecture is completely software defined with total abstraction from the north and southbound allowing compute and storage resources to scale independently from each other.  Everything within ECS is containerized and there are no hardware dependencies or the need to re-code, re-tool or reconfigure applications as ECS provides multi-protocol support. This allows developers to innovate and deliver their applications to market at a much quicker rate.

Bridging the Gap

Enterprises and cloud service providers alike can leverage ECS as a way to fund their ‘digital transformation’ as traditional, line-of-business applications go into decline and cloud-native apps begin to surge over the next decade.  ECS bridges the gap between Platform 2 (traditional) and Platform 3 (next-gen) applications on a single storage system.  Not only can ECS easily handle the extraordinary amount of unstructured data that’s growing, but as a multi-purpose platform it can serve up all the many different workloads you currently manage today and ready your organization for what the future throws at you.

Embrace Digital Transformation with Elastic Cloud Storage (ECS) 3.0

Sam Grocott

Senior Vice President, Marketing & Product Management at EMC ETD

Digital Transformation is drastically changing the business landscape, and the effects are being felt in every industry, and every region of the world. For some, the goal of this transformation is to use technology to leapfrog the competition by offering innovative products and services. For others, the focus is on avoiding disruption from new market entrants. Whatever your situation might be, it’s clear that you can’t ignore the change. In a recent study by Dell Technologies, 78% of global businesses surveyed believe that digital start-ups will pose a threat to their organization, while almost half (45%) fear they may become obsolete in the next three to five years due to competition from digital-born start-ups. These numbers are a stark indication of the pressure that business leaders are feeling to adapt or fall by the wayside.

But for IT leaders, this raises an uncomfortable question: Where will you find the money to make this transformation? You’re already under constant pressure to lower IT costs. How can you invest in new technologies while still doing this?

Elastic Cloud Storage (ECS), Dell EMC’s object storage platform, was built to help organizations with precisely this challenge. After being in market for just under two years, the latest release, ECS 3.0 is being announced at Dell EMC World today. ECS is a next-generation storage platform that simplifies storage and management of your unstructured data, increases your agility, and most importantly, lowers your costs. Let’s take a look at some of the ways ECS can help modernize your datacenter, clearing the way for you to embrace Digital Transformation.

Simplify and Accelerate Cloud-Native Development

The success of companies like Uber and AirBnB has highlighted the transformative power of “cloud native” mobile and web apps. Enterprises everywhere are taking note – in the previously mentioned Dell Technologies survey, 72% of companies indicated that they are expanding their software development capabilities. Often, these software development efforts are directed towards “cloud-native” applications designed for the web and mobile devices.

ECS is designed for cloud-native applications that utilize the S3 protocol (or other REST-based APIs like OpenStack Swift). ECS natively performs many functions like geo-distribution, ensuring strong data consistency and data protection, freeing up application developers to focus on what moves their business forward. This greatly increases developer productivity, and reduces the time to market for new applications that can unlock greater customer satisfaction, as well as new sources of revenue.

Reduce storage TCO and complexity

Legacy storage systems that sit in most enterprise datacenters are struggling to keep up with the explosion in unstructured data. Primary storage platforms are constantly running out of capacity, and it is expensive to store infrequently accessed data on these platforms. Additionally, as many businesses operate on a global scale, data coming in from different corners of the world ends up forming silos, which increase management complexity and lower agility in responding to business needs.
ECS is compatible with a wide range of cloud-enabled tiering solutions for Dell EMC primary storage resources like VMAX, VNX, Isilon and Data Domain.  Additionally, ECS is certified on many 3rd party tiering solutions, which enable it to act as a low cost, global cloud-tier for 3rd party storage platforms. These solutions drive up primary storage efficiency and drive down cost by accessing a lower cost tier with ECS. Tiering to ECS is friction-free, which means that apps or users accessing primary storage don’t have to change any behavior at all.

image1

Tape Replacement

The new ECS dense compute rack D-series increases storage density by more than 60%, making it an ideal replacement for tape archives. The D-Series comes as an eight node system that provides the highest density configurations for ECS at 4.5PB (D-4500) and 6.2PB (D-6200) in a single rack.

These new configurations provide the low cost and scalability benefits of traditional tape solutions, but without the lack of agility, poor reliability, and operational difficulties associated with storing data on tape.  Additionally, ECS makes business data available to BUs in an on-demand fashion. This allows organizations to fully embrace Digital Transformation, which relies on insights mined from business data to create more compelling experiences for customers.

Legacy application modernization

ECS can serve as an ideal storage platform for organizations looking to modernize legacy LoB applications that utilize or generate a large amount of unstructured data. Modifying legacy apps to point to ECS using the S3 (or other REST-based APIs like OpenStack Swift) protocol can help reduce costs, simplify maintenance of the application, and allow them to scale to handle massive amounts of data.

Take the Next Step

Learn more about how ECS can enable your transformation , follow @DellEMCECS on Twitter, or try it out – for free!

 

 

Breakfast with ECS: The Swiss Army Knife of Cloud Solutions

Corey O'Connor

Senior Product Marketing Manager at Dell EMC² ETD

Welcome to another edition of Breakfast with ECS, a series where we take a look at issues related to cloud storage and ECS (Elastic Cloud Storage), EMC’s cloud-scale storage platform.

ECS Cloud Enabling ToolsA Swiss army knife is a multi-layered tool equipped with a variety of attachments that can serve up many different functions. When first introduced in the late 1880s, it revolutionized the way soldiers performed their daily tasks – anything from disassembling service rifles to opening up canned rations in the field.  Fast forward to 2016, the use of the Swiss army knife may have changed quite a bit but the initial concept of consolidating various components into a single multi-purpose tool has certainly influenced organizations and industries across the world.

EMC’s Elastic Cloud Storage (ECS) is without question the Swiss army knife for cloud solutions.  ECS revolutionizes storage management by consolidating varied workloads for object, file, and HDFS into a single, unified system.  You can manage both traditional and ‘next-gen’ or ‘cloud-native’ applications on a platform that spans geographies and acts as a single logical resource.  Just like a Swiss army knife, ECS can maximize capacity by packing a lot into a tiny space.  ECS Appliance can squeeze in sixty 8TB drives into a standard 4U DAE with up to 4PB of storage in single rack, for a highly dense platform with a very economical data center footprint.

(more…)

Breakfast with ECS: Doubling Down on Docker

Corey O'Connor

Senior Product Marketing Manager at Dell EMC² ETD

Welcome to another edition of Breakfast with ECS, a series where we take a look at issues related to cloud storage and ECS (Elastic Cloud Storage), EMC’s cloud-scale storage platform.

Unless you’ve been living under a rock I’m sure you’ve heard of Docker at this point. If you haven’t, it’s time to dust yourself off and understand that Docker containers will wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in. Genius right? Docker_1

The usage of containers has been around quite some time now, but the extra juice worth squeezing came from Docker’s ability to provide total isolation of resources to package and automate applications more effectively than ever before.  Docker provides system administrators and developers the ability to package any kind of software with all its dependencies into a container. Simply put, this resource efficiency standardizes each container and promotes massive scalability – this plugs in very nicely for cloud-scale, geo-distributed systems such as EMC’s Elastic Cloud Storage (ECS). In the early stages of product development, EMC took an early bet on Docker containers and it certainly has proved to payoff.

(more…)

Follow Dell EMC

Categories

Archives

Connect with us on Twitter