Posts Tagged ‘CloudPools’

Your Data Lake Is More Powerful and Easier to Operate with New Dell EMC Isilon Products

Karthik Ramamurthy

Director Product Management
Isilon Storage Division at Dell EMC

Earlier this year Dell EMC released a suite of Isilon products designed to enable your company’s data lake journey. Together IsilonSD Edge, Isilon OneFS 8.0, and Isilon CloudPools transformed the way your organization stores and uses data by harnessing the power of the data lake. Today we are pleased to announce all three of these products have been updated and further enhanced to make your data lake even more powerful and easier to operate from edge to core to cloud.

Starting with the release of OneFS 8.0.1

OneFS 8.0.1 builds on the powerful platform provided by OneFS 8.0 released in February 2016. The intent of this newest release is to provide features important to unique customer datacenter workflows, enhance usability and manageability of OneFS clusters. In addition, OneFS 8.0.1 is the first release that takes full advantage of the non-disruptive upgrade and rollback framework introduced in OneFS 8.0.

Let’s review some of the most compelling features of this software release.

Improved Management, Monitoring, Security and Performance for Hadoop on Isilon

Expanding on the Data Lake, one of the focus areas of this new release was increasing the scope and usefulness of our integration with leading Hadoop management tools. OneFS 8.0.1 delivers support for and integration with Apache Ambari 2.4 and Ranger. A single management point now allows Ambari operators to seamlessly manage and monitor Hadoop clusters with OneFS as the HDFS storage layer. Ranger is an important security management tool for Hadoop.  These Ambari and Ranger integration features benefit all customers using Hortonworks and ODP-I compliant Hadoop distributions with OneFS.

Additionally OneFS 8.0.1 adds new features including Kerberos encryption to secure and encrypt data between HDFS clients and OneFS. In addition, Datanode load balancing avoids overloading nodes and increases cluster resilience. OneFS 8.0.1 also supports the following HFDS distributions: Hortonworks HDP 2.5, Cloudera CDH 5.8.0, and IBM Open Platform (IOP) 4.1.

Introducing Scale-Out NAS with SEC Compliance and Asynchronous Replication for Disaster Recovery

With OneFS 8.0.1, Isilon becomes the first and only Scale-Out NAS vendor that offers SEC-17a4 compliance via SmartLock Compliance Mode combined with the asynchronous replication to secondary or standby clusters via SyncIQ. This powerful combination means companies that must comply with SEC-17a4 are no longer caught in a choice between compliance and data recovery – with OneFS 8.0.1 they have both!

Storage Efficiency Designed for the Healthcare Diagnostic Imaging Needs

For many years PACS (Picture Archiving and Communication System) applications diagnostic imaging data was stored in large “container” files for maximum storage efficiency. In recent years, the way referring physicians’ access individual diagnostic images changed and, as a result, the methods used to store diagnostic imaging files had to change as well. OneFS 8.0.1 has a new storage efficiency feature specifically designed for the Healthcare PACS archive market to provide significantly improved storage efficiency for diagnostic imaging files.  Isilon customers can expect to see storage efficiency similar to OneFS’s large file storage efficiency for diagnostic imaging files when using this feature.   you leverage Isilon to store your PACS application data you will want to talk with your sales representative and learn more about this new feature.

Upgrade with Confidence

OneFS 8.0, released in February 2016, provided the framework for non-disruptive upgrades for all supported upgrades going forward and the addition of release rollback. OneFS 8.0.1 is the first OneFS release that you will be able to test and validate and, if needed, rollback to the previously installed 8.0.x release. This means that you can non-disruptively upgrade to 8.0.1, without impacting users or applications! You will be able to upgrade sets of nodes or the entire cluster for your testing and validation and then, once complete, you decide to commit the upgrade or rollback to the prior release. Once committed to OneFS 8.0.1, future upgrades will be even easier and more transparent with the ability to view an estimate of how long an upgrade will take to complete and transparency of the upgrade process. The WebUI was enhanced to make upgrade management even easier than before.

Manage Performance Resources like Never Before

Even more exciting is the new Performance Resource Management framework introduced in OneFS 8.0.1. This framework is the start of a revolutionary scale-out NAS performance management system. In OneFS 8.0.1 you will be able to obtain and view statistics on the performance resources (CPU, operations, data read, data written, etc.) for OneFS jobs and services. This will allow you to identify quickly if a particular job or service may be the cause of performance issues. These statistics are available via the CLI, Platform API and can be visualized with InsightIQ 4.1. In future releases these capabilities will be expanded to clients, IP addresses, users, protocols and !

These are just some of the new features OneFS 8.0.1 has to offer. OneFS 8.0.1 improves on our support for MAC OS clients, SMB, audit, NDMP and data migrations, to name a few other areas.  The white paper, Technical Overview of New and Improved Features of EMC Isilon OneFS 8.0., provides additional details on these and other new and improved features in OneFS 8.0.1

Isilon SD Edge Management Server version 1.0.1

This July EMC released a new version of IsilonSD Edge Management Server. Version 1.0.1 provides support for VMware ESX 6.0 in addition to previously supported ESX versions. This management server also enables monitoring of the IsilonSD Edge Clusters via EMC’s Secure Remote Support (ESRS) server and tools.

Isilon CloudPools Just Got Easier to Manage

OneFS 8.0.1 provides improved flexibility for CloudPools deployments in the enterprise with the introduction of proxy support. This allows administrators to specify one or more proxy servers between the Isilon cluster and your cloud provider of .

The Data Lake Journey is Just Beginning!

OneFS 8.0.1 is an important step on the data lake journey; however, you can rest assured we are not stopping here! Look forward to amazing new hardware and software features in coming releases as we build on the Performance Resource Management Framework, provide more workload specific enhancements to address our customers’ needs and deliver new levels of supportability, serviceability, scale and performance.   Don’t wait, upgrade .  Click here to download OneFS 8.0.1.

Cloud Computing and EDA – Are we there yet?

Lawrence Vivolo

Sr. Business Development Manager at EMC²

Cloud 9Today anything associated with “Cloud” is all the rage.  In fact, depending on your cellular service provider, you’re probably already using cloud storage to back up your e-mail, pictures, texts, etc. on your cell phone. (I realized this when I got spammed with “you’re out of cloud space – time to buy more” messages). Major companies that offer cloud-based solutions (servers, storage, infrastructure, applications, management, etc.) include Microsoft, Google, Amazon, Rackspace, Dropbox, EMC and others. For those that don’t know the subtleties of Cloud, and the terms, like Public vs Private vs Hybrid vs Funnel, and why some are better suited for EDA, I thought I’d give you some highlights.

Let’s start with the obvious – what is “Cloud”? Cloud computing is a collection of resources which can include servers (for computing), storage, applications, infrastructure (ex: networking) and even services (management, backups, etc.). Public clouds are simply clouds that are made available by 3rd-parties and are shared resources. Being shared is often advertised as a key advantage of public cloud – because the resources are shared, so is the cost. These shared resources can also expand and contract as needs change, allowing companies to precisely balance need with availability.  Back in 2011, Synopsys, a leading EDA company, was promoting this as a means to address peak EDA resource demand [1].

Unfortunately, public cloud has some drawbacks.  The predictability of storage cost is one. Though public cloud appears very affordable at first glance, most providers charge for the movement of data to and from their cloud, which can exceed the actual costs to store the data.  This can be further compounded when data is needed worldwide as it may need to be copied to multiple regions for performance and redundancy purposes. With semiconductor design, these charges can be significant, since many EDA programs generate lots of data.

Perhaps the greatest drawback to EDA adoption of public cloud is the realization that your data might be sitting on physical compute and/or storage resources that are being shared with someone else’s data.  That doesn’t mean you can see other’s data. Access is restricted via OS policy and other security measures. Yet that does create a potential path for unauthorized access. As a result, most semiconductor companies have not been willing to risk the potential to have their most important “golden jewels” (their IP) hacked and stolen from a public cloud environment. Security has improved since 2011, however, and some companies are considering cloud for long-term archiving of non-critical data as well as some less business critical IP.

Private cloud avoids these drawbacks, as it isolates the physical infrastructure – including hardware, storage and networking – from all other users. Your own company’s on-premise hardware is typically a private cloud, even though, increasingly, some of that “walled-off” infrastructure is itself located off-premise and/or owned and managed by a 3rd party. While physical and network isolation reduce the security concerns, they also eliminates some of the flexibility. The number of servers available can’t be increased or decreased with a single key-click to accommodate peak demand changes, at least not without upfront planning and additional costs.

Hybrid cloud is another common term – which simply means a combination of public and private clouds.

In the world of semiconductor design, private cloud as a service has been available for some time and is offered in various forms by several EDA companies today. Cadence® Design Systems, for example, offers both Hosted Design Solutions [2], which includes HW, SW and IT infrastructure, and QuickCycles® Service which offers on-site or remote access to Palladium emulation and simulation acceleration resources [3]. Hybrid cloud is also starting to gain interest, where non-critical data that’s infrequently accessed can be stored with minimal transport costs.

The public cloud market is changing constantly and as time progresses new improvements may arise that make it more appealing to EDA. A challenge of IT administrators today is meeting today’s growing infrastructure needs while avoiding investments that are incompatible with future cloud migrations. This is where you need to hedge your bets and chose a platform that delivers the performance and flexibility EDA companies require, yet enables easy migration from private to hybrid—or even public cloud. EMC’s Isilon, for example, is an EDA-proven high performance network-attached storage platform that provides native connectivity to the most popular public cloud providers, including Amazon Web Services, Microsoft Azure and EMC’s Virtustream.

Not only does native cloud support future-proof today’s storage investment, it makes the migration seamless – thanks to its single point of management that encompasses private, hybrid and public cloud deployments. EMC Isilon supports a feature called CloudPools, which transparently extends an Isilon storage pool into cloud infrastructures. With CloudPools your company’s critical data can remain on-premise yet less critical, rarely accessed data can be encrypted securely and archived automatically and transparently onto the cloud. Isilon can also be configured to archive your business-critical data (IP) to lower-cost on-premise media.  This combination saves budget and keeps more high-performance storage space available locally for your critical EDA jobs.

Semiconductor companies and EDA vendors have had their eyes on public cloud for many years. While significant concerns over security continue to slow adoption, technology continues to evolve. Whether your company ultimately sticks with private cloud, or migrates seamlessly to hybrid or public cloud in the future depends on decisions you make today. The key is to focus on flexibility, and not let fear cloud your judgment.

[1] EDA in the Clouds: Myth Busting: https://www.synopsys.com/Company/Publications/SynopsysInsight/Pages/Art6-Clouds-IssQ2-11.aspx?cmp=Insight-I2-2011-Art6

[2] Cadence Design Systems Hosted Design Services: http://www.cadence.com/services/hds/Pages/Default.aspx

[3] Cadence Design System QuickCycles Service: http://www.cadence.com/products/sd/quickcycles/pages/default.aspx

Breakfast with ECS: The Swiss Army Knife of Cloud Solutions

Corey O'Connor

Senior Product Marketing Manager at Dell EMC² ETD

Welcome to another edition of Breakfast with ECS, a series where we take a look at issues related to cloud storage and ECS (Elastic Cloud Storage), EMC’s cloud-scale storage platform.

ECS Cloud Enabling ToolsA Swiss army knife is a multi-layered tool equipped with a variety of attachments that can serve up many different functions. When first introduced in the late 1880s, it revolutionized the way soldiers performed their daily tasks – anything from disassembling service rifles to opening up canned rations in the field.  Fast forward to 2016, the use of the Swiss army knife may have changed quite a bit but the initial concept of consolidating various components into a single multi-purpose tool has certainly influenced organizations and industries across the world.

EMC’s Elastic Cloud Storage (ECS) is without question the Swiss army knife for cloud solutions.  ECS revolutionizes storage management by consolidating varied workloads for object, file, and HDFS into a single, unified system.  You can manage both traditional and ‘next-gen’ or ‘cloud-native’ applications on a platform that spans geographies and acts as a single logical resource.  Just like a Swiss army knife, ECS can maximize capacity by packing a lot into a tiny space.  ECS Appliance can squeeze in sixty 8TB drives into a standard 4U DAE with up to 4PB of storage in single rack, for a highly dense platform with a very economical data center footprint.

(more…)

IABM Selects EMC as a Game Changer for Storage at NAB Show

Tom "TV" Burns

CTO, Media & Entertainment at EMC

Hello from the NAB Show in Las Vegas! It’s been a terrific couple of days so far and the team is having a great time connecting with our customers, partners, prospects and others in the broadcast and media technology community.

game_changer_logo_for_web_bannerWe’ve got some exciting news to share from the show – the International Association of Broadcasting Manufacturers (IABM) presented EMC with the 2016 Game Changer Award in the storage category for Isilon CloudPools and IsilonSD Edge. These new products help media organizations integrate cloud storage into their current workflows and easily enable new remote office productions.

The media industry is built on change, and today the media business of creation and delivery is no exception. As higher resolution formats and new delivery methods create turbulence in the industry, media professionals are looking for best-of-breed technology that will future-proof their workflows and transform their business to meet tomorrow’s needs.

From day one, EMC has thought differently about the media industry: we’ve always believed in challenging the status quo of how storage is used in creation, processing and delivery of media. With Isilon storage we’ve done that by creating simple, easy to manage, high performance, and cost-effective tiers of storage that are all in one flexible pool.  A single pool of storage lets artists and editors easily share assets across the workflow, while providing a single point of storage management throughout the life cycle of the storage infrastructure.

EMC Isilon SD Edge graphic

Perhaps more importantly, from the ground up, we’ve future-proofed our products by design. The Isilon storage platform can be incrementally and modularly grown as business needs change or technology requires component replacement. There’s no need to acquire the infrastructure today based on predictions of what will happen in the next three years—simply grow the storage system in increments that follow business cycles. If a portion of the infrastructure is no longer cost-effective, gradually replace those portions with new technology—seamlessly and cost-effectively.

The judging panel of 40 independent industry professionals described the products as “a development of a well-established professional, high performance media storage system [that] enables a hybrid approach in migrating to cloud-based storage.”

This Game Changer award is a testament to our team’s dedication and hard work to provide thousands of organizations in the media and entertainment industries the most effective storage solutions. From all of us at EMC, we want to extend our thanks and appreciation to the IABM for this recognition.

Isilon – A Storage Solution that evolves with Next-Generation Sequencing

Biswajit Mishra

Sr. Product Marketing Manager, Life Sciences at EMC²

EMC Isilon Life SciencesThe creation and understanding of the binary code was the cornerstone of the development of the modern digital technology age. The language of 0’s and 1’s opened the door to many new technologies including the internet, modern media platforms, telecommunications, smartphone applications and more. For Life Sciences, something similar happened when we unlocked the genetic code. The language of A’s, T’s, C’s and G’s could explain all life forms and led to an explosion in the development of products and services in the areas of pharmaceuticals, healthcare, and agriculture.

The Beginning

In 1976, a single-stranded RNA virus named “Bacteriophage MS2” that infects E coli became the first genome with 3569 bases to be completely sequenced.  Then in 1995, the Institute for Genomic Research performed the completed genome sequencing of “Haemophilus influenza” – the first full genome sequencing of a living organism with 1.8M base pairs (Mb). But when the Human Genome Project sequenced the entire human genome with 3.2 billion base pairs in 2001, it opened a whole new world of possibilities.

Demands on IT divisions aka “The NGS Data Challenge”

The field of Life Sciences continues to remain at the cutting edge of scientific research with improvements in technologies such as next generation sequencing (NGS), proteomics, etc. NGS instrument such as the Illumina HiSeq X Ten Systems can now sequence over 18,000 human genomes per year which is unbelievable when you consider that it took more than a decade to sequence the entire human genome as part of the Human Genome Project. And, keep in mind each human genome needs about 100 GB of storage, so an Illumina HiSeq X Ten Systems needs about 1.8 PB of storage per year. The improvements, particularly in NGS technology and the growing applications of NGS means that more and more organizations are using NGS services and generating data at a rate as never before. And, in this day and age of flattening IT budgets, the IT departments in these organizations cannot keep up with the pace of data generation.

For most organizations performing NGS, the IT department has to deliver on the following fronts:

  • Meet the performance and capacity needs: Most Life Sciences workflows containing NGS include data generation, analysis, and archiving stages. Each of these stages has unique capacity and performance needs. Companies performing NGS usually require sequencing to run 24/7 and it is left to the IT department to ensure that the right production performance and storage tiers are available as needed.
  • Plan for the future: For IT departments with a longer IT procurement cycle, it is a struggle to ensure that the IT purchase process keeps up with the storage needs of the exponentially growing data produced from NGS processes. IT divisions have to ensure that they plan for today as well as the future to meet the demanding customer SLAs.
  • Cost efficient solutions: IT budget growth typically is nowhere close to the rate of the data growth. This means that IT departments have to ensure that they are smart and cost efficient with their storage solutions.

Isilon, a trusted partner for Life Sciences organizations – now and in the future

EMC Isilon is a leader and trusted partner for hundreds of Life Sciences organizations worldwide including leading genome centers, pharmaceutical companies, and academic research centers. Isilon provides a highly available, and reliable, single file system, OneFS, for Life Sciences workflows and consolidates all the workflows into a single, scalable volume for the ease of management. Isilon can host multiple types of nodes – S, X, NL and HD – in a single cluster. S and X nodes are ideal for high performance workflows such as mapping and alignment of NGS data while NL and HD nodes provide high-density, low cost storage of raw data and analysis results for archive needs. Isilon SmartFail and Auto Balance features also ensure that data is protected across the entire cluster.

Isilon gets better

Recently EMC Isilon announced Data Lake 2.0 – including OneFS 8.0, Isilon SD Edge and Isilon CloudPools – to extend the data lake from the data center to edge locations, including remote and branch offices and to both public and private cloud.

IsilonSD EdgeThe Data Lake 2.0 has massive benefits for Life Science organizations using Isilon. With OneFS 8.0 organizations can now achieve non-disruptive upgrade and rollback capabilities to ensure continuous operations at all times. IsilonSD Edge is a software-defined storage solution that can be deployed on commodity hardware on remote research facilities. This can provide Life Science organizations with the ease of management to connect geographically dispersed research facilities in a cost efficient manner.

Isilon CloudPools is a game changer for organizations Isilon CloudPoolsperforming NGS, providing a cost efficient way of archiving data allowing the IT department to make the most of their budget. Isilon CloudPools offers a transparent, policy-based, automated storage tiering of data from an on-premise Isilon cluster to either a private cloud based on EMC Elastic Cloud Storage (ECS), another Isilon cluster or into a public cloud like AWS or Azure for long term storage of Life Science research data and other archiving needs. By moving the cold archive data to either a public or private cloud, CloudPools helps the IT departments to free up space on production environments and ensure that the NGS analysis pipelines are not affected due to a lack of high performance storage. This also allows IT departments to deliver on customer SLAs while at the same time, reducing costs.

Improve Life Sciences Data Management

We understand the need of Life Science organizations for a versatile storage solution to balance their capacity and performance requirements. That’s why we continue to evolve our Isilon platform to meet the needs of our Life Science customers. New innovative features like EMC Isilon OneFS 8.0, IsilonSD Edge and Isilon CloudPools makes sure that our Life Sciences customers have a proven solution to run NGS non-stop and achieve faster time to insights. With EMC Isilon, Life Sciences organizations have a trusted partner to handle their NGS workloads not only today but in the future too.

For more information on how EMC Emerging Technology Division can help your Life Science organization click here.

 

Get first access to our LifeScience Solutions

Is the Media & Entertainment industry in the middle of a major transformation?

Tom "TV" Burns

CTO, Media & Entertainment at EMC

Media Companies in 2016 find themselves “in the middle of things”:

  • a “peak TV” environment in which cable channels are bursting with high-quality scripted episodic programming
  • an on-demand subscriber environment that has revolutionized the way consumers find and watch entertainment
  • immersive environments which have changed the grammar of visual storytelling

“We can’t really even call it television anymore…”
When I took this gig as CTO, Media & Entertainment for EMC, one thing that intrigued me about the role was how EMC thought differently about the media industry: they focused on helping media companies create and deliver great stories. We do this by leveraging the evolving concepts from traditional enterprise IT such as software-defined storage and converged infrastructure to provide adaptable carrier-grade solutions and a future-proof infrastructure to their media customers. This year is no different as we’re introducing a host of new features to our flagship Isilon product—features that expand a media organization’s reach from “Edge to Core to Cloud.” (I’ll talk about how this relates to media organizations in a sec…)

“The software-defined media facility”
Broadcasters, advertiser- and subscriber-supported OTT providers, content creators, aggregators and the entire ecosystem of partners and suppliers are really in the “Entertainment Services Delivery” market. Technology infrastructures like software-defined storage and software-defined networking will allow broadcasters to make changes to their workflows programmatically. This is the “new normal” – of adding or migrating new services in days rather than months.

As our media customers have responded to this re-definition of their business, we’ve introduced new features that let them do things they’ve never done before. For example, last year we introduced the concept of a “Media Lake”—essentially a single, high-performance shared resource that let media organizations consolidate their data into a central repository. Rather than copying your data to some other silo, transforming it and copying it back – the idea is to leave your data in one place and execute the value-creation operations (whether it’s encoding, archiving, creative editorial, analytics or what have you) while you leave the data in place – it saves you time, enables collaboration and reduces the possibility of error.

Isilon Media and Entertainment Data Lake
This year we’ve doubled our investment in the idea of a media data lake—announcing “Media Lake 2.0,” which gives our media customers new capabilities that allow productions to extend to “edge” (smaller or remote) locations, optimize their core media operations, and leverage public/private/hybrid clouds.

The new Isilon software upgrades and products being introduced include:

OneFS 8.0—improves availability failover reliability for Isilon storage clusters.  OneFS 8.0 includes multiple features, but the ones that are critical for Media & Entertainment include non-disruptive operating system upgrades, upgrade rollback, and non-disruptive operations.  In addition, OneFS 8.0 incorporates support for SMB3.0 Continuous Availability for increased reliability and availability for Windows clients.

One of our long-standing partners Jeff Snyder Nexio Mediaand premier media industry vendor, Imagine Communications, recently tested OneFS 8.0 in their lab and at one of their customers’ sites.  As a  global leader in video and advertising solutions—nearly half of the world’s video channels traverse through more than three million Imagine Communications products deployed across 185 countries.

That’s why we look to Imagine to give us feedback on our solutions as they definitely know how to stress new technology.  For OneFS 8.0 they were specifically interested in the non-disruptive upgrades and the improvements in SMB 3.0.

In testing, Imagine found the scale-out and failover from EMC Isilon makes it one of the “best” NAS solutions they’ve tested. The continuous availability options (the ability to upgrade or update on-the-fly) and improvements in the SMB protocol gave them functionality and bandwidth “they couldn’t get with other NAS products.”

IsilonSD Edge—provides a software-defined storage solution that allows media companies to quickly and seamlessly gain access to production and talent wherever great stories are created and captured. In addition, as a software-defined storage solution, IsilonSD Edge runs on commodity hardware in a virtual environment—substantially reducing costs and increasing agility. IsilonSD Edge extends the Media Data Lake across geographies, allowing “follow-the-sun” content creation and delivery from remote and branch facilities. Media data lake synchronization with remote assets reduces inefficiencies and accelerates production times by sharing critical production media assets.

EMC Isilon CloudPools—enables Isilon media customers to seamlessly archive or tier assets from their Isilon cluster to an in-house (private) cloud based on EMC ECS, or a choice of public cloud providers, or a combination of both. This gives a media organization the flexibility and simplicity to dynamically expand beyond their current capacity and archive storage into any cloud-based solution. With CloudPools, data that is transmitted to the cloud is encrypted for security purposes and compressed to optimize the network bandwidth usage.

“Ch-Ch-Ch-Ch-Changes…”
The industry continues to change at a dramatic pace, and we realize our customers need to not only adapt and transform but also profit from their efforts.  That’s why we continue to challenge the status quo for our media customers.   Whether it’s jointly developing end-to-end workflows solutions with great industry partners like Imagine Communications, or upgrading the Isilon platform with innovative features like the CloudPools and IsilonSD Edge, our goal is to give our customers a foundation for future-proofing their business and workflow needs.

@TVBurns

Categories

Archives

Connect with us on Twitter