What Happens When You Poke A Large Bear (NetApp SnapMirror) And An Aggressive Wolf (EMC RecoverPoint)?

By | Backup, Clariion, Data Loss Prevention, Deduplication, Disaster Recovery, EMC, NetApp, Replication, Security, Storage | No Comments

This month I will take an objective look at two competitive data replication technologies – NetApp SnapMirror and EMC RecoverPoint. My intent is not to create a technology war, but I do realize that I am poking a rather large bear and an aggressive wolf with a sharp stick.

A quick review of both technologies:


  • NetApp’s controller based replication technology.
  • Leverages the snapshot technology that is fundamentally part of the WAFL file system.
  • Establishes a baseline image, copies it to a remote (or partner local) filer and then updates it incrementally in a semi-synchronous or asynchronous (scheduled) fashion.


  • EMC’s heterogeneous fabric layer journaled replication technology.
  • Leverages a splitter driver at the array controller, fabric switch, and/or host layer to split writes from a LUN or group of LUNs to a replication appliance cluster.
  • The split writes are written to a journal and then applied to the target volume(s) while preserving write order fidelity.

SnapMirror consistency is based on the volume or qtree being replicated. If the volume contains multiple qtrees or LUNs, those will be replicated in a consistent fashion. In order to get multiple volumes replicated in a consistent fashion, you will need to quiesce the applications or hosts accessing each of the volumes and then take snapshots of all the volumes and then SnapMirror those snapshots. An effective way to automate this process is leveraging SnapManager.

After the initial synchronization SnapMirror targets are accessible as read-only. This provides an effective source volume for backups to disk (SnapVault) or tape. The targets are not read/write accessible though, unless the SnapMirror relationship is broken or FlexClone is leveraged to make a read/write copy of the target. The granularity of the replication and recovery is based off a schedule (standard SnapMirror) or in a semi-synchronous continual replication.

When failing over, the SnapMirror relationship is simply broken and the volume is brought online. This makes DR failover testing and even site-to-site migrations a fairly simple task. I’ve found that many people use this functionality as much for migration as data protection or Disaster Recovery. Failing back to a production site is simply a matter of off-lining the original source, reversing the replication, and then failing it back once complete.

In terms of interface, SnapMirror is traditionally managed through configuration files and the CLI. However, the latest version of ONCommand System Manager includes an intuitive easy to use interface for setting up and managing SnapMirror Connections and relationships.

RecoverPoint is like TIVO® for block storage. It continuously records incoming write changes to individual LUNs or groups of LUNs in a logical container aptly called a consistency group. The writes are tracked by a splitter driver that can exist on the source host, in the fabric switch or on a Clariion (VNX) or Symmetrix (VMAXe only today) array. The host splitter driver enables replication between non-EMC and EMC arrays (Check ESM for latest support notes).

The split write IO with RecoverPoint is sent to a cluster of appliances that package, compress and de-duplicate the data, then sends it over a WAN IP link or local fibre channel link. The target RecoverPoint Appliance then writes the data to the journal. The journaled writes are applied to the target volume as time and system resources permit and are retained as long as there is capacity in the journal volume in order to be able to rewind the LUN(s) in the consistency group to any point in time retained.

In addition to remote replication, RecoverPoint can also replicate to local storage. This option is available as a standalone feature or in conjunction with remote replication.

RecoverPoint has a standalone Java application that can be used to manage all of the configuration and operational features. There is also integration for management of consistency groups by Microsoft Cluster Services and VMWare Site Recovery Manager. For application consistent “snapshots” (RecoverPoint calls them “bookmarks”) EMC Replication Manager or the KVSS command line utilities can be leveraged. Recently a “light” version of the management tool has been integrated into the Clariion/VNX Unisphere management suite.

So, sharpening up the stick … NetApp SnapMirror is a simple to use tool that leverages the strengths of the WAFL architecture to replicate NetApp volumes (file systems) and update them either continuously or on a scheduled basis using the built-in snapshot technology. Recent enhancements to the System Manager have made it much simpler to use, but it is limited to NetApp controllers. It can replicate SAN volumes (iSCSI or FC LUNs) in NetApp environments – as they are essentially single files within a Volume or qtree.

RecoverPoint is a block-based SAN replication tool that splits writes and can recover to any point in time which exists in the journal volume. It is not built into the array, but is a separate appliance that exists in the fabric and leverages array, and fabric or host based splitters. I would make the case that RecoverPoint is a much more sophisticated block-based replication tool that provides a finer level of recoverable granularity, at the expense of being more complicated.

 Photo Credit: madcowk

Isilon Infiniband: Go Big Storage Or Go Home

By | Backup, Isilon, Replication, Storage | No Comments

It wasn’t so long ago that a 1G disk drive made your computer the envy of every other nerd on the block. Now the tech enthusiast sees green when you have 4 to 10 TB. This is just at home!

Currently in the enterprise space we are seeing regular caches of data well in excess of 100TB. Oddly, the file systems holding this kind of data struggle when the files get much above 16TB in size. So we start carving our repositories into all sorts of folders. Sometimes this makes sense, but more often than not it is to get around the technology limitations of the storage subsystem being used.

The ideal storage system would be able to scale in terms of performance and capacity. This solution could mix and match SAS, SSD and near line SAS technologies so the right data is on the correct tier. Snapshots and replication, native, of course. The solution would allow us to create folders – but not be bound to them. We would be able to slide data from one folder or file system or from one protocol to another to another without having to reorganize the storage. There would be no taking losses due to repartitioning or any loss of performance due to hot spots. This storage would have internal data protection which doesn’t require RAID sets or aggregates that require bunches of parity drives or hot spares. More drives would just mean more storage efficiency. How awesome would it be to literally have 1000’s of drives working together for a single workload with petabytes of continuous space under your home drive or operations folder or development folder?

No crazy ideas here.
Not an odd desire.
Not a unique problem.

Yes, there are ways to build this “ideal” storage system described above yourself – if you have time for a serious science project.

But for the professional IT shop, who are looking for something that is fully supported, shrink wrapped, and ready to run – ENTER ISILON – stacks of nodes with lots of drives and a really smart OS, purpose built for serving (a lot of) files.

I have always seen the benefits of node based architectures, but also the downfall of the “backplane on a LAN” approach. Any kind of storage balancing or motion around the array could crush the network and then starve the end users coming over the same network, or in some cases; the same ports. Isilon sidesteps this common issue and threw an Infiniband network behind its nodes. Now we are talking about a backplane. The best part is that it is expandable as the number of ports on the switch. As the switch technology develops, so does the size and performance of the entire array.

OK, so we all don’t need to go this big, but what a great way to get there if you did. Crazy scalability, crazy performance, all shrink wrapped with support. I just have to figure out how to get one for my home theater…

Photo Credit: ChrisDag

Disaster Recovery, Past vs. Present: New Tech Offers Lean-and-Mean DR Made Easier & Less Expensive

By | Disaster Recovery, Replication | No Comments

Not so long ago, disaster recovery collocation sites were a topic that everyone wanted to talk about … but nobody wanted to invest in. This was largely because the state of the technology left these sites sitting cold—it was simply the most expensive insurance policy one could think of. With the advent of dramatically dropping storage costs, improving costs of WAN connectivity, a wealth of effective and robust replication technologies, and (most importantly) the abstraction of the application from the hardware, we have a new game on our hands.

[image align=”center” width=”400″ height=”267″][/image]

Ye Olde Way:

WAN pipe: BIG, fast, low latency and EXPENSIVE. Replication technologies were not tolerant of falling too far behind. The penalty for falling behind was often a complete resynchronization of both sites sometimes even requiring outages.

Servers: Exact replicas on each side of the solution. There is nothing so thrilling to the CFO as buying servers so they can depreciate as they sit, idle, waiting for the day when the hot site failed. Hopefully everyone remembered to update BIOS code on both sites, otherwise the recovery process was going to be even more painful.

Applications: Remember to do everything twice! Any and every change had to be done again at the collocation site or the recovery would likely fail. By the way, every manual detail counts. Your documentation should rival NASA flight plans to ensure the right change happens in the right order. Aren’t “patch Tuesdays” fun now!

Storage: Expensive since consolidated storage was new and pricey. Host-based replication took up so many resources that it crippled production and often caused more issues with the server than was worth the risk. Array-based replication strategies at the time were young and filled with holes. Early arrays only spoke Fibre Channel, so we had the added pleasure of converting Fibre Channel to IP with pricey bridges.

Process: Every application was started differently and requires a subject matter expert to ensure success. No shortage of finger pointing to see whose fault it is that the application was grumpy on restore. Testing these solutions required the entire team offsite for days and sometimes weeks to achieve only partial success. is looking like a better DR strategy for IT at this point…


WAN pipe: Not as big, fast, low latency and much less expensive. Replication technologies have come a long way providing in-band compression and, due to the much larger processors and faster disk subsystems, even host-based replication strategies are viable for all but the busiest apps. But again, with the falling cost of array-based replication, you may not need host-based replication.

Servers: Keep the CPUs in roughly the same family and you are good to go. I am flexible to choose different manufacturers and specifications to serve my workload best without being overly concerned about messing up my production or disaster recovery farm’s viability. Even better, I can be running test, dev, and even some production apps over at the secondary site while I am waiting for the big bad event. Even your CFO can get on board with that.

Applications: Thanks to server virtualization, I ship the entire image of the server (rather than just data) as replication continues; whatever I do on one side is automatically on the other side. Most importantly, the image is a perfect replica of production. Go one step further and create snapshots before you work on the application in production and disaster recovery, in case you need to fail over or fail backwards to rollback a misbehaving update.

Storage: Remarkably cost effective, especially in the light of solid state drives and connectivity options. Remarkable speed, flexibility, and features are available from a wealth of manufacturers. Support is still key, but even the largest manufacturers are working harder than ever to win your business with huge bang-for-buck solutions. Even small shops can afford storage solutions that ran $1M+ only 5 years ago. Just these “entry level” solutions are doing more than anything on the market back then.

Process: We get to pick how automated we want to go. Solutions exist to detect outages and restart entire environments automatically. Network protocols exist to flatten networks between sites. “Entry level” replication solutions will automatically register clones of machines at offsite locations for simple click-to-start recovery. What does this all mean to the IT team? Test well and test often. Administrators can start machines in the remote sites without affecting production, test to their hearts’ content, hand off to the application owners for testing, and then reclaim the workspace in minutes, instead of days.

We play a new game today. Lots of great options are out there are providing immense value. Do your homework and find the right mix for your infrastructure.

Interview: Net App Storage Powers New Business for Chicago-based Web Services Provider (Video)

By | Backup, NetApp, Replication, Storage | No Comments

Earlier this month, our Storage Practice Manager, Shawn Wagner, sat down with Karl Zimmerman, President and Founder of Steadfast Networks, to discuss the NetApp storage and backup solution we recently designed and deployed for their business’s datacenter in Chicago.

With the cameras rolling, their conversation addressed the reasons for deciding on NetApp, relative to other manufacturer’s technologies or developing a in-house solution, and how that decision and the NetApp technology is enabling Steadfast to drive new business and offer expanded solutions to their existing customers.


What was the primary business driver for you to move forward with the NetApp technology?

Well, we’ve been looking to build more of a sort of cloud solution—we don’t necessarily like to call it “cloud,” but that’s basically the solution. And we needed to back that up with a very redundant storage solution that we knew we could trust. We generally like building things in house; we’re very hands on ourselves. But in house, we couldn’t really build a solution that developed the performance and reliability that we were able to get with NetApp, and NetApp was able to do it for us at an affordable price.

With respect to IDS in general, how do you see our partnership moving forward?

With IDS specifically, we appreciate that it’s someone who’s always there we can contact. There are always people we can get in touch if we need help on certain aspects of the NetApp infrastructure that we don’t understand—we know that support is there. If we run into problems, which we have already, you guys are there to help us through those. It’s nice just knowing we have the strong, reliable partner that we can depend on.

I’d like to end on just a couple of forward thinking questions, one of which is where do you see your business going over the next 18 to 24 months now having made this investment in NetApp?

We’ve seen a higher demand in general for cloud applications where people are moving things away from their own internal datacenters or their own in-office solutions, and moving them to a datacenter environment—so, of course, it could be accessed across their entire company, or have the additional ability of scale. So this NetApp solution allows us a lot to focus on that segment and we can then offer the scalability of a virtualized environment, while having the cost savings and additional processing power you can achieve with dedicated systems and collocation as well. With NetApp, we now serve a wide variety of needs across a broad spectrum of customers at basically whatever price point or scalability they need. And that of course is then all backed up with a NetApp SAN for the storage side—it’s a common storage system for all these systems. It lets us easily scale and expand for whatever the customer might need.

And one final question: in respect to business continuity and disaster recovery, what possibilities have opened up for you and your clients with NetApp?

NetApp is certainly helpful with the snapshots and also being able to replicate the data to multiple sites; it makes it a lot easier, because we do have multiple data centers. Being able to replicate that data and back it up easily with the snapshots helps us to easily integrate customer redundancy and disaster recovery plans into our existing infrastructure—it’s certainly something that the home-built solutions we were looking at doing didn’t allow us to provide so easily. And it’s certainly one of the reasons along with the reliability and the speed of the NetApp solution that led us to that decision.

“You’re Fired!” Why Snapshots + Replication (Donald) Trump Your Old Backup Strategy

By | Backup, NetApp, Replication, Storage | No Comments

Lately, the question on everyone’s mind has been: is it possible to replace your aging backup strategy with array-based snapshot and replication technology? The inevitable follow-up to that question: why is this so hard to swallow by so many of us, and why do we have a hard time accepting it? It all boils down to what we are used to and what we are comfortable with doing. Change is hard to accept and even harder to implement. I’m hoping with further explanation, I can highlight the benefits of moving away from antiquated backup technology.

First let’s delve into the traditional backup strategy:

>> Incrementals
>> Differentials
>> Nightly’s
>> Weekly’s
>> Monthly’s
>> Auto-loaders
>> Offsite
>> Retention-period

These are all terminologies we are familiar with using on a daily basis. Traditional backups are a huge pain in the $#%, but they have to be done because the business dictates it. The gist of it is this: traditional backup strategies have been around since the 90s. They cut into production hours; they take dedicated server and backup hardware; and we are lucky if the backups actually get done most of the time. Lastly, let’s be honest, when it comes time to do a restore, our fingers are crossed and we hit the RESTORE button with a hope and a prayer that things will actually work.

The industry’s fear of snapshot technology boils down to a few reasons:
[framed_box bgColor=”EFEFEF” rounded=”true”]

  • Snapshots don’t protect against drive failures.
  • Snapshots cannot be moved offsite, or offloaded onto physical media.
  • Data that is not stored centrally on my array is suspect to loss if it’s not getting backed up.
  • Too many snapshots will alter the performance of the array.
  • Snapshots are not integrated into my applications.
  • Snapshots take up too much disk space.
[/framed_box] Now, let me break down these fears and sway you towards snapshots…

Snapshots don’t protect against drive failures.
This depends on two things: 1) how your LUN’s or aggregates are carved out, and 2) you are not replicating your snapshots to a secondary array. The easiest way to overcome the drive failure is to use a RAID technology which supports more than a single drive failure at any given time.

Snapshots cannot be moved offsite, or offloaded onto physical media.
Replicating your data to a secondary array can kill two birds with one stone. It can help protect you against drive failures or total disasters on your primary array. The second bird is that certain manufacturers support NDMP, or Network Data Management Protocol, which is basically an open standard which supports offloading centrally attached storage devices directly to tape. Now why would I bring that up when I am trying to get you away from tape, well, there still is a true business case for it which your organization might not be able to get away from for long term retention.

What about my data that is not stored centrally on my array? Isn’t it suspect to loss if it is not getting backed up?
Two things to help you here, MS VSS and OSSV. For this discussion, I will spew forth about OSSV. This is a technology developed by NetApp used to offload data from a server with locally attached storage and allow snapshots to be taken at the array level. These snapshots can then be used rebuild servers, and even aid in bare metal restores if 3rd party agents are used.

Too many snapshots will alter the performance of the array.
You know, I can’t deny this point, but it also depends on the manufacturer. This boils down to the file system on the array and how snapshots are written. If snapshots are done via copy-on-write technology, the more snapshots you take and keep online, the more performance drops. We have seen up to 60% performance degradation in the field on manufacturers using copy-on-write technology. Array’s that use WAFL and a pointer based snapshot technology will see a very slight performance degradation using snapshots and only when snapshots are being saved into the hundreds.

Snapshots are not integrated into my applications.
Again, another point I cannot deny for the majority of array manufacturers. Usually to get high snapshot based RPO or RTO, you need some kind of appliance connected to the array that is specific to that application. High-five to NetApp here for their snapshot-based application integration in SQL, Oracle, Exchange and virtual environments. With a simple licensed add-on, snapshots can be used for even the most demanding RPO and RTO requirements for an organization.

Snapshots take up too much disk space.
Once again, this depends on manufacturer. If it is a copy-on-write technology, then yes, snap-shotting your array will take up a considerable amount of disk space and your snapshots kept online are severely limited. A pointer based snapshot solution will allow you to keep a tremendous amount of snapshots online at any time, while consecutively consuming very little space on your array. Think about being able to keep a years worth of snapshots online for a 10tb dataset, and only using 2Tb of space to save those snapshots.

I hope I’ve helped you to understand how snapshots can be used to replace aging tape and backup environments. Please feel free to drop a comment below if you would like to dive deeper into how snapshot-based technology can help you in your fight to replace traditional tape backup.

Photo Credit: daveoleary

Tape Sucks: Avamar 6.0 Version #tapesucksmoveon

By | Avamar, Backup, Data Loss Prevention, Deduplication, EMC, Replication, Storage, VMware | No Comments

Since its release last week, there has been a lot of buzz around the Avamar 6.0. I am going to take the liberty of reading between the lines and exploring some of the good (but gory) details.

The biggest news in this release is the DDBoost/Data Domain integration into the new Avamar Client binaries. This allows an Avamar client to send a backup dataset stream to a DataDomain system as opposed to an Avamar node or grid. Datasets that are not “dedupe friendly”(too large for Avamar to handle, or have very high change rates) are typically retained for shorter periods of time. These can be targeted to a DD array, but still managed through the same policies, backup and recovery interface.

Client types supported pertaining to this release are limited to Exchange VSS, SQL, SharePoint, Oracle and VMWare Image backups. Replication of data is Avamar to Avamar and DataDomain to DataDomain: there isn’t any mixing or cross-replication. Avamar coordinates the replication and replicates the meta-data so that it is manageable and recoverable from either side. From a licensing perspective, Avamar requires a capacity license for the Data Domain system at a significantly reduced cost per TB. DDBoost and replication licenses are also required on the Data Domain.

There is a major shift in hardware for Avamar 6.0: 

  1. The Gen4 Hardware platform was introduced with a significant increase in storage capacity.
  2. The largest nodes now support 7.8TB per node – enabling grids of up to 124TB.
  3. The new high capacity nodes are based off of the Dell R510 hardware with 12 2TB SATA drives.
  4. To speed up indexing the new 7.8TB nodes also leverage an SSD drive for the hash tables.
  5. There are also 1.3TB, 2.6TB and 3.9TB Gen4 Nodes based off of the Dell R710 hardware.
  6. All nodes use RAID1 pairs and it seems the performance hit going to RAID5 on the 3.3TB Gen3 nodes was too high.
  7. All Gen4 nodes now run SLES (SUSE Linux) for improved security.

There were several enhancements made for grid environments. Multi-node systems now leverage the ADS switches exclusively for a separate internal network that allows the grid nodes to communicate in the event of front-end network issues. There are both HA and Non-HA front end network configurations, depending on availability requirements. In terms of grid support, it appears that the non-RAIN 1X2 is no longer a supported configuration with Gen4 nodes. Also, spare nodes are now optional for Gen4 grids if you have Premium Support.

Avamar 6.0 is supported on Gen3 hardware, so existing customers can upgrade from 4.x and 5.x versions. Gen3 hardware will also remain available for upgrades to existing grids as the mixing of Gen3 and Gen4 systems in a grid is not supported. Gen3 systems will continue to run on Red Hat (RHEL 4).

Avamar 5.x introduced VStorageAPI integration for VMWare ESX 4.0 and later versions. This functionality provides changed block tracking for backup operations, but not for restores. Avamar 6.0 now provides for in-place “Rollback” restores leveraging this same technology. This reduces restore times dramatically by only restoring the blocks that changed back into an existing vm. The other key VMWare feature introduced in version 6.0 is Proxy server pooling – previously, a proxy was assigned to a datastore, but now proxy servers can be pooled for load balancing in large environments.

There were several additional client enhancements on the Microsoft front including Granular Level Recovery (GLR) support and multistreaming (1 to 6 concurrent streams) for Exchange and Sharepoint clients.

All in all, the Avamar 6.0 release provides several key new features and scales significantly further than previous versions. With the addition of Data Domain as a target, tape-less backup is quickly approaching reality.

Photo Credit: altemark

Leveraging EMC,VNX, & NFS To Work For Your VMware Environments #increasestoragecapacity

By | Deduplication, EMC, Replication, Storage, Virtualization, VMware | No Comments

Storage Benefits
NFS (Network File System) is native to UNIX and Linux file systems. Because the NFS protocol is native to UNIX and Linux, it allows the file system to be provisioned thin instead of thick, with ISCSI or fiber channel. Provisioning LUN’s or datastores thin, allows the end user to efficiently manage their NAS capacity. Users have reported a 50% increase in both capacity and usable space. 

Creating NFS datastores is a lot easier to attach to hosts than FC or ISCSI. There is no usage of HBA’s or fiber channel fabric, and all that needs to be created is a VMkernel for networking. NAS and SAN capacity can quickly become scarce if the end user can’t control the amount of storage being used, or if there are VM’s with over provisioned VMDK’s. NFS file systems can also be deduplicated, and not only are user’s saving space via thin provisioning, the VNX can track similar data and store only the changes to the file system. 

EMC and VMware’s best practice is to use deduplication on NFS exports which house ISO’s, templates and other miscellaneous tools and applications. Enabling deduplication on file systems which house VMDK’s is not a best practice due to the fact that the VMDK’s will not compress. Automatic volume manager can also stripe the NFS volumes across multiple RAID groups (assuming their array was purchased with more than just 6 drives). This increases the I/O performance of the file system and VM. Along with AVM extending the datastores, this makes the file system transparent and beneficial to VMware (assuming you are adding drive to the file system). AVM will extend the file system to the next available empty volume, meaning if you add drives to the file systems you will be increasing the performance of your virtual machines. 

Availability Benefits
Using VNX, Snapsure snapshots can be taken of the NFS snapshots and mounted anywhere in both physical and virtual environments. NFS Snapshots will allow you to mount production datastores in your virtual environment to use them for testing VM’s without affecting production data. Leveraging SnapSure will allow the end-user to keep up with certain RTO and RPO objectives. SnapSure can create 96 checkpoints and 16 writable snapshots per file system. Not to mention the ease of use Snapsure has over SnapView. Snapsure is configured at the file system level, just right-click the file system, select how many snapshots you need, add a schedule and you’re finished. 

From my experience in the field the end-user finds this process much easier than SnapView or replication manager. Using VNX, NFS will also enable the user to replicate the file system to an offsite NS4-XXX without adding any additional networking hardware. VNX Replicator allows the user to mount file systems on other sites without affecting production machines. Users can replicate up to 1024 file systems, and 256 active sessions. 

Networking Benefits
VNX datamovers can be purchased with 1 GB/s or 10 GB/s NICs. Depending on your existing infrastructure, the VNX can leverage LACP or ether channel trunks to increase the bandwidth and availability of your NFS file systems. LACP trunks enable the datamover to monitor and proactively reroute traffic from all available NIC’s in the Fail Safe Network, therefore increasing storage availability. It has been my experience interacting with customers who are leveraging 10GB on NFS, that they have seen a huge improvement in R/RW to disk and storage, as well as VMotion from datastore to datastore with up to 100% bandwidth and throughput.

Photo Credit: dcJohn

New Feature! EMC Replication Manager 5.3.1: Create SnapView Replicas of RecoverPoint Targets with Linked Jobs (#fromthefield)

By | Backup, Replication | No Comments

I recently ran across a new feature in EMC Replication Manager 5.3.1. Users now have the ability to create SnapView replicas of RecoverPoint CDP, CRR or CLR target volumes. This improves recoverability and resiliency significantly by reducing the time the target volume has to be in physical access mode for backups or remote side testing activities.

The impact of mounting a RecoverPoint replica has always been a challenge. You have these nice consistent copies of your data in a remote location, so wouldn’t it make sense to mount them to a backup server and back them up? Additionally, wouldn’t it be convenient to test that application patch prior to going live?

The downside regarding these functions is that within logged access or physical access mode, users have a limited amount of journal space for incoming replication, and by default an even smaller amount for target side writes. If either of these fill up, you lose journaling and have to re-sync, and/or lose access to the target side volume.

Previous to RM 5.3.1, the only option was to halt the replication, roll to physical access and perform your activities. This left users without replication for the duration of the testing/backup. If you were to take a snapshot or mirror using SnapView, you could resume the replication much sooner, but now we have all kinds of moving pieces to keep track of. That kind of complexity is sure to breed mistakes…

RM 5.3.1 addresses this issue head on with linked jobs.

Under the covers, all of the previously described complexities are still going on. It is orchestrated, though, so that nothing gets fat-fingered and the application’s consistent bookmark gets mounted, rolled to physical access, SnapView creates its replica, and that replica is mounted to a mount host.

There are two ways to configure SnapView replicas of RecoverPoint targets: two-phase jobs and copy jobs. With a two phase-job, RecoverPoint CDP or CRR is selected as the source and the SnapView snap or clone is selected as the replication technology. This method leverages the link job mechanism. The copy job references an existing RecoverPoint CDP or CRR job as the source of the SnapView replica. I realize this seems redundant, but there are many situations where you may want to separate jobs that are called by separate schedules.

If you want more details on any of these technologies or have any comments, please drop me a line below.

Photo Credit: Jeff Howard

#DisasterRecovery: Avoidable Business Cost, or Social Responsibility?

By | Backup, Replication, Uncategorized | No Comments

The idea of corporate responsibility to bettering society continues to pick up steam. We hear about how businesses should proactively address their manufacturing and business practices to encourage ethical treatment of employees, we hear about ‘green’ companies who only use suppliers who use resource-sustainable manufacturing processes, etc. How many of you think about disaster recovery as a social responsibility? I maintain it may be one of the most important responsibilities organizations have.

Read More

VMware Virtual Machine Backup with Avamar Deduplication Solution (Installation How-to #Fromthefield)

By | Avamar, Backup, Deduplication, Replication, VMware | No Comments

Initially, our customer had a huge challenge backing up his virtual environment. He had to treat his VMs as physical machines. Due to the long backup window of “backup to tape,” the customer was only able to a get a full backup on the weekend. As we all know, backup to tape is the cheapest; but with VMware establishing itself as an industry standard, tape can no longer be our primary backup target.

At first, he attempted to backup to a SAN partition mounted to a VM—although this improved his backup window, the customer found a new challenge of moving the data from disk to tape. Unfortunately, moving the data off the SAN partition to tape was still taking too long and he didn’t have enough space to accommodate a second day’s backups. Given that, he opened up to the suggestion of moving toward backup to disk as part of an Avamar deduplication solution.

The installation was very straight forward. Once I initialized the nodes, we started by deploying the Avamar VM image client. The proxy is a self-deploying VM; once we load the OVA, we simply go through this 4 step process.

  1. Networking (IP, DNS, Hostname)
  2. Time zones
  3. Proxy type Windows or Linux (what type of VM you will be backing up)
  4. Register client to main Avamar node

Take note, this is very important: if you will not be installing a genuine certificate for your virtual center, you must modify the mcserver.xml file as follows.

[framed_box bgColor=”#7b0000″ textColor=”#fefefe” rounded=”true”]You have log in as admin

type ssh-agent bash

ssh-add ~admin/.ssh/admin_key

dpnctl stop mcs

<text editor> /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

Edit this line and set to true as shown in example

<entry key=”ignore_vc_cert” value=”true” />

Once you’ve modified the file type dpnctl, start mcs[/framed_box]

This will allow you to add the virtual center server to Avamar and import clients.  Trust me, if you don’t complete the above importing, the virtual center will fail.

Once we’ve imported the clients and enabled change block tracking, Avamar has a default policy group that includes all of your VM clients. Simply add the schedule and retention of your liking, and your VMs are ready for backups. Utilizing the proxy agent, Avamar will back up the entire virtual machine (.vmx, .nvram and .vmdk files).

The benefit of backing up the server as a Virtual Machine is that it will allow you to restore a server seamlessly, without having to load an OS and an application. We were able to seed the VM to Avamar within 12 hours. The next backup ran for about 15 minutes. Avamar was finding between 99.5 and 100% of common data.

Once we converted all the VMs backups to Avamar, the customer was able to perform full backups of over 25 machines daily. And in order to comply with offsite media, we replicated all of the data on Avamar to a secondary Avamar node, which, after a three day seeding window, would take less than 4 hours to replicate the changed blocks over the WAN.

Leveraging Avamar to backup VMware will increase your productivity, RTO and RPO. And guys, it’s just plain simple to use! DOWN WITH TAPES!!!

[Photo credit to SandiaLabs on Flickr]