Avamar, Cloud Infrastructure, eSata card, Segate GoFlex, Iomega, NAS, Ethernet, Xcopy, Windows, Linux box, grid, Jon Austin,

Speed Up First-time Avamar Backup by “Seeding” the Server

By | Avamar, Backup, How To | No Comments

avamar seeding blog header 1200Avamar is a great tool to backup remote office file servers to a private or hybrid cloud infrastructure. However, performing an initial backup can be a challenge if the server is more than a few GB and the connection to the remote office is less than 100Mb.

In this scenario, the recommended process is to “seed” the Avamar server with the data from the remote server. There are a number of devices that can be used to accomplish this: USB hard drives are the most often used; however, they can be painfully slow, as most modern servers only have USB 2.0 ports that can only transfer around 60MB/sec and are limited to 3-4TB in size. In order to copy 3TB to a USB 2.0 drive, it will typically take 12-16 hours. Not unbearable, but quite a while.

Another option would be to install a USB 3.0 adapter card or eSata card—but that requires shutting down the server and installing drivers, etc. An alternative that I have had a good deal of success with is using a portable NAS device like the Seagate GoFlex drives or, for larger systems, the Iomega/LenovoEMC StorCenter px4-300d. The px4-300d has an added feature that I will touch on later. These NAS devices leverage Gigabit Ethernet and can roughly double the transfer rate of USB 2.0.

portable nas storage devices

Moving the data to these “seeding” targets can be as simple as drag-and-drop or using a command line utility like Xcopy from Windows or Rsync from a Linux box once you plug in the USB device or mount a share from the NAS drives. When the data copy is complete, eject the USB drive or unmount the share, power down the unit, package the drive for shipping and send it back to the site where the Avamar grid lives.

At the target site attach the portable storage device to a client locally and configure a one-time backup of this data. With the Iomega device, it includes a pre-installed Avamar client that can be activated to the grid and backed up without having to go through an intermediary server.

Once you get this copy of the backup into the Avamar grid, activate and start the backup of the remote client. The client will hash it’s local data and compare to what is on the gridfinding that the bulk of the unique data is already populated – reducing the amount of data required to transfer to the data that has changed or added since the “seed”.

Photo credit: Macomb Paynes via Flickr

deduplication, data domain, online storage, back up online, data, data storage

Avamar And Data Domain: The Two Best Deduplication Software Appliances

By | Avamar, Deduplication, Networking, Storage | No Comments

For years backup-to-disk technologies have evolved toward the ingestion of large sums of data very quickly, especially when compared to the newest tape options. This evolution has made backup applications influence disk targets for equally fast restores, even down to the file level.

Essentially what this means is that, today clients can integrate disk-based back-up solutions to fulfill the following conditions:
[framed_box bgColor=”#F0F0F0″ textColor=”undefined” rounded=”true”] – Mitigate risk of traditional tape failures

– Reduce the amount of time it takes to perform large back-up jobs

– Reduce the amount of capacity required at the back-up target by nearly 10-20X more than tape

– Reduce the amount of data traversing the network during a back-up job (Avamar or similar “source-based” technologies)

– Lower the total cost of ownership in comparison to tape

– Enable clients to automate the “off-site” requirement for tape by the way of replicating one disk system to the next over long distances

– Lower the RTO and RPO for clients based on custom policies available

Data Domain deduplication methods are useless without backup software in place. By leveraging Data Domains OST functionality (DDBoost), we can now combine Data Domain’s deep compression ability with the superior archiving abilities of Avamar.

Through Source-Based Deduplication, Avamar’s host side enables environments with lower bandwidth and longer backup windows to push the backup process much faster. Also, after completing the initial backup, this strategy results in less data on disks, which is good for everyone.

deduplication, data domain, online storage, back up online, data, data storage

Where Data Domain shines the most is in its ability to compress the then deduplicated data 10X more than Avamar. This integration allows Avamar to cut weekend, month-end and year-end backups to the Data Domain, allowing for much longer retention. This feature expands Avamar’s reach into extended retention cycles to disk, which is one of the faster restore methods.

Data Domain’s “target-based” deduplication technology means the backup/deduplication process happens at the actual DD Appliance. Data Domain is the actual target, as it is here that the deduplication takes place.

All data has to go over the network to the target when leveraging Data Domain. If there is a need to backup 10TBs then 10TBs need to traverse the network to the DD Appliance. When leveraging Avamar, I may only need to send 2TBs over the network, given the fact that data has been deduped prior to pushing to the target.

Taking Data Domain even further, Avamar can replicate backups to another Data Domain off site.

Allowing Avamar to control the replication enables it to keep the catalogues and track the location of the backup. This ability gives the end user ease of management when a request is made to restore. The prerequisites for DDBoost are both the license enabler for DDBoost and the Replicator on Data Domain. Overall this integration of the two “Best Deduplication Appliances” allows the end user a much wider spectrum of performance, use and compliance.

For a deeper dive into deduplication strategies, read the article from IDS CTO Justin Mescher about Data Domain vs EMC Avamar: which deduplication technology is better.

EMC Avamar Virtual Image Backup: File Level Restore Saves Customer from “Blue Screen” Death

By | Avamar, Backup, Disaster Recovery, EMC | No Comments

Recently, a customer of ours had a mission critical Virtual Machine “blue screen.” Yikes! Good news was their environment was leveraging Avamar Virtual Image backups. Bad news was the VM was in an unstable state for a while, and every time the VM was restored it continued to “blue screen.” Therefore, the OS was corrupted—one of the many joys of IT life!

To lose my title of Debbie Downer, let me explain that their environment also was leveraging “FLR” Avamar Virtual File Level Restore. I must say in my experience restoring applications, the data is priority one.

This picture couldn’t have been more beautiful: they had a win 2k8 template with SQL loaded, and they simply extracted the database files from the VMDK backup using FLR and restored them to the new VM, with the data intact and up to date.  Take that tape! Never had to request or load tapes to restore anything 5 years later!

If you are not familiar with EMC Avamar FLR, basically it is the ability to extract single objects out of the virtual image backups. This is done with a proxy agent which exists within your virtual environment that will mount your backups and extract any data that exist within the VM. That means one single backup to your VM and the ability to restore anything within the VMDK without having to load a new VM.

This feature can be used in many ways: one being the dramatic example I just gave, another being the ability to use the data files for testing in other VMs. Although this is just a single feature example of the many abilities of Avamar, its usage will greatly reduce your RPO and RTO.

In my experience, leveraging Avamar and virtual file restore will improve your virtual restoring procedures and bring the peace of mind that your data is within arms reach anytime of the day. As I continue to post about Avamar features and capabilities from the field, I’ve developed this as my slogan for the series: Keep your backups simple … and down with tape!

Photo Credit: altemark

Tape Sucks: Avamar 6.0 Version #tapesucksmoveon

By | Avamar, Backup, Data Loss Prevention, Deduplication, EMC, Replication, Storage, VMware | No Comments

Since its release last week, there has been a lot of buzz around the Avamar 6.0. I am going to take the liberty of reading between the lines and exploring some of the good (but gory) details.

The biggest news in this release is the DDBoost/Data Domain integration into the new Avamar Client binaries. This allows an Avamar client to send a backup dataset stream to a DataDomain system as opposed to an Avamar node or grid. Datasets that are not “dedupe friendly”(too large for Avamar to handle, or have very high change rates) are typically retained for shorter periods of time. These can be targeted to a DD array, but still managed through the same policies, backup and recovery interface.

Client types supported pertaining to this release are limited to Exchange VSS, SQL, SharePoint, Oracle and VMWare Image backups. Replication of data is Avamar to Avamar and DataDomain to DataDomain: there isn’t any mixing or cross-replication. Avamar coordinates the replication and replicates the meta-data so that it is manageable and recoverable from either side. From a licensing perspective, Avamar requires a capacity license for the Data Domain system at a significantly reduced cost per TB. DDBoost and replication licenses are also required on the Data Domain.

There is a major shift in hardware for Avamar 6.0: 

  1. The Gen4 Hardware platform was introduced with a significant increase in storage capacity.
  2. The largest nodes now support 7.8TB per node – enabling grids of up to 124TB.
  3. The new high capacity nodes are based off of the Dell R510 hardware with 12 2TB SATA drives.
  4. To speed up indexing the new 7.8TB nodes also leverage an SSD drive for the hash tables.
  5. There are also 1.3TB, 2.6TB and 3.9TB Gen4 Nodes based off of the Dell R710 hardware.
  6. All nodes use RAID1 pairs and it seems the performance hit going to RAID5 on the 3.3TB Gen3 nodes was too high.
  7. All Gen4 nodes now run SLES (SUSE Linux) for improved security.

There were several enhancements made for grid environments. Multi-node systems now leverage the ADS switches exclusively for a separate internal network that allows the grid nodes to communicate in the event of front-end network issues. There are both HA and Non-HA front end network configurations, depending on availability requirements. In terms of grid support, it appears that the non-RAIN 1X2 is no longer a supported configuration with Gen4 nodes. Also, spare nodes are now optional for Gen4 grids if you have Premium Support.

Avamar 6.0 is supported on Gen3 hardware, so existing customers can upgrade from 4.x and 5.x versions. Gen3 hardware will also remain available for upgrades to existing grids as the mixing of Gen3 and Gen4 systems in a grid is not supported. Gen3 systems will continue to run on Red Hat (RHEL 4).

Avamar 5.x introduced VStorageAPI integration for VMWare ESX 4.0 and later versions. This functionality provides changed block tracking for backup operations, but not for restores. Avamar 6.0 now provides for in-place “Rollback” restores leveraging this same technology. This reduces restore times dramatically by only restoring the blocks that changed back into an existing vm. The other key VMWare feature introduced in version 6.0 is Proxy server pooling – previously, a proxy was assigned to a datastore, but now proxy servers can be pooled for load balancing in large environments.

There were several additional client enhancements on the Microsoft front including Granular Level Recovery (GLR) support and multistreaming (1 to 6 concurrent streams) for Exchange and Sharepoint clients.

All in all, the Avamar 6.0 release provides several key new features and scales significantly further than previous versions. With the addition of Data Domain as a target, tape-less backup is quickly approaching reality.

Photo Credit: altemark

VMware Virtual Machine Backup with Avamar Deduplication Solution (Installation How-to #Fromthefield)

By | Avamar, Backup, Deduplication, Replication, VMware | No Comments

Initially, our customer had a huge challenge backing up his virtual environment. He had to treat his VMs as physical machines. Due to the long backup window of “backup to tape,” the customer was only able to a get a full backup on the weekend. As we all know, backup to tape is the cheapest; but with VMware establishing itself as an industry standard, tape can no longer be our primary backup target.

At first, he attempted to backup to a SAN partition mounted to a VM—although this improved his backup window, the customer found a new challenge of moving the data from disk to tape. Unfortunately, moving the data off the SAN partition to tape was still taking too long and he didn’t have enough space to accommodate a second day’s backups. Given that, he opened up to the suggestion of moving toward backup to disk as part of an Avamar deduplication solution.

The installation was very straight forward. Once I initialized the nodes, we started by deploying the Avamar VM image client. The proxy is a self-deploying VM; once we load the OVA, we simply go through this 4 step process.

  1. Networking (IP, DNS, Hostname)
  2. Time zones
  3. Proxy type Windows or Linux (what type of VM you will be backing up)
  4. Register client to main Avamar node

Take note, this is very important: if you will not be installing a genuine certificate for your virtual center, you must modify the mcserver.xml file as follows.

[framed_box bgColor=”#7b0000″ textColor=”#fefefe” rounded=”true”]You have log in as admin

type ssh-agent bash

ssh-add ~admin/.ssh/admin_key

dpnctl stop mcs

<text editor> /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

Edit this line and set to true as shown in example

<entry key=”ignore_vc_cert” value=”true” />

Once you’ve modified the file type dpnctl, start mcs[/framed_box]

This will allow you to add the virtual center server to Avamar and import clients.  Trust me, if you don’t complete the above importing, the virtual center will fail.

Once we’ve imported the clients and enabled change block tracking, Avamar has a default policy group that includes all of your VM clients. Simply add the schedule and retention of your liking, and your VMs are ready for backups. Utilizing the proxy agent, Avamar will back up the entire virtual machine (.vmx, .nvram and .vmdk files).

The benefit of backing up the server as a Virtual Machine is that it will allow you to restore a server seamlessly, without having to load an OS and an application. We were able to seed the VM to Avamar within 12 hours. The next backup ran for about 15 minutes. Avamar was finding between 99.5 and 100% of common data.

Once we converted all the VMs backups to Avamar, the customer was able to perform full backups of over 25 machines daily. And in order to comply with offsite media, we replicated all of the data on Avamar to a secondary Avamar node, which, after a three day seeding window, would take less than 4 hours to replicate the changed blocks over the WAN.

Leveraging Avamar to backup VMware will increase your productivity, RTO and RPO. And guys, it’s just plain simple to use! DOWN WITH TAPES!!!

[Photo credit to SandiaLabs on Flickr]

When Snapshots with Replication Aren’t Enough: Benefits of a Purpose-built Backup Solution

By | Avamar, Backup, EMC, Replication, Storage | No Comments

When we work with customers on a storage solution, we often end up talking about backup as well. Why is that? For starters, it’s important to remember that every GB of storage you add is another GB that will have to be backed up by your backup infrastructure.

When I worked at a large pharmaceutical company, this was a problem we often encountered when internal business units would purchase more storage. Six months later as they consumed the new storage, we’d have a disaster on our hands with not enough tapes or tape drives to get backups completed. Of course, you can ask the business for more money to buy additional backup infrastructure, but that is a difficult thing to do after the original purchase. One question I often hear from customers who are looking at a multiple vendors as part of a storage/backup project is, “Why not just use snapshots with my storage array so that I don’t need a backup solution anymore?”

There are different ways to achieve “backup.” Some customers choose to use a true purpose-built backup solution, and some have chosen to eliminate backup by using snapshots with replication. Regardless of which way you decide is the correct fit for your environment, IDS has a solution that can meet your requirements. Our customers, and the majority of customers nationwide, choose to implement a true purpose-built backup solution as a means to achieve data protection. The entire backup deduplication market (not just EMC products) is experiencing high double-digit growth rate every year, or even triple-digit growth in some cases. This validates that purpose-built backup solutions, leveraging de-duplication technology, are very relevant in the marketplace today. Avamar is one of the most popular backup solutions that is in use by both large enterprises and small-medium size customers. VMware actually uses Avamar to backup its own internal VMware infrastructure.

That being said, eliminating backups by using snaps with replication will “work” as a way to achieve a backup of your data. You will typically hear about this method of backup from vendors that only offer storage products rather than storage and backup solutions. However, there are caveats to using this method to eliminate backup, regardless of vendor. Like anything, there are pros and cons of different data protection methodologies. Most commonly, customers wanting to do snaps with replication as a form of backup do so because they see the potential for cost savings. Adding some incremental enhancements to a storage investment that allow it to be used as a backup solution can appear to be more cost-effective than purchasing two separate solutions for storage and backup. However, here are five reasons why we find the majority of customers still choosing to implement backup with a purpose-built solution:

1) At least 85 percent of all restores are typical day-to-day granular file restores (compared to complete server or site recovery). With a LUN snapshot, a file restore on a server becomes more complex. For a simple file restore to an app server, you must take the snap and mount it up to a server, then drill-down into the snap to recover the file and copy it back to the original location. Then, the LUN must be unmounted. While EMC and and some other vendors have tools to help automate this, it’s still not a trivial task. Snapshots are organized by snapshot date—not by files or directories in a server. If a user asks for a file to be restored, and they don’ t know what server it lived on, how are you going to find the right snapshot? There is no equivalent of a backup catalog. Compare this to opening the Avamar console, pick the server, pick the date to restore to, and select the file name (or search for a file name). The file will then be restored back to the original server. This is a simpler task, plus it is very similar to the way that backup administrators do their job today, which means less re-training for staff.

2) The “snaps with replication” methodology only covers servers that are on the SAN. Therefore, a customer may end up having to use different methods of backup for servers on the SAN vs. servers that continue to use internal storage. This makes administration more complex for a backup administrator.

3) There is a greater risk of data loss from not having a secondary copy of the data stored on a different medium. While most customers will never see this, there is always the potential for a file system or LUN to get corrupted in some way, shape, or form. If that happens, and the corruption gets replicated to the secondary side, all snapshots become worthless because snapshots contain pointers that are dependent upon the primary copy of the data. At that point, there will be data loss. A secondary copy of your data stored on a different medium, whether it be classic tape or a next-gen backup solution like Avamar will not be affected by this. Avamar and DataDomain also do data integrity checks, which does not happen on any SAN/NAS or tape product. On a SAN/NAS or tape solution, you may not realize you have corruption until you try to restore and encounter a bad block.

4) Humans are prone to make mistakes and there is a greater risk of data loss from human error using the “snaps with replication” methodology. If there is an accidental misconfiguration of LUN replication, it is not as apparent to administrators as a failed backup job, therefore the problem may not be detected until it is too late and there is data loss. Another example might involve a newly created LUN that does not get added to a replication policy, therefore tens of VM’s could be affected because bits and pieces of the VM’s happen to reside on that particular LUN, which is part of a VMFS datastore. While human mishaps can occur in the traditional backup world, they generally would only affect one server at a time rather than multiple servers.

5) You can achieve greater retention with a true backup solution than you can with keeping snapshots. While you may only need to keep backups for 45 days today, if those requirements increase to one year or more in the future, the snapshot methodology becomes less attractive. The longer you keep snapshots, the more space they require, and the more overhead is required to keep track of all the delta changes in the snapshot. It is not best practice to keep snapshots for longer-term data protection (several months to 1yr+) regardless of vendor. This is best suited for a purpose-built backup solution.

Data Domain vs. EMC Avamar: Which deduplication technology is better?

By | Avamar, Deduplication, EMC | No Comments

Now that EMC owns both Data Domain and Avamar, I am constantly being asked which technology is better. Before the Data Domain acquisition, it was tough to get a straight answer because the two deduplication giants were constantly slugging it out and slandering each other to try and find an edge and gain more market share. With the two technologies now living under the same umbrella, sometimes it is hard to tell where one technology ends and the other begins.

Read More

EMC Avamar Replication Guide: Sizing Your Bandwidth Properly for Offsite Backup

By | Avamar, Backup, EMC, Replication | No Comments

One of the most powerful features of Avamar software is its ability to replicate backup sets to another Avamar grid, providing offsite backups without the hassle of cutting and handling tapes. Avamar’s deduplication functionality not only reduces the amount of storage required but also the bandwidth required to replicate what is functionally the equivalent of a full backup. While the reduction in bandwidth is dramatic, the difference bytes still have to be sent across a WAN link. Sizing that bandwidth properly is critical to the effective operation of the Avamar System.

Read More

EMC Avamar Review from the Field: Backup Made Better with Source-Based Deduplication

By | Avamar, Backup, EMC | No Comments

Companies and organizations are often faced with challenges regarding their backup windows, making them concerned about their overall reliability. EMC Avamar is a powerful backup and recovery system that can provide your organization with the reliability of daily full backups in a fraction of the time required by a traditional tape and disk based backup solution.

Read More

Deduplication Wars: EMC Avamar vs CommVault Simpana

By | Avamar, Deduplication, EMC | No Comments

OK, so you have decided that deduplication is the best thing ever and a must have for your backup needs. The next big question on the horizon is what KIND of deduplication is right for you. Two of the big hitters in the market today are EMC’s Avamar and CommVault’s Simpana products. Both products seem to be doing very well in the wild and both approach deduplication in completely different manners.

In the case of Avamar, the product is deduplicating at the client using variable block deduplication. Once the scan is complete on the client server and the deduplication hash is created the client actually checks back with the Avamar Data Store appliance farm to see which blocks the farm has not seen and then only the truly unique blocks across the environment (not just that server) are sent over the wire. This results is extremely high levels of deduplication AND remarkably fast backups since very little data is normally left to send after deduplication and comparison to the rest of the environment. The data is stored on the EMC Data Store appliance which presents a pretty simple GUI for the recovery. The only major chink in the armor of Avamar is that it does not have the ability to natively create tapes for those data sets you may want to retain longer than you have space to keep on the appliance farm.

CommVault came at the deduplication process a completely different way and leveraged their existing tape archive construct to create a form of fixed block deduplication. In this case the clients do the same thing they always did: run their scans, package the data, and then shoot it out. Once the data gets to the Media Agent the deduplication occurs and the data is spit out onto any disk target supported by CommVault. Since the deduplication is fixed block, the deduplication ratios are not as good as with variable block, aka Avamar, but certainly much better than typical compression. Since the deduplication occurs on the Media Agent, there is no savings in backup window time. The good news is that this is CommVault and cutting tapes is it’s forte and completely automated with the ability to have different retentions on each type of media to fit all your compliance desires within a single tool. Also, since the format of the media archive did not change, restores are just as fast with deduplicated data as they were with plain Jane backup to disk, which is huge if you have a lot of data to restore. Avamar can be sluggish in terms of restore on the smaller deployments but still certainly functional for your every day restore needs.

On the grand scheme, Avamar is the holy grail of backup speed since it only every sends fractions of incremental data over the wire to the target which reduces not only backup times but the impact of backup on both the hosts and the network. I also give CommVault a major tip of the hat in how they leveraged their existing technology and morphed it into a deduplication technology that brings huge benefits to their current customer base while staying on the commodity hardware bandwagon.

Obviously there are many more features to both products worth investigating and comparing but now you know how the two differ technically in terms of the deduplication angle.

Photo Credit: JTony via Flickr