1. Overview
Virtualization backup technology was first provided and initiated by VMware. With the popularity of virtualization applications in enterprises and various industries, mainstream backup products basically support VMware and Hyper-V. , Citrix and virtualization platforms derived from Xen or KVM.
Virtual machine backup, different from virtual machine snapshot, virtual machine backup is the most important basic measure for virtualization data protection. Many users who are new to virtualization often think of virtual machine snapshots as backups, but this is actually a serious mistake. The reasons are as follows:
1. Snapshots can never be the solution for virtualized local backup.
2. Once you use a snapshot to restore the previous state, you can never return to the current state.
3. Once the virtual machine disk file is damaged, the snapshot will also become invalid.
4. Snapshots can only be restored based on the entire virtual machine image, and cannot be restored at the file level or application granularity.
5. Snapshots can only be used as an auxiliary means to protect the rapid recovery of virtualization.
6. Not all virtual machines can use snapshots, but all virtual machines can use backups.
7. Excessive snapshots extremely affect the performance of the virtual machine, and the virtual machine data may be damaged during the creation or deletion of snapshots.
There are currently two mainstream backup solutions for virtualization platform backup, one is agentless backup (Agentless), and the other is agent backup (Agent) or called Guest OS level backup. This article will summarize the best practices of virtualization backup by analyzing and comparing the advantages and disadvantages of agentless and agent-based backup.
2. Agentless backup analysis
Agentless backup usually means that there is no need to install any backup agent program (or client, probe) on the virtual machine, through ESXI Deploy one or several proxy virtual machines (backup proxy applications) on the host or hypervisor cluster to capture the backup VM.
The advantages of agentless backup are very obvious:
1. Deployment and installation are simple. There is no need to install a backup agent on each virtual machine. It can be fully automatically deployed just by configuring hypervisor integration.
2. Agentless backup makes full use of the dedicated backup interface provided by the virtualization manufacturer. When backing up the virtual machine, it can optimize resource consumption and reduce the load pressure of the virtual machine itself during backup.
3. Using agentless backup products on a specially adapted virtualization platform can realize some backup and recovery functions unique to the virtualization platform. (Such as CBT\RCT block tracking, instant recovery, virtual machine replication, etc.)
4. According to the promotion of virtualization manufacturers, agentless backup and recovery are faster.
5. Agentless backup has more advantages in implementing LAN-FREE or Server-Free backup methods.
As mentioned above, agentless backup is highly recommended by many backup manufacturers, especially virtualization manufacturers. Many users also believe that agentless backup can be better integrated with virtualization platforms.
However, there are many problems with agentless backup in practical applications. In actual operation, the defects of agentless backup are found as follows:
1. Limited by the backup interface provided by the virtualization manufacturer, some Proxy backup products are unable to achieve application awareness, fine-grained data recovery, and RDM (raw disk mapping) virtual machine backup.
2. Agentless backup When backing up a VM, the virtualization platform first captures a snapshot of the VM to be backed up, and then passes the snapshot information to the agentless backup software. It is this VM snapshot that is most likely to cause problems for VMs with high I/O or huge amounts of data (TB-level VMs) and VMs with multiple disk structures. The snapshot time may last for hours or even days. Once the virtual machine disk file becomes abnormal during the snapshot process, it is likely to cause the VM to crash. A similar situation may occur when a snapshot is deleted near the end of the backup. Moreover, snapshots of the virtualization platform itself often cannot be applied silently. Especially for database type VMs, data consistency issues may occur during recovery.
3. In actual scenarios, agentless backup resource consumption is not lower than that of agent-based backup, and in some cases consumes more resources. Agentless virtualization backup requires special attention to CPU resource consumption, because the host CPU is a more limited resource, usually 1 core is shared with 6 or more virtual machines. Upon closer analysis, there are two main reasons for the CPU usage spike during backup. For one, there is a CPU spike when the backup agent has to scan the entire file system for files that are eligible for backup (usually those files that have changed since the last backup). For example, during incremental or differential backup, such directory tree traversal is very time-consuming and requires a lot of CPU resources. Second, the actual transfer of data during the backup process will cause CPU spikes. At present, virtualization manufacturers have successively developed block tracking technologies (such as VMware's CBT, Hyper-V 2016's RCT, etc.) to address the first CPU peak problem. By tracking changes in underlying disk blocks, they no longer traverse and compare directories within the VM. file to optimize resource consumption during incremental\differential backup.
4. In real scenarios, agentless backup is slow. Without slowing down business applications, agentless backup generally limits backup to 2 VMs per host at the same time. Despite the claimed advantages of agentless solutions, they use block tracking technology, which reduces the amount of data transferred. However, the agentless backup method is more of a blind scan, which requires a "pull" method for the backup process, which will reduce the speed of the CPU. Many agentless backup products can adjust the number of concurrent backups of VMs. Generally, the maximum number is about 10-15 concurrent backups (the maximum number limit is also limited by the virtualization platform itself and has nothing to do with the backup software). However, in actual scenarios, it is not recommended to enable maximum concurrency, which will greatly increase the load pressure on the virtualization platform. The most reasonable number of concurrent backups must be determined based on the actual number of virtual machines and platform performance.
5. Agentless backup relies heavily on Tools tools (such as VMware Tools, Hyper-v system integration tools, KVM virt-tools, etc.). When the VM tools cannot run normally or are not updated in time, it will As a result, agentless backup cannot use CBT/RCT block tracking or snapshot exceptions, and VM cannot be quiesced.
6. Agentless backup usually requires that the storage volume where the virtual machine is located has at least 25% remaining space. If the storage space is insufficient, agentless backup snapshots will cause storage volume alarms or virtual machine snapshot failures.
7. If the storage volume where the virtual machine is located is lost or inactive, agentless backup will fail.
3. Proxy backup analysis
Agent refers to a small application installed on the server to perform specific functions. A common example is a client installed by a backup application on a server that backs up the server and provides specific services to the applications running on that server. Since virtualization became popular, proxy backup methods have not been popular among virtualization users. The reasons are as follows:
1. The deployment method is complicated and requires the client agent to be installed in the virtual machines to be backed up. This is a fatal problem for users with a large number of virtual machines.
2. Software compatibility issues. If you install an agent in a VM, you usually need to do an environment check first to rule out incompatibility with the backup software (such as anti-virus, system compatibility, special security applications, etc.) .
3. When the VMs to be backed up are too concentrated on certain hosts in the cluster, the host resource load will increase during concurrent backup and impact the business virtual network.
4. Some backup software does not have the disk block tracking function for physical devices. When there is a proxy backup, file-level backup is used. Incremental/differential backup will increase the load pressure on the VM. Also the backup speed is slow.
5. Maintenance with an agent is more difficult than without an agent. For example, shut down VMs cannot be backed up or individual VMs only open some ports due to security needs, resulting in the agent being unable to connect or transmit data, etc.
Although the proxy backup method has obvious disadvantages in a virtualized environment, it also has many advantages:
1. When backing up a VM, it does not rely on the virtualization platform snapshot and is directly called on the Guest OS system. System snapshots (system vss or LVM snapshots, etc.) have better stability for the backup of VMs with high I/O and large data volumes and multi-disk structure VMs.
2. When backing up VM, it is application-aware and can support fine-grained recovery of Exchange, SQL-server, AD, Oracle, SharePoint, files, etc.
3. For backup software that supports physical device block tracking, agent backup has faster backup and recovery speeds than agentless backup.
4. Proxy backup When backing up a virtual machine with database services, you can configure the database backup script to not only back up the database separately, but also ensure the data consistency of the database.
5. Proxy backup is not limited by the number of concurrent backups on the virtualization platform. As long as the network can bear it, there is no upper limit on the number of concurrent VM backups.
6. It supports a wide range of virtualization platforms. The proxy backup method can support almost all virtualization platforms. If the software authorization allows, it is basically not restricted by the virtualization manufacturer.
4. Practical experience in virtualization backup
Based on some of my implementation experience in projects, the following backup steps can be used for large-scale virtual machine backup (taking VMware virtualization as an example ):
1. Extract all virtual machine information in the current virtualization platform into an EXCEL form, and combine large amounts of data (over TB), multi-disk structures, RDMs, and core database types (high I/O ), VMs whose storage volumes have been deleted (or whose storage volumes are inactive), etc. are all filtered out. Agent backup is installed in such VMs where agentless backup cannot be used.
2. Virtual machines other than the above types can be backed up without an agent.
3. When using agentless backup of virtual machines (especially Windows system virtual machines), be sure to confirm that VMware Tools has been installed correctly and that all VMware Tools system services are running normally. If there is a prompt that VMware Tools has been updated or cannot run, you need to update VMware Tools in time or uninstall and reinstall it.
4. Plan the backup network architecture and check whether the environment requirements meet the configuration requirements such as LAN-BASE\LAN-FREE\SERVER-FREE.
1) In the traditional LAN-BASE architecture, the agentless virtualization backup network must at least meet the Gigabit network standard (10 Gigabit network is recommended). Best practices suggest that at least one physical network port should be left on each ESXI host, and the physical network port should be allocated to a backup dedicated virtual network. The backup data should be transmitted through the backup transmission network through the dedicated network port on each ESXI host to communicate with the business. Network isolation prevents the impact of large data transmission on the business network during backup. For backup storage servers, you can consider using multiple network card bindings. At the same time, if the switch supports it, you can use multi-link aggregation on the switch port connected to the backup storage server to increase the bandwidth of the backup storage server. If the best practice requirements cannot be met, it is recommended that the backup data flow away from the non-core business network segment with lower load pressure in the virtual network.
2) In the LAN-FREE architecture, special attention should be paid to pre-implementation environmental inspection, mainly checking the VMFS volume structure and storage status, multi-path mapping, storage LUN structure, etc. If it is found that there is a combined volume (a VMFS volume composed of multiple storage lun) in the virtualized storage, VMware itself does not support LAN-FREE backup for this kind of volume, and can only use LAN-BASE method. In addition, the backup of LAN-FREE architecture involves production storage mapping, and the implementation has certain risks. If it is not operated properly, the consequences will be serious.
3) Server-Free architecture generally requires storage devices and backup software to be compatible with each other. Different backup products support different storage devices, so this method is rarely used in actual projects.
5. A separate backup storage server or backup storage device must be prepared for virtual machine backup, and precious production storage space must not be occupied. At the same time, based on security considerations, if the backup data is placed on the same storage as the production data, once the storage fails, there will be no backup data for recovery. Backup data should be stored separately from production data.
6. Backup time window planning. Any backup product will have varying degrees of business impact on front-end applications during backup. Therefore, when implementing a backup project, you must reserve a good backup time window. The backup time window is generally reserved for periods when there is less business. The backup time can be roughly calculated based on the overall size of the backup data and the transmission rate. Since the virtualization platform has a large number of virtual machines, it is recommended to divide it into virtual machine groups according to different business types and reserve different backup windows for the virtual machine groups.
7. The virtual machine backup cycle directly affects the time point at which data can be recovered. Therefore, different backup cycles must be formulated according to RPO/RTO requirements for virtual machine groups of different businesses.
#8. Whether to use deduplication. Whether to use deduplication should be decided based on the amount of virtualized storage data, the space required for backup storage, and the backup time window. If there are many virtual machines to be backed up, the amount of data is large, the storage space required for backup is insufficient, and the backup window is short, data deduplication is the best solution. However, deduplication has certain requirements on the hardware performance of the backup storage server. Therefore, it is recommended to refer to the backup product manufacturer's requirements to configure the deduplication server. In addition, deduplication has certain risks. Once the deduplication database is damaged, all backups will be irrecoverable. It is recommended that for backup data with deduplication turned on, there must be a second copy to try to meet the 3-2-1 principle of backup. Finally, each backup manufacturer has its best practices for data deduplication, but the basic idea is the same. Generally, first back up a few typical virtual machines in the virtualization platform, and then perform batch backups to achieve the best deduplication effect.
9. There is a limit on the concurrency of agentless backup virtual machines. It is generally recommended that a backup plan follow VMware’s default of concurrent backup of two virtual machines. The number of concurrency can be adjusted based on comprehensive consideration of virtualization platform performance and network bandwidth usage. However, it is recommended not to adjust the number of concurrencies too much or enable the maximum concurrency. Otherwise, the virtualization platform will be under great pressure, communication problems may occur, virtual machine business accidents may occur, and backups may fail.
10. Develop a backup plan based on the business, and ensure that there is a certain time interval between backup plans. Avoid starting backup of a large number of virtual machines at the same time, causing large fluctuations in network and CPU load.
11. Determine the backup retention period based on different business types. It is recommended that backups of time-sensitive services be retained for 1-2 weeks. It is recommended that the retention period of virtual machines that need to be archived be set to more than 3 months. The retention period is closely related to backup storage usage, so the data retention time of different virtual machine groups must be carefully planned.
12. If you use a Windows VM with agent backup, for the convenience of deployment, you can use remote push to install the backup agent. If push conditions are not met, local installation is used. Before pushing or installing the agent locally, you must pay attention to checking the installation environment, which can be checked one by one from aspects such as patches, compatibility, network, and configuration.
13. After the virtualization backup plan is implemented, it is necessary to closely observe the daily backup situation and business impact for 1-2 weeks. If any abnormal backup or impact on normal business is found, the backup strategy should be adjusted in a timely manner, and the backup plan should be continuously optimized until the backup is completed. Stablize.
5. Summary
The virtualization backup project seems simple, but it requires considering the backup plan from the number of virtual machines, storage architecture, network architecture, backup planning cycle, etc., determining the implementation process based on the actual situation of the virtualization platform, and continuously optimizing the backup strategy .