VMware vSphere Deployment and Management

This chapter is from the book

VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment

Learn More Buy

This chapter is from the book

This chapter is from the book 

VMware vSphere and Virtual Infrastructure Security: Securing the Virtual Environment

Learn More Buy

Security Issues during Management and Deployment

A set of common problems that occur when working with VMs could be considered security issues as well. These range from simple mistakes made by administrators to serious issues of data crossing security zones. This section covers a few of the more prevalent issues.

VIC Plug-ins

Several Virtual Infrastructure Client plug-ins are available. These third-party tools allow access to otherwise inaccessible data. Some plug-ins use the authentication contained within the VIC after you log in and start your session, whereas others access other authentication methods. Either way, the plug-ins in use should be used with extreme care because there is no inherent protection from a plug-in within the VIC other than the capability to disable a plug-in, if it provides you the option. Not all plug-ins provide this capability. After you log in, it has the access to the virtual environment that you do. One of the plug-ins available logs all actions within the VIC. If this was done behind the scenes and sent to a hacker, the hacker would now have enough information to possibly pivot an attack into your virtual environment.

Table 6.1 provides a partial list of the currently known plug-ins, what they do, and the possible risks.

Table 6.1. Partial List of Available Plug-ins

Plug-ins are not currently digitally signed, and although some of the plug-ins are extremely useful, they should be used with caution. If possible, be sure you retrieve them from a trusted source, or if they are open source, review the code and compile them yourself to ensure that they are not going to do anything that would be considered dangerous.

VMs on the Wrong Network

It is extremely easy to place a virtual machine or appliance on the wrong virtual network using the VIC connected to either VC or the host. This could cause quite a problem if the VM, for example, was from a hostile environment such as a DMZ. There are different levels of hostile environments. From the virtualization host, all VMs are considered to be hostile, but within the virtual networks, other networks can be seen as hostile. If the VM ended up on a production network from a DMZ network, the VM could have been set up to detect this possibility and pivot an automatic attack into the production network.

These attacks could happen even if the VM had an IP outside the range of the virtual network on which it now resides. If the VM has been compromised, it is quite possible that the VM now has the proper routes to allow traffic through to the VM or has a valid IP. At the very least, it could use MAC-based attacks. At the very least, this could create inadvertent data commingling.

If the VM ends up within a portgroup where promiscuous mode ethernet adapters are allowed, the VM could be used to sniff traffic across a virtual switch. Much of this depends on whether the VM was compromised in some way. As shown in Chapter 2, it is quite possible that the VM has been compromised and could be a ticking time bomb waiting for such an event.

The only solution at this point is to maintain multiple virtual infrastructure clusters, each for its own security zone. In other words, do not place VMs within the DMZ on the same hosts with VMs on other networks. Sometimes this is impractical from a cost perspective, but it is the best solution for preventing the possibility of a VM being placed on the wrong network. The second-best solution is to maintain diligent auditing that will tell you if this possibility has occurred.

VMs or Networks Created Without Authorization

As we discussed earlier in the chapter, it is possible that the person who approves the creation of VMs could also be an administrator of the virtual environment. Any administrator is allowed to create VMs without going through the VM creation approval process. This often leads to the concept of VM sprawl, where VMs exist on the system that are unknown to other administrators or the managers who oversee the virtual environment. It is also possible for administrators to create virtual networks without going through an approval process. These unknown machines and networks could be the source of severe issues further down the line.

In the 2008 Verizon Data Breach Report,² there was a category for unknown unknowns, which comprised unknown machines and networks in use. The fact that they existed was an issue because they were created outside the approval process and therefore may not have the latest set of patches, updates, monitoring, and auditing performed, which implies that they could be the source of security incidents.

The solution is to have a review process in place that you must go through to create a virtual machine, appliance, or network within your virtual environment. This process could be as formal as necessary. Tools exist to help solve this problem (VMware LifeCycle Manager) but a process is still required.

VMs on the Wrong Storage

Another issue that happens is placing VMs on the wrong datastore. Datastores should mimic your security zones at the very least. If you follow the rules of at least one cluster per security zone, you will want to have datastores specific to each cluster. Just because a datastore has space on which to place the VM does not imply it should go there.

This can create three issues. The first is performance. Balancing loads across datastores is very important, and the inadvertent loading of a disk intensive VM onto a datastore that is well balanced for performance across all VMs could cause performance issues, and may cause denial of service as storage becomes overloaded. The second issue is that now there will be commingling of security zone data on a single datastore. In general, this is not a problem from a VM perspective, but a backup tool that has rights to see only one set of VMs will inadvertently be able to see backup VMs that the software or human is not authorized to access. This could lead to information leakage. The last item is that you could end up with a VM on the wrong storage network, or you now have a cluster than handles more than one security zone, which could lead to problems discussed previously. At the very least, we now have data commingling, and if the storage network is ever breached, you now have the possibility of losing more than you expected.

The solution is to audit your systems on a regular basis to ensure that VMs and virtual appliances are not placed on the wrong datastore.

VMs Assigned to Improper Resource Pools

Another issue that happens is placing VMs within the wrong resource pool. This can happen currently when the wrong options are chosen when a VMotion is performed. This could lead to a VM using more resources than planned. If the VM is one that would normally use quite a lot of resources, it could also lead to a denial of service for the resource in question or for the entire virtual environment, such as when CPU and memory get overloaded within a cluster. VMware DRS would kick in. This would force VMs to move about the environment, which could increase contention on disk and the network outside of the expected boundaries.

Normally this would be considered a performance issue, but the culprit is a VM that is outside the constraint of its supposedly assigned resource pool. Unfortunately, the only solution to this problem is to be watchful and audit the placement of each VM within the resource pools.

Premature Propagation of VMs from Quality Assurance to Production

With the advent of VMware Stage Manager it is possible to prematurely propagate a VM from one stage to another, whether that is from development to quality assurance or from quality assurance to production. This premature propagation could bypass a security step because the last phase could be to inspect the virtual machine or appliance for any security defects, install the latest virus signatures, and other security related configuration steps.

If this inspection does not take place, there may be security requirements not being met. In addition, a VM in QA could be reading different data than in production, and if there is a premature propagation the data sources could be incorrect. This could lead to further down time to fix the problem and perhaps a lengthy restoration process to recover the previous iteration of the virtual machine or appliance.

The only solution is to have a very good change control process that has written documentation to follow when VMs are propagated.

Physical to Virtual (P2V) Crossing Security Zones

In some cases you need to virtualize systems that are in one security zone—for example, a DMZ—but to do so you need to access the virtualization administrative security zone. In other words, you need to copy data from the DMZ through the VMware ESX service console or VMware ESXi management appliance. When this happens you are crossing security zones, which you should not do because it allows the hostile environment of the DMZ to directly access your virtualization host's management appliances, which is to be avoided at all costs. This sticky process, however, has a very easy solution. Break the P2V into multiple stages using an intermediary system. Here is how the process works.

1. Create a virtual network for the security zone from which you want to virtualize, in our example, the DMZ. This virtual network should be connected to the physical network in the same security zone.
2. Create a pseudo security zone virtual network that mimics the real virtual network but is fully firewalled off, except to allow the necessary services to boot the VM: Perhaps DNS, DHCP, and the like.
3. Work with the security team to get a computer blessed to work within the source security zone—in this case, the DMZ.
4. Ensure this computer has enough attached removable storage to contain the entire source physical machine's disks.
5. Before attaching the removable storage to the computer, perform a complete disk wipe of the storage device and reformat with the appropriate file system.
6. Run a virus scan on the computer blessed by the security team, and then fix any issue or reinstall as necessary.
7. Using your favorite P2V tool, convert the physical machine to a virtual machine, storing the virtual machine on the removable media.
8. Move the removable media to a workstation outside the DMZ security zone. This workstation should now run a virus scan on the removable media. In addition, this workstation should most likely not be connected to the Web and should mount the removable storage read-only. A good system for this would be a forensic workstation or one blessed by the security team for this purpose. You are not looking for a virus within the VMDK, but one within the removable media itself that could have slipped on while it was within the hostile environment. If there is a virus or worm footprint, start the process over. However, be aware that VMDKs often show false positives.
9. Attach the now safe removable media to a workstation within your virtualization administration network.
10. Use VMware Converter or your favorite P2V tool to import the VM from the removable media into the virtual environment. The target virtual network should be a pseudo security zone that mimics the one from which the VM came.
11. Boot the VM within this protected environment and make any modifications as necessary. Common items to remove are any hardware agents and devices. Install VMware Tools, make any patches for the new hardware, and so on. This is also a good time to boot from a utility CD-ROM that contains tools to find rootkits and analyze the disk for viruses and other issues. Fix any issues found.
12. Power off the VM, move to the real DMZ virtual network, and then power on the VM.

These steps will guarantee that the P2V happens in a safe and secure fashion. There are many checks within these steps to test each phase of the P2V to prevent infiltration of viruses and worms into the administrative and virtual networks of the host.

Some may consider this overkill. However, I do not. It works very well to protect your investment minimally against virus infiltration and inadvertent information leakage while maintaining the integrity of the virtual environment.