In the context of virtualization, backing up VM images to storage nodes involves moving very large files. Many VM images are just copies of the OS and data on top. So data deduplication and compression must offer great savings. In our search, we found various utilities which we list later down. But we settled into reviewing two popular ones zbackupand attic . Another popular tool bupwas considered but few things like unable to prune old versions was major point for us.
The main requirements were data deduplication, compression, easy to script with and encryption all in one tool. In this article, we will give a background on their usage on CentOS 7.1. We don’t plan on extensive evaluation of various other capabilities as we are looking for these basic features to be done well.
ZBackup describes itself as a globally-deduplicating backup tool originating its inspiration from bup and rsync tools. As you add more files to the archives, it will store duplicate regions once. It also supports AES encrypted files.
Installing ZBackup on CentOS 7.1
ZBackup is the easiest to install. Its available in the EPEL repos and you can simply do yum install zbackup.
The archive is called a repository in ZBackup. Its nothing but a folder created for the tool to use where it stores its metadata and all the files added into it for backup.
First step is to initialize the folder , say zbak, with metadata folders.
zbackup init --non-encrypted /kvmbackup/zbak
If you need encryption, you can enable it with a key file.
When constructing complex network in the cloud, you run in to situations where packets seem to be silently dropped. One reason could because of Reverse Path Forwarding/Filter (RPF) check at the routing decision step. In RPF check, when the kernel is making the routing decision for a given packet, the kernel (1) notes the interface this packet arrived on, (2) verifies the source address is valid by finding which interface to use from its routing table, (3) and the interface that will be used for reaching back to the source address is the same as the interface this packet arrived.
If the above check passes, the packet is allowed to go forward.
To reduce spoofed addresses. Reverse Path Forwarding is actually defined in an IETF RFC document 3704 as a best practice. This is more applicable in edge routers and internet facing systems.
However, the default value in CentOS is set to strict. This causes packet drops.
In most cases, if you are not connected to the internet or untrusted networks, it should be fine to disable it or make it ‘loose’ source validation – which means if the packet has a valid route via any interface not just via the arrival interface, it is accepted.
Turn it off?
If your node is not internet facing, generally prefer to turn it off. If its on the internet edge, you may want to think twice as turning it off at that node indicates some network config isn’t natural.
Here’s how to check current values:
sysctl -a | grep \\.rp_filter
How to temporarily turn it off:
[root] # for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do > echo 2 > $i > done
In the above, we are setting a value of 2 which is ‘loose source check’.
Or to permanently set it
sysctl -w "net.ipv4.conf.all.rp_filter=2"
Note: Setting of all, may not be enough. You may have to set individual interfaces.
How to detect RPF is dropping packets
Enable log_martians to see log entries in /var/log/messages that the packets are being dropped :
Its hard to detect in Iptables. But it would be in the class of ‘missing log’ entries. Usually it would show as packet that exited the last rule in PREROUTING, but does not appear on the first rule of POSTROUTING . You can add Iptables log chain rules to trace this. But usually if you suspect drop in routing, enable the log_martians value is a much easier way.
You can make permanent by adding the above rule to /etc/sysconfig/iptables (before the reject rules) saving and restarting iptables.
Config Changes on Compute Node
. . .
# Enable spice related features (boolean value)
# Enable spice guest agent support (boolean value)
# Keymap for spice (string value)
# service openstack-nova-compute restart
# service httpd restart
# service openstack-nova-spicehtml5proxy start
# service openstack-nova-spicehtml5proxy status
# systemctl enable openstack-nova-spicehtml5proxy
Here the control node is an HTML proxy that connects to the SPICE server+port that is running when a VM is instantiated.
Here are some notes on some of the unclear options:
This line indicates the HTML5 proxy should run on localhost without IP binding (0.0.0.0) – control node in this case.
This indicates the base URL to use when you click ‘console’ on the Horizon dashboard. Its noted that this URL must be accessible in the same network as the Horizon dashboard. In our case, this URL is the control node.
Server listen specifies where the VM instances should listen for SPICE connections. This is the local IP address (compute node)
Server_proxyclient_address is the address which clients such as HTML5 proxy will use to connect to the VMs running on the Compute Node. This is an internal address most likely not accessible to the outside world but accessible to the control node. This address is the internal IP address of the compute node.
Be sure about what config change goes in which node. Iptables is another to look out for, if you plan to use consoles regularly, make the iptables rules permanent.
“console is currently unavailable. Please try again later.”
Under the hood,
“ERROR: Invalid console type spice-html5 (HTTP 400)”
when you do
nova get-spice-console spice-html5
This generally means, the VM did not start with SPICE enabled. The causes for that could be one of the services did not restart after config change.
Double check the config file – make sure ‘enabled=true’ is set.
The external networks for an OpenStack deployment can be a combination of your intranet corporate IT network and internet facing external network. For common use case, we look a way to add another external network to OpenStack deployment.
The steps for creating multiple external networks are:
1. Create a routed Virtual Network with 10.0.10.0/24 — This is needed if you are using a multi VM node on a single physical node.
2. Add a NIC connected to the new network to the neutron node
3. Say the device show up as eth3. Create ifcfg-eth3 as and add to a new bridge say br-ex2 on the neutron node.
On the Neutron Nodes, update /etc/neutron/l3_agent.ini to empty values
On OpenStack Kilo, when we use RDO to enable Swift Object Storage service its partially misconfigured (or lack of control in packstack file).
The Swift Proxy is setup on the storage node. I could not find if I can control which node the Swift Proxy can be installed on by packstack. The issue is the swift proxy service endpoint (points to control node) mismatches where the swift proxy really is (on storage node).
Check Swift Endpoint details
Ensure swift service is indeed created.
openstack service list
| ID | Name | Type |
. . .
| a43e0d3e0d3e0d3e0d3e0d3e0d3e0d3e | swift | object-store |
| a5a23a23a23a23a23a23a23a23a23a23 | swift_s3 | s3 |
If swift does not show then you may not have installed it during packstack install. Edit your packstack file to install only swift.
openstack endpoint show swift
| Field | Value |
| adminurl | http://controller_ip:8080/ |
| enabled | True |
| id | c243243243243243243243243243243 |
| internalurl | http://controller_ip:8080/v1/AUTH_%(tenant_id)s|
| publicurl | http://controller_ip:8080/v1/AUTH_%(tenant_id)s|
| region | RegionOne |
| service_id | a43e0d3e0d3e0d3e0d3e0d3e0d3e0d3e |
| service_name | swift |
| service_type | object-store |
The above is wrong. This is because no one is listening on port 8080. Check this by : netstat -plnt | grep 8080
Luckily, everything seems to be setup on the storage node – port 8080 is up, iptables rule for 8080 is set and swift files are all almost good to go.
Correcting the Endpoint
Delete the current swift endpoint ID. On the control node,
The /etc/swift/proxy-server.conf file on the storage_ip must be edited as below. Especially, the identity_uri and auth_uri must point to Keystone IP. One other minor thing to check is if /var/cache/swift that is used for signing directory has correct selinux context. You may try sudo restorecon -R /var/cache/
The important part is the filter_factory its — keystonemiddleware and not keystone.middleware. Then restart the swift proxy service.
sudo service openstack-swift-proxy restart
Testing the Swift S3 API using S3Curl
S3Curl is a tool provided by Amazon. It can be downloaded from https://aws.amazon.com/code/128. Also note the comment in that page where you need to yum install perl-Digest-HMAC package.
You can use Horizon to create a test container and upload a small text file into it.
In our example, we have created a container called “test_container” and simple text file called “test_obj” inside the container.
Make sure you edit the s3curl.pl file to use Openstack’s Swift Proxy end point:
my @endpoints = ( '184.108.40.206');
Retrieve the access keys from Horizon dashboard
Go to Project -> Compute -> Access & Security. Click on the API Access tab.
Note the S3 Service endpoint. In our case: http://220.127.116.11:8080
On the top right click on view credentials:
“EC2 Access Key” –> Is your id for S3 tools such as S3Curl.
“EC2 Secret Key” –> Is your key for S3 tools such as S3Curl.
In our venture into IaaS, we considered a small experimental project to install OpenStack on a single host with a view that when we get budgeted for multiple physical machines, we should be able to scale up from our 1 physical machine to N-physical machine with ease without fear of losing configuration. AKA Project OpenStack-in-a-box. In this article, we use OpenStack Kilo, CentOS 7.1 release using RDO Packstack .
What we get with All-in-One or multi node RDO Packstack?
In this article, we provide a multi node OpenStack installation on a single physical node without using nested virtualization. In our implementation, except the compute node all other nodes are virtual nodes. RDO Packstack provides an all-in-one or multi node install option. Depending on if you are deploying on bare metal server or inside a VM, the lay of the land looks like in figure below for all VM deployment:
In the above picture, the individual control nodes such as Neutron & Compute are Virtual Machines themselves. The instances spawned from OpenStack are running inside the Compute VM. While nested virtualization fine, performance is not the best.
So why not use Packstack all in on bare metal? Solves performance but makes it hard to migrate out easily.
One theory followed is, moving entire VMs or LVM blocks is one of the simpler and safer ways to migrate a system compared to moving configuration files or databases. This is especially so of control nodes such as Horizon/Neutron.
What do we want?
With performance and ease of scale out in mind, we wanted all control nodes as VMs and the single compute node as the host itself. Its easier to replace or add new compute node than to move data from to make a new control node. On the other hand, Compute Nodes are treated as more dispensable. So when we get our shiny new physical servers, theoretically, we move our control nodes to the new machines. Environment we are aiming for is shown in the figure below:
In the above figure, we want the physical host to be the Nova Compute node. The other control nodes such as Neutron, Horizon, Cinder etc are VMs that are manually created and running on the physical host.
Say we scale out, we leave the physical host as compute node and move the VMs to the new hardware still running as VM.
For sake of pushing the limits, we wanted to keep available physical NICs to exactly 1.
We had some requirements in this project:
Instances must be as fast as a running a VM in KVM
Nested KVM was considered, but many companies don’t offer support for their software if run in a nested KVM environment.
Easy to implement in enterprise environments
Easy to migrate to multi physical out when we get more physical machines.
Must use only one physical NIC and 1 physical host
Must be repeatable
Security wise, iptables of the host is difficult to control because Nova updates iptables for instance launch. But at least its in one server for now. One big thing we lose with this configuration is High Availability – which by definition needs 2 physical nodes. While you can implement HA as VMs, but physical or environmental failure means the environment is lost. So beware of this case.
We evaluated various options:
Too complex for simple setups as it targets multi hypervisors. Hard to justify any experiments, unless in a heterogeneous environment.
All virtual machines with nested KVM
Performance limited. Not supported in enterprise environment (Red Hat and such)
Packstack all in one
We eventually use packstack with multi node config, but this requires a little pre configuration
We need a refined approach.
The solution has a key concept: Use the physical host to do two things:
1. Be the compute host (Nova) and
2. Run few virtual machines that we manually made from XML via virsh or virt-manager that are used as controller, network, storage, authentication (FreeIPA) nodes. Everything else was driven by OpenStack.
…And the Trick for Packstack
Packstack install scripts need the name of the vNIC on Compute Node to match the name of the NIC on neutron. So we rename our virtual bridge name to the vNIC’s name on Neutron.
The overall single physical host OpenStack setup is shown in the following diagram.
In the above picture, the important pieces are:
The network setup (virbrx) – Manually defined networks in XML for virsh.
Our OpenStack internal network usually gets named virbr1. We edited the network XML to have bridge name = eth1. This is important to satisfy a packstack install requirement that interface name on control and compute match.
The manually created Virtual Machines for use as OpenStack nodes such as Network, Storage and Controller nodes. These are simple CentOS templates with static IPs.
After packstack is run, the OpenStack spawned VMs (“Open Stack VM Instance x”) communicate via the virbrx that was setup for OpenStack.
Here are the things necessary for this setup
Step 1: Create few virtual machines by hand or virt-manager.
Step 2: These virtual machines are controller, network and storage
Step 3: Create LVMs that you can attach to the storage VM
Step 4: Create the networks that OpenStack uses via virsh
Step 5: The important part is the host should become a part of private network
External IP Addresses
In general, the external network is any network that is connected to network host/L3 host. It could be your upstream ISP network or another internal corporate network. Normally, you’ll have multiple IP addresses assigned so that can be used as floating IP address ranges. Managing how to connect these IP addresses to the Neutron host is a decision based on performance and features requirements.
Note that we are using only one NIC. Sometimes we may need the NIC to access the host so it may not be possible to do PCI device pass through of the NIC to the Neutron host.
Here are few options for using the NIC:
Bridging needs to be done carefully as it has a performance impact because the NIC becomes a slave to the bridge and enters promiscuous mode, listening to all traffic on the network.
Use libvirt‘s routed network to route the traffic to the Neutron host. By itself, this is not enough because you need to set the routes of your public IP block to point to your host as the router. If you have control over the upstream router, its best to add static routes that direct the public IP to your physical host as the the gateway. Then the physical host will forward it to the internal virtual bridge serving the external network. This is a clean and more maintainable approach.
If you have no control over the upstream router, Proxy ARP method can be used. This needs to be carefully handled.
Proxy ARP is a mode of the NIC where it answers to ARP requests for all IP addresses for which it has a non-default route (i.e static route that is not a default gateway)
proxy_arp can easily lead to network issues if used without care. Common problem is the host responds to ARP requests for answer for internal (virtual) networks to requests coming from the external network.
To be good citizen, you may add bridge filter rules to drop outgoing ARP requests that are not sourced from your public IP list. Or drop incoming ARP requests that don’t match the public IP addresses that you intend to answer for.
When installing with packstack there are some requirements that make us do interesting things. The first one is to rename virbr1 on the physical host to eth1 by editing the XML used for creating the network in virsh. This is needed because the packstack expects the interface on Nova and Controller node to have the same name (for ex. eth1 on both). If you’re uncomfortable with that naming, after packstack install, you could destroy the ‘eth1’ virtual network and rename it back to ‘virbr1’ before going back to full operations. For general operation, it does not matter what the virtual network is named as.
You’ll have to decide which way you want the naming to be. This depends mainly on what NIC and how many you have. We decided ‘eth1’ will be our NIC’s name in the control node. Our server did not use eth1 so it saved an additional step of renaming our physical NICs.
Planning the networks
The NIC to network mapping are provided as a reference:
eth2 -> 18.104.22.168/29 or /28 [External network, IP address provided by data center]
eth1 -> 172.16.xx.0/24 [OpenStack Data network]
eth0 -> 192.168.xx.0/24 [Corporate NAT access network to internet ]
IP address Management – Floating and Internal
An overall network diagram is shown in the following diagram. This also marks out IP addresses that are used in various networks. 111.0.0.x is external IP (floating IP). Other addresses are internal.
The key aspect to understand about Floating IP network is the host owning the physical NIC (peth0) acts as a gateway. When the virtual network 22.214.171.124/28 is started in libvirt. This causes libvirt to add the routes to the physical host’s routing table. This presence of route to 126.96.36.199/28 network allows the physical NIC to accept packets arriving at the NIC for forwarding to its final destination in the Neutron host’s br-ex bridge which further routes it to the correct OpenStack VM instance that the IP has been assigned to. proxy_arp also causes the peth0 to respond to any ARP requests for the IP addresses in the route table. This means for security, we need an iptable rule to drop packets arriving at peth0 that are not destined for the floating IP or the peth0 range.
Any request for the 188.8.131.52 network from the OpenStack instances goes all the way from the OpenStack router to Phys Host as they are really on connected bridge:
OpenStack Router -> Br-Ex -> virbr0 -> routing table on physical host -> final interface selection.
Any request for other IP addresses uses the gateways:
IF IP Recoverable? Why?
peth0 184.108.40.206 No Network (zero) address can't be used by VMs anyway.
virbr0 220.127.116.11 No Hop 0 routing gateway for OpenStack instances.
This gateway has a route to datacenter/ISP gateway.
For security, we should have IP tables to reject
INPUT to 18.104.22.168 from instances.
br-ex 22.214.171.124 Yes Neutron host br-ex does not need a public IP.
OpStkRtr 111.0.0.x No Acts as gateway for instances
How to recover IP addresses?
Permanently: On the Neutron host (after you install OpenStack with Packstack) Update the ifcfg-br-ex to have no IP address and add a static route in route-br-ex. This means unless there is another NIC with access to external network, Neutron node itself will not have access to external network on br-ex. This is good for security no one can access Neutron host using external IP. It also saves a precious external IP address.
Prepare the host for packstack
Host is CentOS 7.1 with SELinux enabled.
Disable network manager:
service NetworkManager stop
chkconfig NetworkManager off
service network start
chkconfig network on
Disable firewalld because many scripts are not yet firewalld ready.
The main control nodes of OpenStack – Controller/Horizon, Neutron, Storage are all virtual machines. Before you begin make sure you have atleast virsh command installed. These VMs are created using virt-manager. If you plan not to install X window system, you can edit the XMLs based on some existing template XML after dumping it out using virsh-dumpxml command.
VM Sizing Guide
Overall, the size of OpenStack cloud is determined by the power of your CPU and memory. A dual socket with copious amount of RAM usually gives better runway for maintaining this setup for slightly longer. You’ll also need a lot of HDD space to hold all the VMs (including instances launched from OpenStack). We had 2 low end Haswell Xeon CPUs with 6 cores each. Giving us 24 cores with HT, 64GB of RAM & 1.5TB HDD.
Controller is heavy on various communications and CPU. General guideline is to allow atleast 4 vCPUs and 4 to 6 GB on controller. Neutron is CPU heavy so 2-4 vCPUs with 2 GB memory for a small sized cloud.
Roughly, 12 to 14 vCPUs were used for various control nodes (control, neutron, storage) with about 12 to 14 GB memory.
In reality only, 1-2 control VMs are actually actively used other are generally idle.
So most of the vCPUs can be over allocated. Memory may not be that flexible.
A separate VM which has attached LVMs created on the physical host. This allows for easy management of storage space for Cinder and Swift. Once we have a separate storage server, its easy to move and add more storage to the VM.
In our scenario, we use another separate VM for NFS for our internal reasons. But NFS can also be part of the VM. Note that for packstack, the LVM must already have a Volume Group: ‘cinder-volumes’ created.
Note that /etc/hosts and hostname must be be correctly set
External Network Setup
Conceptually, open stack connects to the external network via bridge (usually OVS). For production type usage, you may have multiple external networks but basic concept is the same: create bridge to the NIC that is connected to the external network and tell Opens Stack about it.
Next, set how to route traffic to br-ex. This is needed if neutron has a different default gateway than that would carry the external traffic.
So create a following file.
File: route-br-ex (make it chmod +x)
#Gateway(Note can't add GW in 0
# - route not found error - probably a bug)
GATEWAY1=126.96.36.199 #Note this should be the reachable IP on virbr0