Category Archives: RDO

Long Term OpenStack Usage Summary

Its been about 9 months since we first kicked off a limited production install of OpenStack. There were few variables we were very interested in: How much specialized support was needed to maintain a small OpenStack and stability.

The workload and management structure were as follows:

  • 3 Different projects – production, development and test.
  • Each project had its own private network and a network to access public network.
  • 10 long term instances – alive for nearly the entire duration of 9 month run.
  • About 10 short term instances – alive for about month each time
  • Work loads in production were real – any disruption had serious consequences.

Summary

There is no doubt OpenStack is a huge beast. However, once it was running it rarely required constant maintenance. Over the last year documentation and consistency of commands has improved making life easier. So far everything has been stable – most common issue is overflowing logs. In fact this caused a critical control node failure only recently which is described in the rest of this post.

Ceilometer Space Issue

The default setting of Ceilometer saves all events and alarms forever. Given that Ceilometer uses mongodb this monotonically increases the hard disk space consumed. Over 9 months it occupied over 33GB  or about 99% of the hard disk. One simple solution is RDO packstack or other distributions to set a sane defaults in ceilometer –  /etc/ceilometer/ceilometer.conf. For starters,  most non-monitor critical installations, we may change unlimited quota to some limit.

  1. Change time to live from forever to some thing sane like a week or a month.
  2. Record history – can’t see a reason why we need this in normal cases.

# Number of seconds that samples are kept in the database for
# (<= 0 means forever). (integer value)
time_to_live=604800

# Record alarm change events. (boolean value)
 #record_history=true
 record_history=True

Restart ceilometer.

Reduce elements in the database

Now we need to reduce items(documents) in the database. Need install mongo tools. Simply download from their website.

./mongo --host  172.0.0.10
>show dbs;
admin (empty)
ceilometer 0.125GB
local 0.031GB
test (empty)
>use ceilometer;
>db.meter.storageSize();
>db.meter.count();
>db.meter.remove({"timestamp":{"$lt":ISODate("2016-18-03T19:30:07.805Z")}});

But space occupancy is till 99% .

Ceilometer Space Issue – Continues

Enter Mongodb. It does not release space just because we reduced space. When you are at 99% occupancy by Mongodb, you can’t use db.repairDatabase() -which is supposed to return unused empty space. 

So, we have to take the hacky solution to drop and recreate the database.
./mongodump --host=172.0.0.10
./mongo --host 172.0.0.10
>use ceilometer;
>db.dropDatabase();
>exit
./mongorestore --host=172.0.0.10 --db=ceilometer dump/ceilometer/

This restores the database and reduces the consumed space. Given that we have limited the time upto which events are stored, we shouldn’t have to use this hack often.