Resetting Calamari password

Calamari‘ is the monitoring interface for a Ceph cluster.

The Calamari interface password can be reset/changed using the ‘calamari-ctl’ command.

# calamari-ctl change_password –password {password} {user-name}

calamari-ctl can also be used to add a user, as well as disable, enable, and rename the user account. A ‘–help’ should print out all the available ones.

# calamari-ctl –help

Compacting a Ceph monitor store

The Ceph monitor store growing to a big size is a common occurrence in a busy Ceph cluster.

If a ‘ceph -s‘ takes considerable time to return information, one of the possibility is the monitor database being large.

Other reasons included network lags between the client and the monitor, the monitor not responding properly due to the system load, firewall settings on the client or monitor etc..

The best way to deal with a large monitor database is to compact the monitor store. The monitor store is a leveldb store which stores key/value pairs.

There are two ways to compact a levelDB store, either on the fly or at the monitor process startup.

To compact the store dynamically, use :

# ceph tell mon.[ID] compact

To compact the levelDB store every time the monitor process starts, add the following in /etc/ceph/ceph.conf under the [mon] section:

mon compact on start = true

The second option would compact the levelDB store each and every time the monitor process starts.

The monitor database is stored at /var/lib/ceph/mon/<hostname>/store.db/ as files with the extension ‘.sst‘, which is the synonym for ‘Sorted String Table

To read more on levelDB, please refer:

https://en.wikipedia.org/wiki/LevelDB

http://leveldb.googlecode.com/svn/trunk/doc/impl.html

http://google-opensource.blogspot.in/2011/07/leveldb-fast-persistent-key-value-store.html

What is data scrubbing?

Data Scrubbing is an error checking and correction method or routine check to ensure that the data on file systems are in pristine condition, and has no errors. Data integrity is of primary concern in today’s conditions, given the humongous amounts of data being read and written daily.

A simple example for a scrubbing, is a file system check done on file systems with tools like ‘e2fsck’ in EXT2/3/4, or ‘xfs_repair’ in XFS. Ceph also includes a daily scrubbing as well as weekly scrubbing, which we will talk about in detail in another article.

This feature is available on most hardware RAID controllers, backup tools, as well as softwares that emulate RAID such as MD-RAID.

Btrfs is one of the file systems that can schedule a internal scrubbing automatically, to ensure that corruptions are detected and preventive measures taken automatically. Since Btrfs can maintain multiple copies of data, once it finds an error in the primary copy, it can check for a good copy (if mirroring is used) and replace it.

We will be looking more into scrubbing, especially how it is implemented in Ceph, and the various tunables, in an upcoming post.