Mongodb – delete old ‘Monitoring Agent’ metrics and ‘Automation Agent’ Debug logs via scheduler

mongodb

We use Ops Manager 1.6.2.251

# mongo
MongoDB shell version: 2.6.10
connecting to: test
> show dbs;
admin                   (empty)
alerts                  0.203GB
apiv3                   0.578GB
backupagent             0.078GB
backupalerts            0.078GB
backupbilling           0.078GB
backupconfig            0.078GB
backupjobs              0.078GB
backuplogs              2.077GB
backupoplogs            0.203GB
backupsnapshot          0.078GB
cloudconf               0.203GB
local                   0.078GB
mmsdb                   0.078GB
mmsdbagenterrors        0.328GB
mmsdbagentlog           1.078GB
mmsdbanalytics          0.078GB
mmsdbautomation         0.578GB
mmsdbautomationlog      1.578GB
mmsdbbilling            0.078GB
mmsdbconfig             1.078GB
mmsdbdays               0.078GB
mmsdbhours              0.203GB
mmsdbjobs               0.078GB
mmsdbminutes-even       0.203GB
mmsdbminutes-odd        0.203GB
mmsdbpings              0.578GB
mmsdbprofilerstats      0.078GB
mmsdbprovisioning       0.078GB
mmsdbprovisionlog       0.203GB
mmsdbqueues             0.078GB
mmsdbratelimit          0.078GB
mmsdbrrdcache           0.078GB
mmsdbrrdsampling        0.078GB
mmsdbserverlog          0.203GB
mongo-distributed-lock  0.078GB

Our MMS DB use 15 GB of 150 GB available. We can't extend the volume.
We don't use MMS for backup. We use mongodump on other node.

Ops Manager Application Database This database contains application
metadata used by the Ops Manager Application. The database stores: •
Monitoring data collected from Monitoring Agents. • Metadata for Ops
Manager users, groups, hosts, monitoring data, and backup state. For
topology and specifications, see Ops Manager Application Database
Hardware.

At the moment we only enabled Monitoring Agent on Prod. But we wish to enable it on all environments, but cleanup metrics data older than 30 days.

In Oracle you can do something like this. This statement can be scheduled jobs with Oracle Scheduler.

 DELETE
  FROM YOUR_TABLE
 WHERE creation_date <= TRUNC(SYSDATE) - 30

The same we wish to to with Automation agent DEBUG logs. Howto do this with MongoDB MMS meta databases?

From FAQ

What is the data retention policy for Ops Manager? Ops Manager retains
two distinct types of data: metrics, which describe usage; and
snapshots, which back up your data.

Data-retention policies, as defined in the Terms of Service, are
always subject to change.

As of this writing, Ops Manager preserves:

Minute-level metrics for 48 hours. Hourly metrics for 94 days.
Snapshots according to their retention policy.

Best Answer

>use mmsdbagentlog
switched to db mmsdbagentlog
> show collections
app.agentLogs
> db.app.agentLogs.stats()
{
    "ns" : "mmsdbagentlog.app.agentLogs",
    "count" : 1711298,
   "size" : 524287948,
    "avgObjSize" : 306,
    "storageSize" : 71323648,
    "capped" : true,
    "max" : -1,
    "maxSize" : 524288000,
    ... }

Looking in OpsManager Database , they use capped size collection to store the agent log , it seems we don't need to worry about the volume. If the collection confuse you is not capped size , you could try to schedule a job with crontab in Linux(just an example) to remove those records :

db.collection.remove({_id : {$lt : ObjectId('55b0d1c40000000000000000')}})

first 8 character is hexadecimal timestamp can be generate in Linux Shell with command :

Today=`date +%F`
T=`date --date="$Today 00:00:00" +%s`
Stamp=`printf "%X" $T`''0000000000000000
echo $Stamp
Related Question