logparse rrdtool
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=7136
Printed Date: 26 December 2024 at 7:39am
Topic: logparse rrdtool
Posted By: danego
Subject: logparse rrdtool
Date Posted: 08 June 2016 at 5:25pm
|
Hi, I have written a vbs script to get some information from the logfile and put it in rrdtool. Then we generate an image with rrdtool. At the moment I collect #connections, rejected mails, forwarded mails, quarantined mails. You can schedule the script with the windows task scheduler. I run the script every 5 minutes to get the data in rrdtool. The image then shows the # of messages per minute (this is the average of the 5 minutes) The script writes a tempfile in the logfile folder to keep up where it left off. It detects a log rotate and cleans up the tempfile from yesterdays logfile before it starts with the new day. The script is almost done, was just curious if anyone is interested. (needs more comment and I bet it's not monkey proof yet :) Regards, Emiel. |
Replies:
Posted By: danego
Date Posted: 09 June 2016 at 10:18am
|
Here is an image with an example of this |
Posted By: LogSat
Date Posted: 09 June 2016 at 5:54pm
|
Great job danego! We ourselves here at LogSat could use your script on our server :-) Plus 1 for us for a copy of it! ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 13 June 2016 at 9:26pm
|
That is certainly impressive looking, I would love to get a copy of that as well. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: danego
Date Posted: 16 June 2016 at 4:27pm
|
Ok here is the download link of the script with an extra textfile with the rrdtool commands used. http://www.danego.net/spamfilter/logsatparser_v1.zip" rel="nofollow - http://www.danego.net/spamfilter/logsatparser_v1.zip You also need rrdtool for windows : http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.2.30-win32-perl510.zip" rel="nofollow - http://oss.oetiker.ch/rrdtool/pub/rrdtool-1.2.30-win32-perl510.zip There is an rrdtool.exe in of the the folders. I have copied our current running script. (posted the latest day and week graph on http://www.danego.net/spamfilter" rel="nofollow - www.danego.net/spamfilter ) Always curious if someone else but me gets it up and running without any problems.  Be aware you need to run it at least three times with 5 minutes intervals before you see something appear in the graphs. It counts the different values from the log and adds this to the rrdtool database. rrdtool recalculates this to a value per second and during the graph generation it gets multiplied by 60 to get a count per minute. So the value is per minute from a 5 minute average. |
Posted By: yapadu
Date Posted: 19 June 2016 at 1:13am
|
Thanks for the code. Thought I would share my findings so far. Just running it, VBS crashes on this line: EpochUTC = DateDiff("s", "01/01/1970 00:00:00", Left(DateAdd("n", UTC_Offset, Now()),Len(DateAdd("n", UTC_Offset, Now()))-2) & "00") I think you are trying to get the date (with UTC offset taken into consideration) and then remove the seconds? I converted it to this to make it run on my system: EpochUTC = DateDiff("s", "01/01/1970 00:00:00", FormatDateTime(DateAdd("n", UTC_Offset, Now()),2) & " " & FormatDateTime(DateAdd("n", UTC_Offset, Now()),4)) That gets past the error. The link to rrdtool does not contain any .exe file that I can find. I found a compiled copy on https://sites.google.com/site/christiankrausse/projekte/perl/rrdtool-v1-4-8-win32-binaries-rrds-for-active-state-perl-5-16-2-windows" rel="nofollow - this page , way at the bottom ( https://sites.google.com/site/christiankrausse/projekte/perl/rrdtool-v1-4-8-win32-binaries-rrds-for-active-state-perl-5-16-2-windows/RRDs_1_4008_Release_Bin.zip?attredirects=0&d=1" rel="nofollow - direct link ). That has a compiled version and some associated files it needs. I just put them all into the same directory as your VBS and it runs now. Now I'm working on a problem where the VBS assumes the logs are all in the format yyyymmdd.log but spamfilter has an option to rotate the logs more frequently and may contain a number after, like this yyyymmdd-x.log and the script does not pick up on that. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: yapadu
Date Posted: 19 June 2016 at 9:50am
|
Have had some success on this end, thanks for the code. I've modified it so it does not need to run every 5 minutes, it can calculate the timestamp needed from the logs themselves. The result is it can now also process historical data (assuming you start the RRD DB back far enough). Now I am looking for a way to merge the databases from multiple servers to get one complete picture of everything. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: danego
Date Posted: 26 June 2016 at 5:18pm
|
Hi, I was on a short break last week. Good to read that you modified the script to your situation and that you found it useful. I never had the need to change the logrotation so I was unaware of this feature in logsat in my post I also had a url to the rrdtool website to download the exe. When I read what you have modified it shouldn't be that hard to combine multiple servers? It's a more or less simple add up of some numbers :) One thing I want to add to the script is the number of files in the queue folder. This should be very low but if there is a problem with your mailserver this queue folder could go skyhigh
|