Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - logparse rrdtool
  FAQ FAQ  Forum Search   Register Register  Login Login

logparse rrdtool

 Post Reply Post Reply
Author
danego View Drop Down
Newbie
Newbie


Joined: 22 March 2011
Location: NL
Status: Offline
Points: 9
Post Options Post Options   Thanks (0) Thanks(0)   Quote danego Quote  Post ReplyReply Direct Link To This Post Topic: logparse rrdtool
    Posted: 08 June 2016 at 5:25pm
Hi,
 
I have written a vbs script to get some information from the logfile and put it in rrdtool.
Then we generate an image with rrdtool.
 
At the moment I collect #connections, rejected mails, forwarded mails, quarantined mails.
 
You can schedule the script with the windows task scheduler.
I run the script every 5 minutes to get the data in rrdtool.
The image then shows the # of messages per minute (this is the average of the 5 minutes)
 
The script writes a tempfile in the logfile folder to keep up where it left off.
It detects a log rotate and cleans up the tempfile from yesterdays logfile before it starts with the new day.
 
The script is almost done, was just curious if anyone is interested.
(needs more comment and I bet it's not monkey proof yet :)
 
Regards,
Emiel.
 
Back to Top
danego View Drop Down
Newbie
Newbie


Joined: 22 March 2011
Location: NL
Status: Offline
Points: 9
Post Options Post Options   Thanks (0) Thanks(0)   Quote danego Quote  Post ReplyReply Direct Link To This Post Posted: 09 June 2016 at 10:18am
Here is an image with an example of this
 
http://danego.net/spamfilter/
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 09 June 2016 at 5:54pm
Great job danego! We ourselves here at LogSat could use your script on our server :-) Plus 1 for us for a copy of it!
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
yapadu View Drop Down
Senior Member
Senior Member


Joined: 12 May 2005
Status: Offline
Points: 297
Post Options Post Options   Thanks (0) Thanks(0)   Quote yapadu Quote  Post ReplyReply Direct Link To This Post Posted: 13 June 2016 at 9:26pm
That is certainly impressive looking, I would love to get a copy of that as well.
--------------------------------------------------------------
I am a user of SF, not an employee. Use any advice offered at your own risk.
Back to Top
danego View Drop Down
Newbie
Newbie


Joined: 22 March 2011
Location: NL
Status: Offline
Points: 9
Post Options Post Options   Thanks (0) Thanks(0)   Quote danego Quote  Post ReplyReply Direct Link To This Post Posted: 16 June 2016 at 4:27pm
Ok here is the download link of the script with an extra textfile with the rrdtool commands used.
 
 
You also need rrdtool for windows :
 

There is an rrdtool.exe in of the the folders.
 
 
I have copied our current running script.
(posted the latest day and week graph on www.danego.net/spamfilter)
 
Always curious if someone else but me gets it up and running without any problems.Smile
 
Be aware you need to run it at least three times with 5 minutes intervals before you see something appear in the graphs.
 
It counts the different values from the log and adds this to the rrdtool database.
rrdtool recalculates this to a value per second and during the graph generation it gets multiplied by 60 to get a count per minute.
So the value is per minute from a 5 minute average.
 
 
 
Back to Top
yapadu View Drop Down
Senior Member
Senior Member


Joined: 12 May 2005
Status: Offline
Points: 297
Post Options Post Options   Thanks (0) Thanks(0)   Quote yapadu Quote  Post ReplyReply Direct Link To This Post Posted: 19 June 2016 at 1:13am
Thanks for the code.  Thought I would share my findings so far.

Just running it, VBS crashes on this line:

EpochUTC = DateDiff("s", "01/01/1970 00:00:00", Left(DateAdd("n", UTC_Offset, Now()),Len(DateAdd("n", UTC_Offset, Now()))-2) & "00")

I think you are trying to get the date (with UTC offset taken into consideration) and then remove the seconds?

I converted it to this to make it run on my system:

EpochUTC = DateDiff("s", "01/01/1970 00:00:00", FormatDateTime(DateAdd("n", UTC_Offset, Now()),2) & " " & FormatDateTime(DateAdd("n", UTC_Offset, Now()),4))

That gets past the error.  The link to rrdtool does not contain any .exe file that I can find.

I found a compiled copy on this page, way at the bottom (direct link).  That has a compiled version and some associated files it needs.  I just put them all into the same directory as your VBS and it runs now.

Now I'm working on a problem where the VBS assumes the logs are all in the format yyyymmdd.log but spamfilter has an option to rotate the logs more frequently and may contain a number after, like this yyyymmdd-x.log and the script does not pick up on that.


Edited by yapadu - 19 June 2016 at 1:32am
--------------------------------------------------------------
I am a user of SF, not an employee. Use any advice offered at your own risk.
Back to Top
yapadu View Drop Down
Senior Member
Senior Member


Joined: 12 May 2005
Status: Offline
Points: 297
Post Options Post Options   Thanks (0) Thanks(0)   Quote yapadu Quote  Post ReplyReply Direct Link To This Post Posted: 19 June 2016 at 9:50am
Have had some success on this end, thanks for the code.

I've modified it so it does not need to run every 5 minutes, it can calculate the timestamp needed from the logs themselves.

The result is it can now also process historical data (assuming you start the RRD DB back far enough).

Now I am looking for a way to merge the databases from multiple servers to get one complete picture of everything.
--------------------------------------------------------------
I am a user of SF, not an employee. Use any advice offered at your own risk.
Back to Top
danego View Drop Down
Newbie
Newbie


Joined: 22 March 2011
Location: NL
Status: Offline
Points: 9
Post Options Post Options   Thanks (0) Thanks(0)   Quote danego Quote  Post ReplyReply Direct Link To This Post Posted: 26 June 2016 at 5:18pm
Hi, I was on a short break last week.
Good to read that you modified the script to your situation and that you found it useful.
 
I never had the need to change the logrotation so I was unaware of this feature in logsatSmile
 
in my post I also had a url to the rrdtool website to download the exe.
 
When I read what you have modified it shouldn't be that hard to combine multiple servers?
It's a more or less simple add up of some numbers :)
 
One thing I want to add to the script is the number of files in the queue folder.
This should be very low but if there is a problem with your mailserver this queue folder could go skyhighSmile
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.211 seconds.