consider changing log file format |
Post Reply 
|
| Author | |
MarvinFS 
Guest Group 
|
 Post Options
 Thanks(0)
 Quote Reply
 Topic: consider changing log file formatPosted: 07 April 2003 at 12:55pm |
|
Roberto, Please consider changing log files format to cvs or tabbed to be MORE suitable for analyzing with external tools. i advise something like this: (copy&paste it to wide console\screen) #Software: SpamFilter ISP |
|
 |
|
|
LogSat
Admin Group 
Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
Quote Reply
Posted: 07 April 2003 at 7:37pm |
|
We are going to have to find a balance between readability/format and troubleshooting usefulness. Having a clean log like the one you mention is good for cleanliness and readability. However it does not provide any indication of the steps SpamFilter performed to reach the reject decision. We are currently indicating step-by-step what happens in the various stages of a connection. We now have to figure out how to provide the same step-by-step detail, but in a more readable format. ..any suggestions? :-) Roberto Franceschetti |
|
|
|
|
MarvinFS
Guest Group
|
Post Options
Thanks(0)
Quote Reply
Posted: 07 April 2003 at 11:00pm |
|
I suggest to make some sort of events based logging system. I saw such a system is other software. There may be several events types and may be several severety types: 1. whenever mail is blocked it's block event 2. dns error, uploading to mail server error, maps errors, logging errors it's warnings or errors severety events. 3. all internal errors are to be critical severety event group. where spamfilter should decide what to do may be it will be safer to shutdown. make several check boxes in logging tab WHAT TO DISPLAY and LOG for each group of events (i.e i want to log only criticals and blocks, and to display only them too, i dont what to display passed messages) and refering to my prev. post with log file, we had there ip addresses, e-mal addresses, severety of event, type of event (block) and THE LAST column description WHAT CAUSED the messages to be blocked. in our case it may be maps lookup, or the message was blocked because of keyword filter's entry or, in the futute, actual senders domain mx check. I think something like that, i may give some more details. -------- |
|
|
|
|
StevenJohns
Senior Member 
Joined: 03 August 2006 Status: Offline Points: 119 |
Post Options
Thanks(0)
Quote Reply
Posted: 09 August 2006 at 3:58pm |
|
Is there any chance that you could implement (maybe as an option that you could turn on or off) a method of logging to the database. I'm thinking about a master table and a details table. The master table could hold emailID, date/time, from, to, subject, sender IP etc... while the details table could hold all of the transaction details for the emailID in question. This would make reporting a piece of cake, and we could then run an sql script to either delete or export entried that were x days/months old. SF would be the correct place to log this info, rather than trying to trawl through text based log files periodically.
|
|
|
|
|
WebGuyz
Senior Member
Joined: 09 May 2005 Location: United States Status: Offline Points: 348 |
Post Options
Thanks(0)
Quote Reply
Posted: 09 August 2006 at 6:11pm |
|
Db based logging sounds like a good option!! Would make life easier for running reports and getting stats for those who care about them, but keep text based for those who don't need that much info. |
|
|
http://www.webguyz.net
|
|
|
|
|
Desperado
Senior Member
Joined: 27 January 2005 Location: United States Status: Offline Points: 1143 |
Post Options
Thanks(0)
Quote Reply
Posted: 09 August 2006 at 6:27pm |
|
Oh my god! Do you realize how many hours I have spent with SawMill getting the logs to parse REAL NICELY and now we want to change. Will someone help me re-write the parsing plugin?
|
|
|
The Desperado
Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
|
|
|
|
WebGuyz
Senior Member
Joined: 09 May 2005 Location: United States Status: Offline Points: 348 |
Post Options
Thanks(0)
Quote Reply
Posted: 09 August 2006 at 6:41pm |
|
|
|
|
http://www.webguyz.net
|
|
|
|
|
LogSat
Admin Group
Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
Quote Reply
Posted: 09 August 2006 at 10:09pm |
|
Sorry... no plans to change logging format anytime soon. There are 3rd party utilities that rely on the current format, and we do not wish to create problems with them.
Logging to a database would prevent logging if there were any database issues, and for admins who have 100s of MB logs every day, the database would grow too large too soon. |
|
|
|
|
StevenJohns
Senior Member
Joined: 03 August 2006 Status: Offline Points: 119 |
Post Options
Thanks(0)
Quote Reply
Posted: 10 August 2006 at 4:15am |
|
As I suggested, this could be an option that you can turn on or off, also logging to text files could also be a turn on/off option so as to not upset people who have written apps based on the text logging. If there is a problem with the database, then you could write these critical errors to text files. With reference to the size of the logs, these people could, as I suggested earlier periodically either delete or export entries that are x days/months old. And, if you have customers with 100s MB of text logs every day, can you imagine the hassle they must have parsing these?? How do they efficiently trace a connection in real time to diagnose problems???? VERY hard I would suggest. As mentioned in other forum posts, logging is an ESSENTIAL part of a professional ISP, not one that apperas to be an afterthought,Just search your forum and see how many people are having problems with parsing the current text logs....why???? just stick the logs into a database (it's not hard, we already have one !) ....I bet 99% of your customers would be happier.
|
|
|
|
|
Web123
Newbie 
Joined: 26 January 2005 Location: Finland Status: Offline Points: 31 |
Post Options
Thanks(0)
Quote Reply
Posted: 10 August 2006 at 4:26am |
|
We really need to get all the stats directly from SF
|
|
|
|
|
Desperado
Senior Member
Joined: 27 January 2005 Location: United States Status: Offline Points: 1143 |
Post Options
Thanks(0)
Quote Reply
Posted: 10 August 2006 at 8:47am |
|
StevenJohns,
Just a comment or 2 on your post. As an ISP we are bound by "rules" about accountability. My comments may be slanted by that and my general experience of using log files for everything we do.
Your Post: "With reference to the size of the logs, these people could, as I suggested earlier periodically either delete or export entries that are x days/months old. And, if you have customers with 100s MB of text logs every day, can you imagine the hassle they must have parsing these?? How do they efficiently trace a connection in real time to diagnose problems???? VERY hard I would suggest."
I just had a situation with the FBI where they needed information for 14 months ago. Deleting logs is NEVER an option. Also, I really do not have any issues (hassles) parsing my logs and tracing anything and I have 3 separate machines with their own logging that each message passes through. I guess the "real time" ting is an issue but with 500,000 messages a day, real time is a relative term anyway.
Your Post: "we currently log everything to our main reporting DB so that each customer can get stats and graphs of exactly how many emails they have received within a certain time frame, and more importantly (for our billing) how much crap we have stopped going to their domain / mailbox. Our customers DEMAND this type of logging/reporting, otherwise how will they know what they are paying for"
We do all this with both SawMill, which looks at the actual log files and with custom SQL queries against the quarantine DB.
We also have scripts that run at midnight to archive all logs over 2 days old to a NAS server if we need to get back to them. Our logging is over 250MB a day (uncompressed) and I shudder to think what kind of machine I would need to do this logging in a DB that would not impact performance.
My 3 cents.
|
|
|
The Desperado
Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
|
|
|
|
WebGuyz
Senior Member
Joined: 09 May 2005 Location: United States Status: Offline Points: 348 |
Post Options
Thanks(0)
Quote Reply
Posted: 10 August 2006 at 9:56am |
|
Desperado, Since there is no 'One Size Fits All' then having a choice would be great. You can keep text logging or go with DB, the key is having the choice. You must be a public company to have to keep records for that long. Mine are deleted after 3 months. Can't give anyone what I don't have. |
|
|
http://www.webguyz.net
|
|
|
|
|
StevenJohns
Senior Member
Joined: 03 August 2006 Status: Offline Points: 119 |
Post Options
Thanks(0)
Quote Reply
Posted: 10 August 2006 at 10:13am |
|
Dan, I can see where you are comming from. As Webguyz says, there is no one size fits all, give us the option of how WE want to do our logging, rather than how we are TOLD that we have to do it. That's all. By the way, as LogSat was clearly worried about the size of the database....exactly how big is your DB?? If you have 250MB of logs each day, then your quarantine DB must be huge. If you keep your logs for over 14 months, is it reasonable to assume that you keep the quarantined email for as log too?? What would be the point of having a log saying "you emailed fred at 5:30 on 5/5/06, but I have no idea what the email content was"? I don't mean to sound picky, just wondered what an ISP of your size does.
Cheers
|
|
|
|
|
Desperado
Senior Member
Joined: 27 January 2005 Location: United States Status: Offline Points: 1143 |
Post Options
Thanks(0)
Quote Reply
Posted: 10 August 2006 at 11:02am |
|
StevenJohns, Our mail logs go back to 1999 but we do not care what the content of a message was/is except for our own internal mail and we use an exchange clone for that. Our SpamFilter Quarantine expires between 2 and 14 days depending on the company we are supporting. Our DB is about 13BG for that. WebGuys, We are not "public" but we are an ISP (privately owned). |
|
|
The Desperado
Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
|
|
|
|
StevenJohns
Senior Member
Joined: 03 August 2006 Status: Offline Points: 119 |
Post Options
Thanks(0)
Quote Reply
Posted: 10 August 2006 at 11:12am |
|
Dan,
This 13GB DB, is it MySQL?? How well is the quarantine DB performing?? Cheers |
|
|
|
Topic Options
|
Post Reply
|
|
Tweet
|
| Forum Jump | Forum Permissions You cannot post new topics in this forum You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |
This page was generated in 0.191 seconds.