PUBLIC MARKS from camel with tag log

CMON is a daemon that aggregates information from MySQL Cluster that earlier was only accessible from the cluster log or the management client, such as: * cluster state * node state * backup statistics * statistics * cluster events (cluster log basically) .. and let's you access the information using SQL, because CMON logs the information into ordinary MYISAM tables! So, it is really easy to use! In the package you also get get php scripts that you can put on your webserver to generate graphs and get a www interface to CMON. CMON can also start ndbd nodes and make decisions on how they should be started (with or without --initial). CMON starts as a daemon and will automatically create cmon database and install the necessary tables automatically.

Screen is a full-screen window manager that multiplexes a physical terminal between several processes. In short, you can have a few virtual terminal by using only one physical terminal. Screen is useful when you are accessing servers remotely, or running jobs on the background. When you want to run jobs on the background, it is very useful to run it inside one screen so that you can detached it, and simply log out without worry. Below are a few useful command to be used with screen to get you started.

Many people use SSH to log in to remote machines, copy files around, and perform general system administration. If you want to increase your productivity with SSH, you can try a tool that lets you run commands on more than one remote machine at the same time. Parallel ssh, Cluster SSH, and ClusterIt let you specify commands in a single terminal window and send them to a collection of remote machines where they can be executed. Why you would need a utility like this when, using openSSH, you can create a file containing your commands and use a bash for loop to run it on a list of remote hosts, one at a time? One advantage of a parallel SSH utility is that commands can be run on several hosts at the same time. For a short-running task this might not matter much, but if a task needs an hour to complete and you need to run it on 20 hosts, parallel execution beats serial by a mile. Also, if you want to interactively edit the same file on multiple machines, it might be quicker to use a parallel SSH utility and edit the file on all nodes with vi rather than concoct a script to do the same edit. Many of these parallel SSH tools include support for copying to many hosts at once (a parallel version of scp) or using rsync on a collection of hosts at once. Because the parallel SSH implementations know about all the hosts in a group, some of them also offer the ability to execute a command "on one host" and will work out which host to pick using load balancing. Finally, some parallel SSH projects let you use barriers so that you can execute a collection of commands and explicitly have each node in the group wait until all the nodes have completed a stage before moving on to the next stage of processing.

The configuration described here is not very complicated but still needs to be done carefully. You are expected to have at least basic knowledge of: * MySQL (creating a database, granting access to users, basic SQL queries) * SMTP (what it is and what a basic SMTP dialog looks like) * POP3, IMAP (what they do and what the differences are) * basic Postfix configuration (understand the default settings in your main.cf, have read through the basic configuration document and know that your mail log file is at /var/log/mail.log) * Debian/Linux (general system administration, using a text editor, reading log files)

asql is a simple console shell which allows you to run SQL queries against an Apache logfile, something that can be very useful - especially at getting information which is not easily available via static logfile analysers.

Awstats is a commonly used program to calculate web statistics from your webserver logfiles. It can detect useragents, referers, unique visitors… But one of its another feature is to build usage reports from your mail server’s logfiles, as seen on this demo. Here is how to quickly configure Awstats to take profit of your Qmail log files…

Scalp! est un analyseur de logs Apache qui sert dans la recherche de problèmes de sécurité et de tentatives d'intrusions, il est basé sur des requêtes du même style que celles du utilisées dans le projet PHP-IDS

IE WebDeveloper is an add-on for Microsoft Internet Explorer. The rich web debugging toolset allows you to inspect and edit the live HTML DOM and current cascaded styles, evaluate expressions and display error messages, log messages, explore source code of webpage and monitor DHTML Event and HTTP Traffic.

Splunk is IT Search. Search, navigate, alert and report on all your IT data in real time. Logs, configurations, messages, traps and alerts, script, code, metrics and more. If a machine can generate it -- Splunk can eat it.

wtop is really cool application for web server log analysis and to see server stats at a glance. It also has powerful log grepping capability. It is just like 'top' for your webserver. It can find out number of searches or signups per seconds. It can also create histogram of response time. There is also another tool called logrep a powerful command-line program for ad-hoc analysis and filtering for log files. You can dig up lots of information using wtop tools.

C’est en parcourant le dernier numéro hors série de Linux Magazine que je suis tombé sur un article sur un outil qui peut changer la vie des administrateurs systèmes. Cet outil s’appelle Multitail et est présenté par son concepteur (Folker Van Heusden) comme un “tail sur stéroides”. Comme moi, vous devez avoir en permancence plusieurs “tail -f” lancée en // pour surveiller les logs de vos serveurs. Multitail propose de réunir toutes ces informations dans une seule et même fenêtre.

Typical Scenario: You have a web server that serves your domain. You write a simple script to restart apache each night and pipe the logs off to your analyzer. It works. ISP/Hosting Scenario: Each server hosts many domains. You may also have load balanced servers (multiple machines) serving each domain. A tool like this is necessary to: 1. collect all the log files from each server 2. split the logs based on the virtual host(s) 3. discard invalid entries (phishing, referrer spam, etc) 4. sort them into chronological order 5. feed logs into analyzer 6. do something with the raw logs (compress, save to vhost dir, etc) [edit]

Logs in Apache are more configurable than most people realize. Not only can you organize the fields in your logs, but you can also create formats and layouts. Access logs can be split and divided up to make them easier to process by reporting specific items or ignoring those items that have no relevance.

To get awstats to work with qmail log files I use this little script from Bob Hutchinson called qlogs4awstats.

A log file analyzer for qmail, dnscache, tinydns, rbldns, axfrdns, publicfile and qpsmtpd.

qpsmtpd est un serveur SMTP très léger et très modulaire qui se base sur le principe des plugins et des hooks qu’il peut appeller à chaque instant du protocole SMTP : mail from, rcpt to, data, ... Le principe est de lancer qpsmtpd à l’écoute du port SMTP (25), charge à lui ensuite de faire les vérifications d’usage (beaucoup de plugins sont livrés dans le produit de base pour ce type d’opérations) pour ensuite relayer au serveur de messagerie qui écoutera sur un autre port (2525) par exemple. En gros qpsmtpd joue le rôle de firewall ou de frontal mail avant d’entrer dans l’infrastructure de la messagerie interne. L’avantage de cette technique est de pouvoir développer un plugin maison, avec des hooks à appeler selon ses besoins : on peut introduire des règles métiers sur les mails : vérification de l’existence du compte, vérification de l’expéditeur selon des règles qui nous sont propres, etc…et de couper la transaction SMTP avant même de recevoir le message : gain de temps et de ressources pour le serveur MTA en bout de chaîne.

Rsyslog is an enhanced multi-threaded syslogd supporting, among others, MySQL, syslog/tcp, RFC 3195, permitted sender lists, filtering on any message part, and fine grain output format control. It is quite compatible to stock sysklogd and can be used as a drop-in replacement. Its advanced features make it suitable for enterprise-class, encryption protected syslog relay chains while at the same time being very easy to setup for the novice user. An optional web interface - phpLogCon - can be used to visualize all data online."

Un bon admin doit lire ses logs régulièrement. En général parce qu'un système de surveillance lui a remonté un problème. Les logs sont en souvent très denses. Il existe un moyen pour les rendre plus lisibles. Il s'agit de coloriser le texte.

Raphael propose un truc pour transformer les fichiers de logs PHP en fichier RSS, à l'aide du Zend Framework. En les lisant avec un client RSS, vous pouvez même faire remonter plus facilement les erreurs qui apparaissent (j'imagine qu'un site qui produit beaucoup d'erreurs sera moins intéressé). Autrement, le format XML du fil permet un traitement externe, automatisé. Une idée détaillée avec le code pour réaliser le fil grâce au Zend Framework.

Un monde de log en couleur .... - eMerzh's Blog