Configuring log format for console output (catalina.out)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Configuring log format for console output (catalina.out)

Joan
Hi all,

I am trying to forward the logs generated on a server to a centralized log
collector (using rsyslog), one of the features I would like to use the
ability to group messages by using their timestamp as the start of the
message.
By reading the documentation at
https://tomcat.apache.org/tomcat-9.0-doc/logging.html#Console, I can't see
the proper way to  get a common format (preferably with timestamp), or ar
tleast save the outputs into three different types in different files?

Some of the logs are displayed like this (severity in brackets but no
timestamp)

>
> [WARN] Web application uses
> org.apache.axis2.transport.http.AxisAdminServlet; please update web.xml to
> use org.apache.axis2.webapp.AxisAdminServlet instead
> [INFO] Clustering has been disabled
> [INFO] Deploying module: script-1.6.2 -
> file:/var/lib/tomcat8/webapps/WS_Revisiones/WEB-INF/modules/scripting-1.6.2.mar
>
Stack traces are displayed like this (no severity and no timestamp)

> MultiException stack 1 of 2
> java.lang.NoClassDefFoundError: javax/activation/DataSource
>         at java.base/java.lang.Class.getDeclaredConstructors0(Native
> Method)
>         at
> java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
>         at
> java.base/java.lang.Class.getDeclaredConstructors(Class.java:2357)
>         at org.jvnet.hk2.internal.Utilities$3.run(Utilities.java:1306)
>         at org.jvnet.hk2.internal.Utilities$3.run(Utilities.java:1302)
>         at java.base/java.security.AccessController.doPrivileged(Native
> Method)
>
Also some other are like this (timestamp with ms + severity + context in
brackets

> 21-May-2019 13:03:22.474 INFO [localhost-startStop-1]
> org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned
> for TLDs yet contained no TLDs. Enable debug logging for this logger for a
> complete list of JARs that were scanned but no TLDs were found in them.
> Skipping unneeded JARs during scanning can improve startup time and JSP
> compilation time.
>
Reply | Threaded
Open this post in threaded view
|

Re: Configuring log format for console output (catalina.out)

Christopher Schultz-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Joan,

On 5/21/19 09:11, Joan wrote:

> Hi all,
>
> I am trying to forward the logs generated on a server to a
> centralized log collector (using rsyslog), one of the features I
> would like to use the ability to group messages by using their
> timestamp as the start of the message. By reading the documentation
> at https://tomcat.apache.org/tomcat-9.0-doc/logging.html#Console, I
> can't see the proper way to  get a common format (preferably with
> timestamp), or ar tleast save the outputs into three different
> types in different files?

I think you are being confused by the fact that the default log
pattern contains a newline character. For example, these are the first
two entries in my local catalina.out file:

Apr 12, 2019 1:55:03 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-nio-8215"]
Apr 12, 2019 1:55:03 PM org.apache.tomcat.util.net.NioSelectorPool
getSharedSelector
INFO: Using a shared selector for servlet write/read
Apr 12, 2019 1:55:03 PM org.apache.coyote.AbstractProtocol init

(Apologies for the word-wrapping)

The first entry is [Apr 12, 2019 1:55:03 PM] and the text ends with
["ajp-nio-8215"]

The second entry has the same timestmp and the text ends with
"AbstractProtocol init"

Stack traces are always impossible to handle when looking at the
resulting file, since they almost always span multiple lines of text
- -- that is, they have embedded newlines in the message.

Are you asking about how to make sense of the log file, how to send
log messages to a central server, or how to coalesce log messages by
timestamp?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzkEO8ACgkQHPApP6U8
pFg8EA/+KsH4JGirKakL+/OxqAz1W7T7MoR2QE9Y1Tk26FL6XI+VITk11gqLegH5
yx3zwmXLS9v2XilppnSvRB9xPd073Uf4v0NPSjCt1vSY7obPHYcRSIuA43QU9Pqb
7pEgxuGn+uUifRRK+DLd6Wcd3IkFo2hfKG/YHFL5McKxqsrEvPsspWU7uLRHoKML
BR1MM5iKbsiBEJfncfyZqeGY2Yq4HOarcp//iv5bC20RgakLxlSjiWAyJqC77PKH
3dy9XaV/VyQpjEQe8/ADt1NqhF6qBVj/KcKeA41AAwL7grxX0R23ZeSQsKJeulgf
jnXcK8mHWXs4Dybu7hhUbkS47whbfW6M3H4ZhGYutsoOB58shzbVR0MUtEYlOBMg
GbpCt5xPW6NZ5LMbEDTMlQW95XBnrLdxe8dFdTo+qgEdSoe3o4mdvGUtrlgAxZlq
Ay28U+/maI84StrScDwfC96chakdaGj/4CxyFwoj+5uVeeiBbglubR1gCghqVjM7
MQWDKrTACmcEu4ZbLy1sgNtBb/xjCYRtp05JrB531E9Py7//f6qprIKjAYOBLcxF
uWDKP204C3UlC3/mQt6Ewd6QzQM0xCXPm0k+6817wf3xIaN/aST4R0kbR0OjkB6F
CfAm8Qf2FVgRHOP4zzKHvm82wjET2FJmcw38do2ki89yml1iKXg=
=UolI
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Configuring log format for console output (catalina.out)

Joan
Thanks a lot for your answer Chris

> > I am trying to forward the logs generated on a server to a
> > centralized log collector (using rsyslog), one of the features I
> > would like to use the ability to group messages by using their
> > timestamp as the start of the message. By reading the documentation
> > at https://tomcat.apache.org/tomcat-9.0-doc/logging.html#Console, I
> > can't see the proper way to  get a common format (preferably with
> > timestamp), or ar tleast save the outputs into three different
> > types in different files?
>
> I think you are being confused by the fact that the default log
> pattern contains a newline character. For example, these are the first
> two entries in my local catalina.out file:
>
> Apr 12, 2019 1:55:03 PM org.apache.coyote.AbstractProtocol init
> INFO: Initializing ProtocolHandler ["ajp-nio-8215"]
> Apr 12, 2019 1:55:03 PM org.apache.tomcat.util.net.NioSelectorPool
> getSharedSelector
> INFO: Using a shared selector for servlet write/read
> Apr 12, 2019 1:55:03 PM org.apache.coyote.AbstractProtocol init
>
> (Apologies for the word-wrapping)
>
> The first entry is [Apr 12, 2019 1:55:03 PM] and the text ends with
> ["ajp-nio-8215"]
>
> The second entry has the same timestmp and the text ends with
> "AbstractProtocol init"
>
> Stack traces are always impossible to handle when looking at the
> resulting file, since they almost always span multiple lines of text
> - -- that is, they have embedded newlines in the message.
>
> Are you asking about how to make sense of the log file, how to send
> log messages to a central server, or how to coalesce log messages by
> timestamp?
>

Rsyslog allows to read multiline messages by matching a regular expression
(see
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html#startmsg-regex),
this allow the messages to be forwarded to a third party (graylog,
logstash, ..) as a a single message.
So my intention would be to be able to make all the messages to start with
a timestamp (or a defined format)

After your message I undertand that this is a single message (I didn't
realize it) and all the messages in catalina.out are obeying the same
format. Being the next lines a single message, am I right?

19-May-2019 06:25:20.937 INFO [localhost-startStop-1]

> org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned
> for TLDs yet contained no TLDs. Enable debug logging for this logger for a
> comple
> te list of JARs that were scanned but no TLDs were found in them. Skipping
> unneeded JARs during scanning can improve startup time and JSP compilation
> time.
> [ERROR] Unable to find config file.  Creating new servlet engine config
> file: /WEB-INF/server-config.wsdd
> [WARN] Web application uses
> org.apache.axis2.transport.http.AxisAdminServlet; please update web.xml to
> use org.apache.axis2.webapp.AxisAdminServlet instead
> [WARN] Unable to load manifest for StAX implementation at jrt:/java.xml/
> [INFO] Clustering has been disabled
>


Thanks,

Joan

>
>
Reply | Threaded
Open this post in threaded view
|

RE: Configuring log format for console output (catalina.out)

Jäkel, Guido
Dear Joan,

by use of the common scripts, the file  catalina.out  will contain the console output (stdout/tderr) of the *JVM* process and -- if not configured in another way -- of the applications. Therefore, you have to deal with the features of output formatting of current JVMs. Or -- as me -- you might pipe this file descriptors through a tiny script that will prepend a timestamp in a format of your choice. This will have the advantage that it also handle "console" output of Java application; despite of the fact that using stdout/stderr instead of a java logging mechanism is a very bad style.

If you don't need to process thousands of lines per second, a simple shell script may do the job:

        while read line; do echo "`date -Ins` $line"; done

To avoid "double-stamping", you may add an heuristic check of the incoming line; please adjust the RegExpr to your neeeds

        while read line; do [[ ! "$line" =~ ^\d\d\d\d ]] && echo -n "[`date -Ins`] "; echo $line; done


Guido

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Configuring log format for console output (catalina.out)

Joan
Thanks Guido,

I hope that with the rsyslog magic (see
startmsg.regex="^[[:digit:]]{2}-[[:alpha:]]{3}-[[:digit:]]{4}
[[:digit:]]{1,2}:[[:digit:]]{1,2}:[[:digit:]]{1,2}\\.[[:digit:]]{3}"  for
the reference), I will be able to group this messages, otherwise I'll jump
into your solution.

Joan

Missatge de Jäkel, Guido <[hidden email]> del dia dc., 22 de maig 2019 a
les 8:55:

> Dear Joan,
>
> by use of the common scripts, the file  catalina.out  will contain the
> console output (stdout/tderr) of the *JVM* process and -- if not configured
> in another way -- of the applications. Therefore, you have to deal with the
> features of output formatting of current JVMs. Or -- as me -- you might
> pipe this file descriptors through a tiny script that will prepend a
> timestamp in a format of your choice. This will have the advantage that it
> also handle "console" output of Java application; despite of the fact that
> using stdout/stderr instead of a java logging mechanism is a very bad style.
>
> If you don't need to process thousands of lines per second, a simple shell
> script may do the job:
>
>         while read line; do echo "`date -Ins` $line"; done
>
> To avoid "double-stamping", you may add an heuristic check of the incoming
> line; please adjust the RegExpr to your neeeds
>
>         while read line; do [[ ! "$line" =~ ^\d\d\d\d ]] && echo -n
> "[`date -Ins`] "; echo $line; done
>
>
> Guido
>