Skip to main content

Monitoring WSO2 products with logstash JMX input plugin

These days, I got the chance to play with ELK (Elasticsearch, Logstash & Kibana). These tools are a great way to analyze & visualize all logs.

You can easily analyze all wso2carbon.log files from ELK. However we also needed to use ELK for monitoring WSO2 products and this post explains the essential steps to use logstash JMX input plugin to monitor WSO2 servers.

Installing Logstash JMX input plugin


Logstash has many inputs and the JMX input plugin is available under "contrib"

We can use "plugin install contrib" command to install extra plugins.

cd /opt/logstash/bin
sudo ./plugin install contrib

Note: If you use logstash 1.4.0 and encounter issues in loading the jmx4r, please refer Troubleshooting below.

Logstash JMX input configuration


When using the JMX input plugin, we can use a similar configuration as follows. We are keeping the logstash configs in "/etc/logstash/conf.d/logstash.conf"


input {
jmx{
    path => "/etc/logstash/jmx"
    polling_frequency => 30
    type => "jmx"
    nb_thread => 4
  }
}

output {
  elasticsearch { host => localhost }
}

Note that the path points to a directory. We have the JMX configuration in "/etc/logstash/jmx/jmx.conf"

{
  //The WSO2 server hostname
  "host" : "localhost",
  //jmx listening port
  "port" : 9999,
  //username to connect to jmx
  "username" : "jmx_user",
  //password to connect to jmx
  "password": "jmx_user_pw",
  "alias" : "jmx.dssworker1.elasticsearch",
  //List of JMX metrics to retrieve
  "queries" : [
    {
      "object_name" : "java.lang:type=Memory",
      "attributes" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
      "object_alias" : "Memory"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=Code Cache",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolCodeCache"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Perm Gen",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolPermGen"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Old Gen",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolOldGen"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Eden Space",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolEdenSpace"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Survivor Space",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolSurvivorSpace"
    }, {
      "object_name" : "java.lang:type=GarbageCollector,name=*MarkSweep",
      "attributes" : [ "Name", "CollectionCount", "CollectionTime" ],
      "object_alias" : "GarbageCollectorMarkSweep"
    }, {
      "object_name" : "java.lang:type=GarbageCollector,name=ParNew",
      "attributes" : [ "Name", "CollectionCount", "CollectionTime" ],
      "object_alias" : "GarbageCollectorParNew"
    }, {
      "object_name" : "java.lang:type=ClassLoading",
      "attributes" : [ "LoadedClassCount", "TotalLoadedClassCount", "UnloadedClassCount" ],
      "object_alias" : "ClassLoading"
    }, {
      "object_name" : "java.lang:type=Runtime",
      "attributes" : [ "Uptime", "StartTime" ],
      "object_alias" : "Runtime"
    }, {
      "object_name" : "java.lang:type=Threading",
      "attributes" : [ "ThreadCount", "TotalStartedThreadCount", "DaemonThreadCount", "PeakThreadCount" ],
      "object_alias" : "Threading"
    }, {
      "object_name" : "java.lang:type=OperatingSystem",
      "attributes" : [ "OpenFileDescriptorCount", "FreePhysicalMemorySize", "CommittedVirtualMemorySize", "FreeSwapSpaceSize", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "TotalPhysicalMemorySize", "TotalSwapSpaceSize", "SystemLoadAverage" ],
      "object_alias" : "OperatingSystem"
    } ]
}


This is all we need to configure logstash to get JMX details from WSO2 servers. Note that we have given a directory as the path for JMX configuration. This means that all the configs inside "/etc/logstash/jmx" will be loaded. So, we need to make sure that there are no other files.

I'm querying only the required attributes for now. It is possible to add as many queries as you need.

Securing JMX access of WSO2 servers.


WSO2 servers by default start the JMX service and you should be able to see the JMX Service URL in wso2carbon.log

For example:
TID: [-1234] [] [DSS] [2014-05-31 01:09:11,103]  INFO {org.wso2.carbon.core.init.JMXServerManager} -  JMX Service URL  : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi

You can see JMX configuration in <CARBON_HOME>/repository/conf/etc/jmx.xml and the JMX ports in <CARBON_HOME&gt/repository/conf/carbon.xml

       
        <!-- The JMX Ports -->
        <JMX>
            <!--The port RMI registry is exposed-->
            <RMIRegistryPort>9999</RMIRegistryPort>
            <!--The port RMI server should be exposed-->
            <RMIServerPort>11111</RMIServerPort>
        </JMX>

You may change ports from this configuration.

It is recommended to create a role with only "Server Admin" permission and assign to the "jmx_user". Then the "jmx_user" will have the required privileges to monitor WSO2 servers.

Also if we enable Java Security Manager, we need to have following permissions. Usually the WSO2 servers are configured to use the security policy file at  <CARBON_HOME>/repository/conf/sec.policy if the Security Manager is enabled.


grant {
    // JMX monitoring requires following permissions. Check Logstash JMX input configurations
    permission javax.management.MBeanPermission "-#-[-]", "queryNames";
    permission javax.management.MBeanPermission "sun.management.MemoryImpl#*[java.lang:type=Memory]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.MemoryPoolImpl#*[java.lang:type=MemoryPool,name=*]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.GarbageCollectorImpl#*[java.lang:type=GarbageCollector,name=*]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.ClassLoadingImpl#*[java.lang:type=ClassLoading]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.RuntimeImpl#*[java.lang:type=Runtime]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.ThreadImpl#*[java.lang:type=Threading]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#*[java.lang:type=OperatingSystem]", "queryNames,getMBeanInfo,getAttribute";
}

That's it. You should be able to push JMX stats via logstash now.


Troubleshooting

First of all you can check whether the configurations are correct by running following command.

logstash --configtest

This must tell that the configuration is OK.

However I encountered following issue in logstash 1.4.0 when running the logstash command.

LoadError: no such file to load -- jmx4r
     require at org/jruby/RubyKernel.java:1085
     require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
     require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:53
     require at /opt/logstash/lib/logstash/JRUBY-6970.rb:27
     require at /opt/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.4/lib/polyglot.rb:65
  thread_jmx at /opt/logstash/bin/lib//logstash/inputs/jmx.rb:132
         run at /opt/logstash/bin/lib//logstash/inputs/jmx.rb:251


For this issue, we need to extract the plugins to the same directory of logstash installation instead of contrib plugin installation. I got help from #logstash IRC to figure this out. Thanks terroNZ!

I did following steps

wget  --no-check-certificate -O logstash-contrib-1.4.0.tar.gz http://download.elasticsearch.org/logstash/logstash/logstash-contrib-1.4.0.tar.gz
tar -xvf logstash-contrib-1.4.0.tar.gz
sudo rsync -rv --ignore-existing logstash-contrib-1.4.0/* /opt/logstash/

Please note that JMX input plugin works fine in logstash-1.4.1 after installing contrib plugin and above steps are not required.

Then the next issue can occur is when connecting to WSO2 server. Check <CARBON_HOME>/repository/logs/audit.log and see whether the user can connect successfully. If it is not successful, you should check user permissions.

Another issue can be the failure of JMX queries. You can run logstash with "--debug" option and see debug logs.

I noticed following.
{:timestamp=>"2014-05-30T00:09:29.373000+0000", :message=>"Find all objects name java.lang:type=Memory", :level=>:debug, :file=>"logstash/inputs/jmx.rb", :line=>"165"}
{:timestamp=>"2014-05-30T00:09:29.392000+0000", :message=>"No jmx object found for java.lang:type=Memory", :level=>:warn, :file=>"logstash/inputs/jmx.rb", :line=>"221"}
{:timestamp=>"2014-05-30T00:09:29.393000+0000", :message=>"Find all objects name java.lang:type=Runtime", :level=>:debug, :file=>"logstash/inputs/jmx.rb", :line=>"165"}
{:timestamp=>"2014-05-30T00:09:29.396000+0000", :message=>"No jmx object found for java.lang:type=Runtime", :level=>:warn, :file=>"logstash/inputs/jmx.rb", :line=>"221"}


This issue came as we have enabled the Java Security Manager and after adding permissions as mentioned above, the logstash JMX input plugin worked fine.

Next is to create dashboards in Kibana using these data. Hopefully I will be able to write a blog post on that as well.


Labels:
Location: Seattle, WA, USA

Comments

Post a Comment

Popular posts from this blog

Finding how many processors

I wanted to find out the processor details in my laptop and I found out that there are several ways to check. For example, see The RedHat community discussion on  Figuring out CPUs and Sockets . In this blog post, I'm listing few commands to find out details about CPUs. I'm using Ubuntu in my Lenovo ThinkPad T530 laptop and following commands should be working any Linux system. Display information about CPU architecture $ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 2 Core(s) per socket: 2 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 58 Model name: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz Stepping: 9 CPU MHz: 1199.988 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 5787.1...
Post a Comment
Read more

Java Mission Control & Java Flight Recorder

Last year, I got two opportunities to talk about Java Mission Control & Java Flight Recorder. I first talked about " Using Java Mission Control & Java Flight Recorder " as an internal tech talk at WSO2 . I must thank Srinath for giving me that opportunity. After that, Prabath also invited me to do a talk at Java Colombo Meetup . Prabath, Thank you for inviting me and giving me the opportunity to talk at the Java Colombo Meetup! I'm also very excited to see that Marcus Hirt , the Team Lead for Java Mission Control has mentioned about the Java Colombo Meetup in his blog post: " My Favourite JMC Quotes ". It's so nice to see "Sri Lanka" was mentioned in his blog post! :) From Marcus' Blog Here are the slides used at the meetup. Java Colombo Meetup: Java Mission Control & Java Flight Recorder from Isuru Perera Marcus Hirt's blog posts really helped me to understand JMC & JFR concepts and his tutorials were very helpful...
Post a Comment
Read more

Flame Graphs with Java Flight Recordings

Flame Graphs Brendon D. Gregg , who is a computer performance analyst , has created  Flame Graphs to visualize stack traces in an interactive way. You must watch his talk at USENIX/LISA13 , titled Blazing Performance with Flame Graphs , which explains Flame Graphs in detail. There can be different types of flame graphs and I'm focusing on  CPU Flame Graphs  with Java in this blog post. Please look at the Flame Graphs Description  to understand the Flame Graph visualization. CPU Flame Graphs and Java Stack Traces As  Brendon  mentioned in his talk, understanding why CPUs are busy is very important when analyzing performance.  CPU Flame Graphs  is a good way to identify hot methods from sampled stack traces. In order to generate CPU Flame Graphs for Java Stack Traces , we need a way to get sample stack traces. Brendon has given examples to use jstack  and Google's lightweight-java-profiler . Please refer to his perl program on g...
Post a Comment
Read more