Parsing XML freezes logstash

[proteansec]

I'm using the following /etc/logstash/conf.d/10-use.conf configuration file to run Logstash. Basically I'm trying to parse the data out of the XML: you can see the XML later in this issue, but I want to point out that I need to parse the 'test' string out of the XML. This is why I'm using the "//*:getData" XPATH expression, which works in desktop XML/XPATH editors.

input {
  stdin { }
}
filter {
  xml {
    source => "message"
    xpath => [ "//*:getData", "data" ]
  }
}
output {
  stdout { codec => "json" }
  file { 
    codec => "json"
    path => "/tmp/debug-filters.json"
  }
}

Then we run logstash as:

# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/10-use.conf -l /var/log/logstash/logstash-debug.log  --debug

Sending logstash logs to /var/log/logstash/logstash-debug.log.
Reading config file {:file=>"logstash/agent.rb", :level=>:debug, :line=>"301", :method=>"local_config"}
Compiled pipeline code:
        @inputs = []
        @filters = []
        @outputs = []
        @periodic_flushers = []
        @shutdown_flushers = []

          @input_stdin_1 = plugin("input", "stdin")

          @inputs << @input_stdin_1

          @filter_xml_2 = plugin("filter", "xml", LogStash::Util.hash_merge_many({ "source" => ("message") }, { "xpath" => [("//*:getData"), ("data")] }))

          @filters << @filter_xml_2

            @filter_xml_2_flush = lambda do |options, &block|
              @logger.debug? && @logger.debug("Flushing", :plugin => @filter_xml_2)

              events = @filter_xml_2.flush(options)

              return if events.nil? || events.empty?

              @logger.debug? && @logger.debug("Flushing", :plugin => @filter_xml_2, :events => events)

              events.each{|e| block.call(e)}
            end

            if @filter_xml_2.respond_to?(:flush)
              @periodic_flushers << @filter_xml_2_flush if @filter_xml_2.periodic_flush
              @shutdown_flushers << @filter_xml_2_flush
            end

          @output_stdout_3 = plugin("output", "stdout", LogStash::Util.hash_merge_many({ "codec" => ("json") }))

          @outputs << @output_stdout_3

          @output_file_4 = plugin("output", "file", LogStash::Util.hash_merge_many({ "codec" => ("json") }, { "path" => ("/tmp/debug-filters.json") }))

          @outputs << @output_file_4
  def filter_func(event)
    events = [event]
    @logger.debug? && @logger.debug("filter received", :event => event.to_hash)
              events = @filter_xml_2.multi_filter(events)

    events
  end
  def output_func(event)
    @logger.debug? && @logger.debug("output received", :event => event.to_hash)
    @output_stdout_3.handle(event)
    @output_file_4.handle(event)

  end {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"28", :method=>"initialize"}

Using version 0.1.x filter plugin 'xml'. This plugin isn't well supported by the community and likely has no maintainer. {:level=>:info, :file=>"logstash/config/mixin.rb", :line=>"223", :method=>"print_version_notice"}
config LogStash::Filters::Xml/@source = "message" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@xpath = {"//*:getData"=>"data"} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@type = "" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@exclude_tags = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@add_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@remove_tag = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@remove_field = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@periodic_flush = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@store_xml = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
config LogStash::Filters::Xml/@remove_namespaces = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"112", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"133", :method=>"lookup"}

We copy a sample XML into the stdin:

<?xml version="1.0" encoding="utf-8"?><soap:Envelope><soap:Body><getResponse xmlns="http://www.google.com/"><getData>test</getData></getResponse></soap:Body></soap:Envelope>

The problem is that logstash just freezes, doesn't output anything in neither of the logs, doesn't accept new input strings, but just isn't working anymore. The only way to actually stop it is to .

Does anybody have any clues why that happens and what we're doing wrong? We're guessing this has something to do with the XML input filter, which is defined in the following file: /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-0.1.5/lib/logstash/filters/xml.rb.

[wiibaa]

Stability and error handling of xpath filter will improve in logstash-plugins/logstash-filter-xml#35

@suyograo can you close please