AttributeError: 'KafkaLoggingHandler' object has no attribute 'producer'

[axil76]

Hello
I use the check_kafka script and I have the following error message, and I do not see where the problem is, below the logging.conf file

[loggers]
keys=root

[handlers]
keys=consoleHandler,fileHandler,kafkaHandler

[formatters]
keys=simpleFormatter,logstashFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler,fileHandler,kafkaHandler
formatter=simpleFormatter

[handler_fileHandler]
class=logging.handlers.WatchedFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/check_kafka.log', 'a')

[handler_kafkaHandler]
class=python_kafka_logging.KafkaHandler.KafkaLoggingHandler
level=DEBUG
formatter=logstashFormatter
args=("192.168.0.35:2181, 192.168.0.193:2181","apps-openshift")

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(levelname)s - %(name)s - %(message)s

[formatter_logstashFormatter]
class=logstash_formatter.LogstashFormatter
format={"extra": {"appName": "myPythonApp", "environment": "AWS-Test"}}

can you help me ?
thanks

[avihoo]

Hi,

This repo is not maintained anymore. You can check my fork of this repo for an updated version of the library and try your script against it.
If you have any problems open an issue on the forked library.

Thanks,
Avihoo

[axil76]

thank you for your reply
I just removed python-kafka-logging and reinstall your fork
and ask me kafka-python 1.3.1 while I have 1.4.2 which is mandatory to use the script check_kafka.py how can I do?

[avihoo]

You can submit a PR with the necessary fix to use the 1.4.2 version.

I might be able to do that in the following days.

[avihoo]

Although I just looked at the new version and it should work. Can you show me the relevant code to see what's wrong?

[axil76]

./check_kafka.py
/ Usr / lib / nagios / plugins / pylib
Traceback (most recent call last):
File "./check_kafka.py", line 58, in
from harisekhon.utils import log, log_option, ERRORS, CriticalError, UnknownError
File "/usr/lib/nagios/plugins/pylib/harisekhon/init.py", line 26, in
from harisekhon.cli import CLI
File "/usr/lib/nagios/plugins/pylib/harisekhon/cli.py", line 50, in
from import log, getenvs2, isBlankOrNone, isInt, isHost, isPort, isStr, isList, validate_int
File "/usr/lib/nagios/plugins/pylib/harisekhon/utils.py", line 96, in
logging.config.fileConfig (os.path.join (libdir, 'resources', 'logging.conf'))
File "/usr/lib/python2.7/logging/config.py", line 85, in fileConfig
handlers = _install_handlers (cp, formatters)
File "/usr/lib/python2.7/logging/config.py", line 160, in _install_handlers
klass = _resolve (klass)
File "/usr/lib/python2.7/logging/config.py", line 95, in _resolve
found = __import __ (used)
ImportError: No module named python_kafka_logging

the name of the module is python-kafka-logger and not python-kafka-logging

/usr/bin/pip2.7 list
blessings (1.6.1)
kafka-python (1.4.2)
logstash-formatter (0.5.8)
pip (9.0.3)
py-curses-editor (1.2.1)
python-kafka-logger (0.2)
PyYAML (3.12)
setuptools (39.0.1)
six (1.9.0)

I modified the section handler_kafkaHandler in logging.conf

[Handler_kafkaHandler]
class = python-Kafka-logger.KafkaHandler.KafkaLoggingHandler
level = DEBUG
formatter = logstashFormatter

but I have the same error message
/usr/bin/python2.7 check_kafka.py
/ Usr / lib / nagios / plugins / pylib
Traceback (most recent call last):
File "check_kafka.py", line 58, in
from harisekhon.utils import log, log_option, ERRORS, CriticalError, UnknownError
File "/usr/lib/nagios/plugins/pylib/harisekhon/init.py", line 26, in
from harisekhon.cli import CLI
File "/usr/lib/nagios/plugins/pylib/harisekhon/cli.py", line 50, in
from import log, getenvs2, isBlankOrNone, isInt, isHost, isPort, isStr, isList, validate_int
File "/usr/lib/nagios/plugins/pylib/harisekhon/utils.py", line 96, in
logging.config.fileConfig (os.path.join (libdir, 'resources', 'logging.conf'))
File "/usr/lib/python2.7/logging/config.py", line 85, in fileConfig
handlers = _install_handlers (cp, formatters)
File "/usr/lib/python2.7/logging/config.py", line 160, in _install_handlers
klass = _resolve (klass)
File "/usr/lib/python2.7/logging/config.py", line 95, in _resolve
found = __import __ (used)
ImportError: No module named python-kafka-logger

an idea ?
thank you

[avihoo]

OK, I just released a new version (0.4) of my library with the versions updated and the example file fixed.

The class you should use is:

class=python_kafka_logging.KafkaHandler.KafkaLoggingHandler

It worked on my Python console:

>>> from python_kafka_logging.KafkaHandler import KafkaLoggingHandler
>>> kafka_handler = KafkaLoggingHandler(['srv1', 'srv2'], 'topic')

[axil76]

Hello,
I reinstalled a platform and installed all python modules and I always have the same error message, do you have an idea?

./check_kafka.py
/usr/lib/nagios/plugins/pylib
Traceback (most recent call last):
File "./check_kafka.py", line 58, in
from harisekhon.utils import log, log_option, ERRORS, CriticalError, UnknownError
File "/usr/lib/nagios/plugins/pylib/harisekhon/init.py", line 26, in
from harisekhon.cli import CLI
File "/usr/lib/nagios/plugins/pylib/harisekhon/cli.py", line 50, in
from harisekhon.utils import log, getenvs2, isBlankOrNone, isInt, isHost, isPort, isStr, isList, validate_int
File "/usr/lib/nagios/plugins/pylib/harisekhon/utils.py", line 96, in
logging.config.fileConfig(os.path.join(libdir, 'resources', 'logging.conf'))
File "/usr/lib/python2.7/logging/config.py", line 85, in fileConfig
handlers = _install_handlers(cp, formatters)
File "/usr/lib/python2.7/logging/config.py", line 163, in _install_handlers
h = klass(*args)
File "/usr/lib/python2.7/site-packages/python_kafka_logger-0.4-py2.7.egg/python_kafka_logging/KafkaHandler.py", line 11, in init
self.kafka_client = SimpleClient(hosts_list)
File "/usr/lib/python2.7/site-packages/kafka_python-1.4.2-py2.7.egg/kafka/client.py", line 57, in init
self.load_metadata_for_topics() # bootstrap with all metadata
File "/usr/lib/python2.7/site-packages/kafka_python-1.4.2-py2.7.egg/kafka/client.py", line 525, in load_metadata_for_topics
resp = self.send_metadata_request(topics)
File "/usr/lib/python2.7/site-packages/kafka_python-1.4.2-py2.7.egg/kafka/client.py", line 590, in send_metadata_request
return self._send_broker_unaware_request(payloads, encoder, decoder)
File "/usr/lib/python2.7/site-packages/kafka_python-1.4.2-py2.7.egg/kafka/client.py", line 177, in _send_broker_unaware_request
raise KafkaUnavailableError('All servers failed to process request: %s' % hosts)
kafka.errors.KafkaUnavailableError: KafkaUnavailableError: All servers failed to process request: [('192.168.0.10', 2181, 2), ('192.168.0.11', 2181, 2)]
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/logging/init.py", line 1669, in shutdown
h.close()
File "/usr/lib/python2.7/site-packages/python_kafka_logger-0.4-py2.7.egg/python_kafka_logging/KafkaHandler.py", line 42, in close
if self.producer is not None:
AttributeError: 'KafkaLoggingHandler' object has no attribute 'producer'
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/logging/init.py", line 1669, in shutdown
h.close()
File "/usr/lib/python2.7/site-packages/python_kafka_logger-0.4-py2.7.egg/python_kafka_logging/KafkaHandler.py", line 42, in close
if self.producer is not None:
AttributeError: 'KafkaLoggingHandler' object has no attribute 'producer'

thanks for you help

[avihoo]

Can you please share your script with me? Or at least the part where you initiate the KafkaLoggingHandler object?

[axil76]

config file [loggers]
keys=root

[handlers]
keys=consoleHandler,fileHandler,kafkaHandler

[formatters]
keys=simpleFormatter,logstashFormatter

[logger_root]
level=DEBUG
handlers=consoleHandler,fileHandler,kafkaHandler
formatter=simpleFormatter

[handler_fileHandler]
class=logging.handlers.WatchedFileHandler
level=DEBUG
formatter=simpleFormatter
args=('/var/log/check_kafka.log', 'a')

[handler_kafkaHandler]
class=python_kafka_logging.KafkaHandler.KafkaLoggingHandler
level=DEBUG
formatter=logstashFormatter
args=("192.168.0.10:9092, 192.168.0.11:9092","wom-apps-openshift")

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(levelname)s - %(name)s - %(message)s

[formatter_logstashFormatter]
class=logstash_formatter.LogstashFormatter
format={"extra": {"appName": "myPythonApp", "environment": "AWS-Test"}}

the script can be found here
https://github.com/HariSekhon/nagios-plugins/blob/master/check_kafka.py

[axil76]

little extra information the kafka is configured in ssl mode
it must be specified or in the logging.conf file

[avihoo]

OK after a closer look at the exception I found the problem:

raise KafkaUnavailableError('All servers failed to process request: %s' % hosts) kafka.errors.KafkaUnavailableError: KafkaUnavailableError: All servers failed to process request: [('192.168.0.10', 2181, 2), ('192.168.0.11', 2181, 2)]

This means Kafka can't be reached via these hosts, which makes sense because you're using the port which usually reserved for the zookeeper and not Kafka itself.

What you probably need to do is to change the port to the one Kafka is using, which by default is 9092.