Connector pubsub/events support and other issues

[chrisk314]

Currently support for pubsub connectors, and by extension event based models, is not available for all combinations of Process class and Connector implementation due to various issues / incompatibilies. This is a tracking issue for multiple issues under this umbrella topic.

• LocalProcess with ZMQConnector when PLUGBOARD_FLAGS_ZMQ_PUBSUB_PROXY=false causes failure in test_process_stop_event.test_process_stop_event. Failure due to serialisation issue occuring in ExportMixin.dict which raises TypeError: cannot pickle 'async_generator' object. Severity medium as PLUGBOARD_FLAGS_ZMQ_PUBSUB_PROXY=true should be preferred; however, PLUGBOARD_FLAGS_ZMQ_PUBSUB_PROXY=false is the current default.

• LocalProcess with ZMQConnector with PLUGBOARD_FLAGS_ZMQ_PUBSUB_PROXY set to either true or false causes failure in test_component_event_handlers.test_component_event_handlers_with_field_inputs. Failure manifests as hanging test. Exact cause unknown at time of writing. Requires investigation. Severity low as test_process_stop_event.test_process_stop_event currently working for preferred PLUGBOARD_FLAGS_ZMQ_PUBSUB_PROXY=true case suggesting a bug in the test code rather than lib code.

• RayProcess with ZMQConnector with PLUGBOARD_FLAGS_ZMQ_PUBSUB_PROXY=true causes failure in test_process_stop_event.test_process_stop_event. Test assertion fails. Maximum iterations is reached before StopEvent is received and causes model to stop as expected. Exact cause unknown at time of writing. Requires investigation. Severity low as ZMQConnector not expected to work with networking config in multi-host Ray context, with alternatives (e.g. RabbitMQConnector) planned to address this use-case.

• RayProcess with ZMQConnector with PLUGBOARD_FLAGS_ZMQ_PUBSUB_PROXY=false causes failure in test_process_stop_event.test_process_stop_event. Serialisation error occurs when setting up Ray worker due to ZMQConnector. Most likely cause is the zmq poll async generator created in the _ZMQPubsubConnector constructor. Severity low as ZMQConnector not expected to work with networking config in multi-host Ray context, with alternatives (e.g. RabbitMQConnector) planned to address this use-case.

• RayProcess with RayConnector does not currently have an implementation for ConnectorMode.PUBSUB.

• LocalProces or RayProcess when using ZMQConnector in "pipeline" mode will hang on calling connect_recv unless connect_send has already been called, or is called concurrently. This is due to a blocking read of a ray.Queue on the receiver side to share the port for comms socket. This behaviour is in contrast to the AsyncioConnector where it is possible to call connect_send and connect_recv without risk of blocking, although a model could hang at runtime if some connectors have not been connected.

[toby-coleman]

Revert #109 when this issue is completed.