Grok TransformItems task requires Ion based file

[shrutimantri]

Expected Behavior

The log file generally is as follows:

2020-12-14T12:18:51.397Z INFO Hi from Kestra team!
2020-12-14T12:18:51.397Z INFO Your output is: 12345
2020-12-14T12:18:51.397Z WARN Warning message!!
2020-12-14T12:18:51.397Z ERROR Somethiong went wrong..
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
  File "/tmp/teste.py", line 9, in <module>
    run_my_stuff()
NameError: name 'run_my_stuff' is not defined
2020-12-14T12:18:51.397Z INFO Hi from Kestra team!

The expected behavior will be for the Grok TransformItems to be able to transform the file.

Actual Behaviour

Grok TransformItems is expecting an Ion file.

Using this type of log file (which is how log file generally is), results in an error:

2020-12-14T12:18:51.397Z INFO Hi from Kestra team!
2020-12-14T12:18:51.397Z INFO Your output is: 12345
2020-12-14T12:18:51.397Z WARN Warning message!!
2020-12-14T12:18:51.397Z ERROR Somethiong went wrong..
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
  File "/tmp/teste.py", line 9, in <module>
    run_my_stuff()
NameError: name 'run_my_stuff' is not defined
2020-12-14T12:18:51.397Z INFO Hi from Kestra team!

Error:

2024-08-16 19:06:48.705com.fasterxml.jackson.core.JsonParseException: Corrupt content to decode; underlying IonReader problem: (com.amazon.ion.impl.IonReaderTextRawX$IonReaderTextParsingException) Syntax error at line 1 offset 50: invalid syntax [state:STATE_BEFORE_ANNOTATION_DATAGRAM on token:TOKEN_SYMBOL_OPERATOR]
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
2024-08-16 19:06:48.705Corrupt content to decode; underlying IonReader problem: (com.amazon.ion.impl.IonReaderTextRawX$IonReaderTextParsingException) Syntax error at line 1 offset 50: invalid syntax [state:STATE_BEFORE_ANNOTATION_DATAGRAM on token:TOKEN_SYMBOL_OPERATOR]
 at [Source: UNKNOWN; byte offset: #UNKNOWN]
 Syntax error at line 1 offset 50: invalid syntax [state:STATE_BEFORE_ANNOTATION_DATAGRAM on token:TOKEN_SYMBOL_OPERATOR]

The task expects an Ion file like:

"2020-12-14T12:18:51.397Z INFO Hi from Kestra team!"
"2020-12-14T12:18:51.397Z INFO Your output is: 12345"
"2020-12-14T12:18:51.397Z WARN Warning message!!"
"2020-12-14T12:18:51.397Z ERROR Somethiong went wrong.."
"DEBUG:root:This message should go to the log file"
"ERROR:root:Got exception on main handler"Traceback
"Traceback (most recent call last):"
"  File \"/tmp/teste.py\", line 9, in <module>"
"    run_my_stuff()"
"NameError: name 'run_my_stuff' is not defined"
"2020-12-14T12:18:51.397Z INFO Hi from Kestra team!"

We do not have any task that will transform the given file in the Ion format. So, I am not sure how we can leverage this task.

Steps To Reproduce

1. Use the flow given in the example.
2. The input file can be log_file.txt with the following contents:

2020-12-14T12:18:51.397Z INFO Hi from Kestra team!
2020-12-14T12:18:51.397Z INFO Your output is: 12345
2020-12-14T12:18:51.397Z WARN Warning message!!
2020-12-14T12:18:51.397Z ERROR Somethiong went wrong..
DEBUG:root:This message should go to the log file
ERROR:root:Got exception on main handler
Traceback (most recent call last):
  File "/tmp/teste.py", line 9, in <module>
    run_my_stuff()
NameError: name 'run_my_stuff' is not defined
2020-12-14T12:18:51.397Z INFO Hi from Kestra team!

Environment Information

• Kestra Version: 0.18.1
• Plugin version: 0.18.1
• Operating System (OS / Docker / Kubernetes): Docker
• Java Version (If not docker): N/A

Example flow

id: grok-flow
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: grok
    type: io.kestra.plugin.transform.grok.TransformItems
    pattern: "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
    from: "{{ inputs.file }}"