Grok and Flink

[Aarti Gupta]

Hi,

We are using the Grok filter in Logstash to parse and enrich our data. Grok provides inbuilt parsing for common log sources such as Apache, this allows us to add structure to unstructured data.

After the data has been parsed in Logstash, we then stream the data over Kafka to Flink for further CEP processing. 

We are looking to see if we can get rid of the Logstash piece and do all of the data enrichment and parsing in Flink.  

Our question - does Flink have an inbuilt library similar to Grok that provides out of the box parsing for common log formats. 

Thanks in advance,

Aarti

--

[Lehuede sebastien]

Hi,

To parse my logs and reuse all my Grok pattern, i use the Java Grok API directly in my DataStream. Please see : https://github.com/thekrakken/java-grok

With that you should be able to get rid of the full Logstash piece and use only the Grok part.

Another solution, for example if you have logs/events in CEF Format, you can just use 'split' in the flatmap function for example.

Hope will help.

Regards,

Sebastien.

[Aarti Gupta]

Interesting, thanks Lehuede. Will take a look. 

--Aarti

On Thu, Aug 30, 2018 at 5:59 PM, Lehuede sebastien <[hidden email]> wrote:

Hi,

To parse my logs and reuse all my Grok pattern, i use the Java Grok API directly in my DataStream. Please see : https://github.com/thekrakken/java-grok

With that you should be able to get rid of the full Logstash piece and use only the Grok part.

Another solution, for example if you have logs/events in CEF Format, you can just use 'split' in the flatmap function for example.

Hope will help.

Regards,

Sebastien.

--