Gluster as file system for state backend

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Gluster as file system for state backend

chiggi_dev
Hi,

I am evaluating some File Systems as state backend. I can see that Flink currently supports S3, MAPRFS and HDFS as file systems. 

However, I was hoping I can use Gluster as my state backend, since its already a part of existing eco system. Since I have stateful operators in my job and I am expecting each operator to accumulate large amount of state(in RocksDB) in 10seconds(checkpointing interval), I am looking for a write optimized  file system. 
 

Does anyone has any prior experience with Gluster as file state backend? Are there any caveats with using a replicated/striped replicated volume for Flink state data? 

Thanks,

Chirag