program args size for running jobs

[Esteban Serrano]

Hi Flink users

Looking for some ideas on alternatives/workaround for running a job which requires a large number of parameters (one of them being a long JSON string). When all params are taken together, the resulting REST API URL puts the size of the URL over the current 4096 bytes limit. Ideally, the POST submit could allow named parameters in the body in addition to query params, but was wondering if people have run into this limitation and have found ways to work around it for submitting jobs. 

The following JIRA [1] documents the current shortcoming for other operations in the Web UI with the same sort of issue/limitation.

Best,

E

[1] - https://issues.apache.org/jira/browse/FLINK-8949

[Fabian Hueske-2]

Hi Esteban,

If you need the parameters to configure specific operators (instead of the over all flow), you could pass the parameters as a file using the distributed cache [1].

Note, the docs point to the DataSet (batch) API, but the feature works the same way for DataStream programs as well.

Best,
Fabian

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/batch/#distributed-cache

2018-05-22 23:57 GMT+02:00 Esteban Serrano <[hidden email]>:

Hi Flink users

Looking for some ideas on alternatives/workaround for running a job which requires a large number of parameters (one of them being a long JSON string). When all params are taken together, the resulting REST API URL puts the size of the URL over the current 4096 bytes limit. Ideally, the POST submit could allow named parameters in the body in addition to query params, but was wondering if people have run into this limitation and have found ways to work around it for submitting jobs. 

The following JIRA [1] documents the current shortcoming for other operations in the Web UI with the same sort of issue/limitation.

Best,

E

[1] - https://issues.apache.org/jira/browse/FLINK-8949