Run programs w/ params including comma via REST api

[Dongwon Kim-2]

Hi,

I'm trying to run a program by sending POST requests.

I've already spawned dispatcher in YARN and uploaded a jar file to the dispatcher.

I want to execute my application using the following arguments (--topic and --kafkaParams):

--topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245,group.id=trajectory-tracker

As you can see, there's a comma in the value of kafkaParams.

When I'm sending my application using the following command from Bash (%20 is a space and $2C is a comma), 

bash> curl -X POST 'http://dacoe4.weave.local:45097/jars/7b6880d7-b899-4243-8b9b-a01ad7f8a854_Tmap-1.0-SNAPSHOT.jar/run?entry-class=com.skt.tmap.trajectorytracker.TrajectoryTracker&program-args=--topic%20gps-topic%20--kafkaParams%20bootstrap.servers=dacoe2:20245%2Cgroup.id=trajectory-tracker'

I get the following response from the dispatcher:

{
    "errors": [
        "Expected only one value [--topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245, group.id=trajectory-tracker]."

    ]
}

What I found from the source code is that org.apache.flink.runtime.rest.messages.MessageQueue tries split the value of program-ages using comma as a delimiter.

I think I could modify my program to get arguments in a different way but it is not going to be intuitive to use different characters for a delimiter instead of comma.

How you guys think?

Or there's a way to avoid this behavior of splitting the value of program-args into multiple pieces?

best,

- Dongwon

[Chesnay Schepler]

Currently I don't see a way to circumvent the splitting. You will have to use a different delimiter, I guess a semi-colon could work?

The code is rather optimistic in that it assumes commas to not occur within a parameter value, and doesn't support any kind of escaping or quoting. (And this is a rabbit hole I'd rather avoid)

Ultimately I would love to change this call to send the parameters as JSON instead (then you wouldn't have to deal with escaping characters...), but we can't do that until the API versioning is in place (no ETA).

On 20.04.2018 12:37, Dongwon Kim wrote:

Hi,

I'm trying to run a program by sending POST requests.

I've already spawned dispatcher in YARN and uploaded a jar file to the dispatcher.

I want to execute my application using the following arguments (--topic and --kafkaParams):

--topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245,group.id=trajectory-tracker

As you can see, there's a comma in the value of kafkaParams.

When I'm sending my application using the following command from Bash (%20 is a space and $2C is a comma), 

bash> curl -X POST 'http://dacoe4.weave.local:45097/jars/7b6880d7-b899-4243-8b9b-a01ad7f8a854_Tmap-1.0-SNAPSHOT.jar/run?entry-class=com.skt.tmap.trajectorytracker.TrajectoryTracker&program-args=--topic%20gps-topic%20--kafkaParams%20bootstrap.servers=dacoe2:20245%2Cgroup.id=trajectory-tracker'

I get the following response from the dispatcher:

{
    "errors": [
        "Expected only one value [--topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245, group.id=trajectory-tracker]."

    ]
}

What I found from the source code is that org.apache.flink.runtime.rest.messages.MessageQueue tries split the value of program-ages using comma as a delimiter.

I think I could modify my program to get arguments in a different way but it is not going to be intuitive to use different characters for a delimiter instead of comma.

How you guys think?

Or there's a way to avoid this behavior of splitting the value of program-args into multiple pieces?

best,

- Dongwon

[Dongwon Kim-2]

Hi Chesnay,

I already modified our application to use semicolon as delimiter and now I can run a job using rest API via dispatcher.

Nevertheless, though I understand the difficulty of modifying API, it needs to be adjusted as many users are highly likely to use a list of brokers as an argument of their streaming applications and use comma to parse the list.

Otherwise, it must be written on the document to warn users to use other characters as delimiter for program arguments.

Best,

- Dongwon

2018. 4. 25. 오후 5:28, Chesnay Schepler <[hidden email]> 작성:

Currently I don't see a way to circumvent the splitting. You will have to use a different delimiter, I guess a semi-colon could work?

The code is rather optimistic in that it assumes commas to not occur within a parameter value, and doesn't support any kind of escaping or quoting. (And this is a rabbit hole I'd rather avoid)

Ultimately I would love to change this call to send the parameters as JSON instead (then you wouldn't have to deal with escaping characters...), but we can't do that until the API versioning is in place (no ETA).

On 20.04.2018 12:37, Dongwon Kim wrote:

Hi,

I'm trying to run a program by sending POST requests.

I've already spawned dispatcher in YARN and uploaded a jar file to the dispatcher.

I want to execute my application using the following arguments (--topic and --kafkaParams):

--topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245,group.id=trajectory-tracker

As you can see, there's a comma in the value of kafkaParams.

When I'm sending my application using the following command from Bash (%20 is a space and $2C is a comma), 

bash> curl -X POST 'http://dacoe4.weave.local:45097/jars/7b6880d7-b899-4243-8b9b-a01ad7f8a854_Tmap-1.0-SNAPSHOT.jar/run?entry-class=com.skt.tmap.trajectorytracker.TrajectoryTracker&program-args=--topic%20gps-topic%20--kafkaParams%20bootstrap.servers=dacoe2:20245%2Cgroup.id=trajectory-tracker'

I get the following response from the dispatcher:

{
    "errors": [
        "Expected only one value [--topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245, group.id=trajectory-tracker]."

    ]
}

What I found from the source code is that org.apache.flink.runtime.rest.messages.MessageQueue tries split the value of program-ages using comma as a delimiter.

I think I could modify my program to get arguments in a different way but it is not going to be intuitive to use different characters for a delimiter instead of comma.

How you guys think?

Or there's a way to avoid this behavior of splitting the value of program-args into multiple pieces?

best,

- Dongwon

[Chesnay Schepler]

We may be able to modify the handler to just concatenate the arguments again with a comma. I'll look into that tomorrow.

On 25.04.2018 11:02, Dongwon Kim wrote:

Hi Chesnay,

I already modified our application to use semicolon as delimiter and now I can run a job using rest API via dispatcher.

Nevertheless, though I understand the difficulty of modifying API, it needs to be adjusted as many users are highly likely to use a list of brokers as an argument of their streaming applications and use comma to parse the list.

Otherwise, it must be written on the document to warn users to use other characters as delimiter for program arguments.

Best,

- Dongwon

2018. 4. 25. 오후 5:28, Chesnay Schepler <[hidden email]> 작성:

Currently I don't see a way to circumvent the splitting. You will have to use a different delimiter, I guess a semi-colon could work?

The code is rather optimistic in that it assumes commas to not occur within a parameter value, and doesn't support any kind of escaping or quoting. (And this is a rabbit hole I'd rather avoid)

Ultimately I would love to change this call to send the parameters as JSON instead (then you wouldn't have to deal with escaping characters...), but we can't do that until the API versioning is in place (no ETA).

On 20.04.2018 12:37, Dongwon Kim wrote:

Hi,

I'm trying to run a program by sending POST requests.

I've already spawned dispatcher in YARN and uploaded a jar file to the dispatcher.

I want to execute my application using the following arguments (--topic and --kafkaParams):

--topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245,group.id=trajectory-tracker

As you can see, there's a comma in the value of kafkaParams.

When I'm sending my application using the following command from Bash (%20 is a space and $2C is a comma), 

bash> curl -X POST 'http://dacoe4.weave.local:45097/jars/7b6880d7-b899-4243-8b9b-a01ad7f8a854_Tmap-1.0-SNAPSHOT.jar/run?entry-class=com.skt.tmap.trajectorytracker.TrajectoryTracker&program-args=--topic%20gps-topic%20--kafkaParams%20bootstrap.servers=dacoe2:20245%2Cgroup.id=trajectory-tracker'

I get the following response from the dispatcher:

{
    "errors": [
        "Expected only one value [--topic gps-topic --kafkaParams bootstrap.servers=dacoe2:20245, group.id=trajectory-tracker]."

    ]
}

What I found from the source code is that org.apache.flink.runtime.rest.messages.MessageQueue tries split the value of program-ages using comma as a delimiter.

I think I could modify my program to get arguments in a different way but it is not going to be intuitive to use different characters for a delimiter instead of comma.

How you guys think?

Or there's a way to avoid this behavior of splitting the value of program-args into multiple pieces?

best,

- Dongwon