Re: API request to submit job takes over 1hr

Posted by rmetzger0 on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/API-request-to-submit-job-takes-over-1hr-tp7319p7600.html

Hi,

Regarding Shannon first point: I agree. We can improve the user experience a lot, and documenting the behavior is the first step we should do here.

I see your points. I agree that we should use a separate thread for running the main method and report better to the front end what's happening.

Re 2.: The main() method is executed on the Application Master / JobManager / Web Interface JVM, when submitting the job through the web interface.

When using the ./bin/flink tool, its executed there.

As Gordon said correctly, Flink doesn't pick up newly added partitions. Leader changes are handled transparently since 1.0.0.

I hope that clarify things.

On Tue, Jun 14, 2016 at 5:57 AM, Tzu-Li (Gordon) Tai <[hidden email]> wrote:

Hi Shannon,

Thanks for your investigation on the issue and the JIRA. There's actually a
previous JIRA on this problem already:
https://issues.apache.org/jira/browse/FLINK-4023. Would you be ok with
tracking this issue on FLINK-4023, and close FLINK-4069 as a duplicate
issue? As you can see, I've also referenced a link to FLINK-4069 on
FLINK-4023 for your additional info on the problem.

A little help with answering your last questions:
1. We're doing the partition distribution across consumers ourselves: the
Kafka consumer connector creates a Kafka client on subtasks, and each
subtask independently determines which partitions it should be in charge of.
There's also information on this blog here for more info:
http://data-artisans.com/kafka-flink-a-practical-how-to/, on the last FAQ
section. As Robert has mentioned, the consumer is currently depending on the
fixed ordered list of partitions sent to all subtasks so that each of them
always determine the same set of partitions to fetch from across restarts.
2. Following the above description, currently the consumer is only
subscribing to the fixed partition list queried in the constructor. So at
the moment the Flink Kafka consumer doesn't handle repartitioning of topics,
but it's definitely on the todo list for the Kafka connector and won't be
too hard to implement once querying in the consumer is resolved (perhaps
Robert can clarify this a bit more).

Best,
Gordon



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/API-request-to-submit-job-takes-over-1hr-tp7319p7558.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.