Enforce an operation to run in an exact host in flink

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Enforce an operation to run in an exact host in flink

Eranga Heshan
Hi all,

I have 4 VMs. There is an input stream and an output stream in my job. To get the exact time interval for an event to be processed, I need to run both input and output bolts inside the same host/node.

Is there a way to run a bolt inside a host we define?

Thank you,
Regards,

Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" target="_blank">+94 71 138 2686
Email:[hidden email]
    
Reply | Threaded
Open this post in threaded view
|

Re: Enforce an operation to run in an exact host in flink

Eranga Heshan
Any help please?


Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" target="_blank">+94 71 138 2686
Email:[hidden email]
    

On Thu, Nov 17, 2016 at 11:56 AM, Eranga Heshan <[hidden email]> wrote:
Hi all,

I have 4 VMs. There is an input stream and an output stream in my job. To get the exact time interval for an event to be processed, I need to run both input and output bolts inside the same host/node.

Is there a way to run a bolt inside a host we define?

Thank you,
Regards,

Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" target="_blank">+94 71 138 2686
Email:[hidden email]
    

Reply | Threaded
Open this post in threaded view
|

Re: Enforce an operation to run in an exact host in flink

Aljoscha Krettek
Hi,
I'm afraid that it is currently not possible to specify the machine on which a Flink operator should run. What do you need this for? Are you trying to measure processing latency?

Cheers,
Aljoscha

On Thu, 17 Nov 2016 at 09:15 Eranga Heshan <[hidden email]> wrote:
Any help please?


Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" class="gmail_msg" target="_blank">+94 71 138 2686
Email:[hidden email]
    

On Thu, Nov 17, 2016 at 11:56 AM, Eranga Heshan <[hidden email]> wrote:
Hi all,

I have 4 VMs. There is an input stream and an output stream in my job. To get the exact time interval for an event to be processed, I need to run both input and output bolts inside the same host/node.

Is there a way to run a bolt inside a host we define?

Thank you,
Regards,

Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" class="gmail_msg" target="_blank">+94 71 138 2686
Email:[hidden email]
    

Reply | Threaded
Open this post in threaded view
|

Re: Enforce an operation to run in an exact host in flink

Eranga Heshan

Yeah, I need to process latency. Any suggestions?


On Nov 18, 2016 12:47 AM, "Aljoscha Krettek" <[hidden email]> wrote:
Hi,
I'm afraid that it is currently not possible to specify the machine on which a Flink operator should run. What do you need this for? Are you trying to measure processing latency?

Cheers,
Aljoscha

On Thu, 17 Nov 2016 at 09:15 Eranga Heshan <[hidden email]> wrote:
Any help please?


Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" class="m_-1109831957807996094gmail_msg" target="_blank">+94 71 138 2686
Email:[hidden email]
    

On Thu, Nov 17, 2016 at 11:56 AM, Eranga Heshan <[hidden email]> wrote:
Hi all,

I have 4 VMs. There is an input stream and an output stream in my job. To get the exact time interval for an event to be processed, I need to run both input and output bolts inside the same host/node.

Is there a way to run a bolt inside a host we define?

Thank you,
Regards,

Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" class="m_-1109831957807996094gmail_msg" target="_blank">+94 71 138 2686
Email:[hidden email]
    

Reply | Threaded
Open this post in threaded view
|

Re: Enforce an operation to run in an exact host in flink

Aljoscha Krettek
If I'm not mistaken Flink already has metrics that report latencies. These are not 100% correct because they machine one which an element originates is not necessarily the machine on which we measure the latency.

I'm afraid that that's the best we can do right now, however. Other than running your pipeline on only one machine.

On Fri, 18 Nov 2016 at 01:32 Eranga Heshan <[hidden email]> wrote:

Yeah, I need to process latency. Any suggestions?


On Nov 18, 2016 12:47 AM, "Aljoscha Krettek" <[hidden email]> wrote:
Hi,
I'm afraid that it is currently not possible to specify the machine on which a Flink operator should run. What do you need this for? Are you trying to measure processing latency?

Cheers,
Aljoscha

On Thu, 17 Nov 2016 at 09:15 Eranga Heshan <[hidden email]> wrote:
Any help please?


Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" class="m_1119984193662350350m_-1109831957807996094gmail_msg gmail_msg" target="_blank">+94 71 138 2686
Email:[hidden email]
    

On Thu, Nov 17, 2016 at 11:56 AM, Eranga Heshan <[hidden email]> wrote:
Hi all,

I have 4 VMs. There is an input stream and an output stream in my job. To get the exact time interval for an event to be processed, I need to run both input and output bolts inside the same host/node.

Is there a way to run a bolt inside a host we define?

Thank you,
Regards,

Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" class="m_1119984193662350350m_-1109831957807996094gmail_msg gmail_msg" target="_blank">+94 71 138 2686
Email:[hidden email]
    

Reply | Threaded
Open this post in threaded view
|

Re: Enforce an operation to run in an exact host in flink

Eranga Heshan
Thank you Aijosha. 😃 I understood the concept.


Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" target="_blank">+94 71 138 2686
Email:[hidden email]
    

On Mon, Nov 21, 2016 at 10:28 PM, Aljoscha Krettek <[hidden email]> wrote:
If I'm not mistaken Flink already has metrics that report latencies. These are not 100% correct because they machine one which an element originates is not necessarily the machine on which we measure the latency.

I'm afraid that that's the best we can do right now, however. Other than running your pipeline on only one machine.

On Fri, 18 Nov 2016 at 01:32 Eranga Heshan <[hidden email]> wrote:

Yeah, I need to process latency. Any suggestions?


On Nov 18, 2016 12:47 AM, "Aljoscha Krettek" <[hidden email]> wrote:
Hi,
I'm afraid that it is currently not possible to specify the machine on which a Flink operator should run. What do you need this for? Are you trying to measure processing latency?

Cheers,
Aljoscha

On Thu, 17 Nov 2016 at 09:15 Eranga Heshan <[hidden email]> wrote:
Any help please?


Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" class="m_-2358234479326797989m_1119984193662350350m_-1109831957807996094gmail_msg m_-2358234479326797989gmail_msg" target="_blank">+94 71 138 2686
Email:[hidden email]
    

On Thu, Nov 17, 2016 at 11:56 AM, Eranga Heshan <[hidden email]> wrote:
Hi all,

I have 4 VMs. There is an input stream and an output stream in my job. To get the exact time interval for an event to be processed, I need to run both input and output bolts inside the same host/node.

Is there a way to run a bolt inside a host we define?

Thank you,
Regards,

Eranga Heshan
Undergraduate
Computer Science & Engineering
University of Moratuwa
Mobile: <a href="tel:%2B94%2071%20552%202087" value="+94715522087" style="color:rgb(17,85,204)" class="m_-2358234479326797989m_1119984193662350350m_-1109831957807996094gmail_msg m_-2358234479326797989gmail_msg" target="_blank">+94 71 138 2686
Email:[hidden email]