debug for Flink

[Qian Ye]

Hi

I’m wondering if new debugging methods/tools  are urgent for Flink development. I know there already exists some debug methods for Flink, e.g., remote debugging of flink clusters(https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters). But are they are convenient enough? 

Best regards.

[Fabian Hueske-2]

Hi,

You can run Flink applications locally in your IDE and debug a Flink program just like a regular Java/Scala application.

Best, Fabian

2018-04-19 0:53 GMT+02:00 Qian Ye <[hidden email]>:

Hi

I’m wondering if new debugging methods/tools  are urgent for Flink development. I know there already exists some debug methods for Flink, e.g., remote debugging of flink clusters(https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters). But are they are convenient enough? 

Best regards.

[Kien Truong]

Hi,

Our most useful tool when debugging Flink is actually the simple log
files, because debugger just slow things down too much for us.

However, having to re-deploy the entire cluster to change the logging
level is a pain (we use YARN),

so we would really like an easier method to change the logging level at
runtime.


Regards,

Kien


On 4/19/2018 5:53 AM, Qian Ye wrote:
> Hi
>
> I’m wondering if new debugging methods/tools  are urgent for Flink
> development. I know there already exists some debug methods for Flink,
> e.g., remote debugging of flink
> clusters(https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters).
> But are they are convenient enough?
>
> Best regards.

[Qian Ye]

That sounds nice. But the lazy evaluation feature of Flink seems to cause the debug process more different with a regular Java/Scala application. Do you know how to debug a Flink application, e.g., tracing some local variables, in IDE?

Best,

Stephen

On Apr 19, 2018, at 6:29 AM, Fabian Hueske <[hidden email]> wrote:

Hi,

You can run Flink applications locally in your IDE and debug a Flink program just like a regular Java/Scala application.

Best, Fabian

2018-04-19 0:53 GMT+02:00 Qian Ye <[hidden email]>:

Hi

I’m wondering if new debugging methods/tools  are urgent for Flink development. I know there already exists some debug methods for Flink, e.g., remote debugging of flink clusters(https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters). But are they are convenient enough? 

Best regards.

[Qian Ye]

Thanks for your kind reply.
But I still have some question. What does the logging level mean in your system? Why do you need to re-deploy the cluster to change the logging level?  As far as I know, the debugging information can be divided into level like info, warn, error, etc. Is these information totally logged during the system running? And then they can further be analyzed by category them by level?

Best,
Stephen

> On Apr 19, 2018, at 7:19 AM, Kien Truong <[hidden email]> wrote:
>
> Hi,
>
> Our most useful tool when debugging Flink is actually the simple log files, because debugger just slow things down too much for us.
>
> However, having to re-deploy the entire cluster to change the logging level is a pain (we use YARN),
>
> so we would really like an easier method to change the logging level at runtime.
>
>
> Regards,
>
> Kien
>
>
> On 4/19/2018 5:53 AM, Qian Ye wrote:
>> Hi
>>
>> I’m wondering if new debugging methods/tools  are urgent for Flink development. I know there already exists some debug methods for Flink, e.g., remote debugging of flink clusters(https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters). But are they are convenient enough?
>>
>> Best regards.

[Kien Truong]

Hi,

We put a lot of logging in our code the flow of events as they flow
between operators,

this is done at the DEBUG level.

Normally this logging would be turn off by setting the logging level to
INFO or above,

because logging everything is expensive and a cluttered log is harder to
analyze.

When we need to switch the logging to DEBUG level,

we currently change the configuration files, then restart the job,

so that Flink would pick up and use the updated files during redeployment.

The other option is to manually track down the location of each YARN
container,

and update the configuration files directly,  but that's a lot of work,

and not possible if you don't have superuser right on that machine, due
to our security setup.

That's why we want an easier way to change the logging level.


Best regards,

Kien


On 4/20/2018 12:15 AM, Qian Ye wrote:

> Thanks for your kind reply.
> But I still have some question. What does the logging level mean in your system? Why do you need to re-deploy the cluster to change the logging level?  As far as I know, the debugging information can be divided into level like info, warn, error, etc. Is these information totally logged during the system running? And then they can further be analyzed by category them by level?
>
> Best,
> Stephen
>
>> On Apr 19, 2018, at 7:19 AM, Kien Truong <[hidden email]> wrote:
>>
>> Hi,
>>
>> Our most useful tool when debugging Flink is actually the simple log files, because debugger just slow things down too much for us.
>>
>> However, having to re-deploy the entire cluster to change the logging level is a pain (we use YARN),
>>
>> so we would really like an easier method to change the logging level at runtime.
>>
>>
>> Regards,
>>
>> Kien
>>
>>
>> On 4/19/2018 5:53 AM, Qian Ye wrote:
>>> Hi
>>>
>>> I’m wondering if new debugging methods/tools  are urgent for Flink development. I know there already exists some debug methods for Flink, e.g., remote debugging of flink clusters(https://cwiki.apache.org/confluence/display/FLINK/Remote+Debugging+of+Flink+Clusters). But are they are convenient enough?
>>>
>>> Best regards.