Replaying logs with microsecond delay

[Dhruv Kumar]

Hi

I am trying to replay a log file in which each record has a timestamp associated with it. The time difference between the records is of the order of microseconds. I am trying to replay this log maintaining the same delay between the records (using Thread.sleep()) and sending it to a socket. And then the Flink program reads the incoming data from this socket. Currently, replay of the entire log file takes much more time (3 times) then the expected time (last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

[Ted Yu]

Please see the following:

http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html

https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds

On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar <[hidden email]> wrote:

Hi

I am trying to replay a log file in which each record has a timestamp associated with it. The time difference between the records is of the order of microseconds. I am trying to replay this log maintaining the same delay between the records (using Thread.sleep()) and sending it to a socket. And then the Flink program reads the incoming data from this socket. Currently, replay of the entire log file takes much more time (3 times) then the expected time (last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

[Dhruv Kumar]

Thanks a lot, Ted. Appreciate your help!

The approaches specified in the below links, are giving a very good level of accuracy. Solves my problem for now.

Thanks

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

On May 15, 2018, at 13:11, Ted Yu <[hidden email]> wrote:

Please see the following:

http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html

https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds

On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar <[hidden email]> wrote:

Hi

I am trying to replay a log file in which each record has a timestamp associated with it. The time difference between the records is of the order of microseconds. I am trying to replay this log maintaining the same delay between the records (using Thread.sleep()) and sending it to a socket. And then the Flink program reads the incoming data from this socket. Currently, replay of the entire log file takes much more time (3 times) then the expected time (last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

[Xingcan Cui]

Hi Dhruv,

since there are timestamps associated with each record, I was wondering why you try to replay them with a fixed interval. Can you give a little explanation about that?

Thanks,

Xingcan

On May 16, 2018, at 2:11 AM, Ted Yu <[hidden email]> wrote:

Please see the following:

http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html

https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds

On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar <[hidden email]> wrote:

Hi

I am trying to replay a log file in which each record has a timestamp associated with it. The time difference between the records is of the order of microseconds. I am trying to replay this log maintaining the same delay between the records (using Thread.sleep()) and sending it to a socket. And then the Flink program reads the incoming data from this socket. Currently, replay of the entire log file takes much more time (3 times) then the expected time (last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

[Dhruv Kumar]

As a part of my PhD research, I have been working on few optimization algorithms which try to jointly optimize delay and traffic (WAN traffic) in a geo-distributed streaming analytics setting. So, to show that the optimization actually works in real life, I am trying to implement these optimization algorithms on top of Apache Flink. For emulating a real life example, I need to generate a stream of records with some realistic delay (order of microseconds for fast incoming stream) between any two records. This stream will then by ingested and processed by Flink. 

Using the timestamps as is, in the form of event timestamps, only proves the algorithms from a theoretical/simulation perspective. 

Hope this answers your question to some extent at least. Let me know. 

Thanks!

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

On May 15, 2018, at 20:29, Xingcan Cui <[hidden email]> wrote:

Hi Dhruv,

since there are timestamps associated with each record, I was wondering why you try to replay them with a fixed interval. Can you give a little explanation about that?

Thanks,

Xingcan

On May 16, 2018, at 2:11 AM, Ted Yu <[hidden email]> wrote:

Please see the following:

http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html

https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds

On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar <[hidden email]> wrote:

Hi

I am trying to replay a log file in which each record has a timestamp associated with it. The time difference between the records is of the order of microseconds. I am trying to replay this log maintaining the same delay between the records (using Thread.sleep()) and sending it to a socket. And then the Flink program reads the incoming data from this socket. Currently, replay of the entire log file takes much more time (3 times) then the expected time (last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

[Xingcan Cui]

Yes, that makes sense and maybe you could also generate dynamic intervals according to the time spans.

Thanks,

Xingcan

On May 16, 2018, at 9:41 AM, Dhruv Kumar <[hidden email]> wrote:

As a part of my PhD research, I have been working on few optimization algorithms which try to jointly optimize delay and traffic (WAN traffic) in a geo-distributed streaming analytics setting. So, to show that the optimization actually works in real life, I am trying to implement these optimization algorithms on top of Apache Flink. For emulating a real life example, I need to generate a stream of records with some realistic delay (order of microseconds for fast incoming stream) between any two records. This stream will then by ingested and processed by Flink. 

Using the timestamps as is, in the form of event timestamps, only proves the algorithms from a theoretical/simulation perspective. 

Hope this answers your question to some extent at least. Let me know. 

Thanks!

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

On May 15, 2018, at 20:29, Xingcan Cui <[hidden email]> wrote:

Hi Dhruv,

since there are timestamps associated with each record, I was wondering why you try to replay them with a fixed interval. Can you give a little explanation about that?

Thanks,

Xingcan

On May 16, 2018, at 2:11 AM, Ted Yu <[hidden email]> wrote:

Please see the following:

http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html

https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds

On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar <[hidden email]> wrote:

Hi

I am trying to replay a log file in which each record has a timestamp associated with it. The time difference between the records is of the order of microseconds. I am trying to replay this log maintaining the same delay between the records (using Thread.sleep()) and sending it to a socket. And then the Flink program reads the incoming data from this socket. Currently, replay of the entire log file takes much more time (3 times) then the expected time (last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

[Dhruv Kumar]

Yes, thanks!

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

On May 15, 2018, at 21:31, Xingcan Cui <[hidden email]> wrote:

Yes, that makes sense and maybe you could also generate dynamic intervals according to the time spans.

Thanks,

Xingcan

On May 16, 2018, at 9:41 AM, Dhruv Kumar <[hidden email]> wrote:

As a part of my PhD research, I have been working on few optimization algorithms which try to jointly optimize delay and traffic (WAN traffic) in a geo-distributed streaming analytics setting. So, to show that the optimization actually works in real life, I am trying to implement these optimization algorithms on top of Apache Flink. For emulating a real life example, I need to generate a stream of records with some realistic delay (order of microseconds for fast incoming stream) between any two records. This stream will then by ingested and processed by Flink. 

Using the timestamps as is, in the form of event timestamps, only proves the algorithms from a theoretical/simulation perspective. 

Hope this answers your question to some extent at least. Let me know. 

Thanks!

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

On May 15, 2018, at 20:29, Xingcan Cui <[hidden email]> wrote:

Hi Dhruv,

since there are timestamps associated with each record, I was wondering why you try to replay them with a fixed interval. Can you give a little explanation about that?

Thanks,

Xingcan

On May 16, 2018, at 2:11 AM, Ted Yu <[hidden email]> wrote:

Please see the following:

http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html

https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds

On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar <[hidden email]> wrote:

Hi

I am trying to replay a log file in which each record has a timestamp associated with it. The time difference between the records is of the order of microseconds. I am trying to replay this log maintaining the same delay between the records (using Thread.sleep()) and sending it to a socket. And then the Flink program reads the incoming data from this socket. Currently, replay of the entire log file takes much more time (3 times) then the expected time (last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me