Re: Replaying logs with microsecond delay

Posted by Xingcan Cui on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Replaying-logs-with-microsecond-delay-tp20181p20193.html

Yes, that makes sense and maybe you could also generate dynamic intervals according to the time spans.

Thanks,

Xingcan

On May 16, 2018, at 9:41 AM, Dhruv Kumar <[hidden email]> wrote:

As a part of my PhD research, I have been working on few optimization algorithms which try to jointly optimize delay and traffic (WAN traffic) in a geo-distributed streaming analytics setting. So, to show that the optimization actually works in real life, I am trying to implement these optimization algorithms on top of Apache Flink. For emulating a real life example, I need to generate a stream of records with some realistic delay (order of microseconds for fast incoming stream) between any two records. This stream will then by ingested and processed by Flink. 

Using the timestamps as is, in the form of event timestamps, only proves the algorithms from a theoretical/simulation perspective. 

Hope this answers your question to some extent at least. Let me know. 

Thanks!

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me

On May 15, 2018, at 20:29, Xingcan Cui <[hidden email]> wrote:

Hi Dhruv,

since there are timestamps associated with each record, I was wondering why you try to replay them with a fixed interval. Can you give a little explanation about that?

Thanks,

Xingcan

On May 16, 2018, at 2:11 AM, Ted Yu <[hidden email]> wrote:

Please see the following:

http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html

https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds

On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar <[hidden email]> wrote:

Hi

I am trying to replay a log file in which each record has a timestamp associated with it. The time difference between the records is of the order of microseconds. I am trying to replay this log maintaining the same delay between the records (using Thread.sleep()) and sending it to a socket. And then the Flink program reads the incoming data from this socket. Currently, replay of the entire log file takes much more time (3 times) then the expected time (last_timstamp - first_timstamp).

I wanted to know what are the standard ways of replaying log files if one wants to maintain the same arrival delay between the records.

Let me know if I am not clear above.

Thanks 

--------------------------------------------------

Dhruv Kumar
PhD Candidate
Department of Computer Science and Engineering
University of Minnesota
www.dhruvkumar.me