Stateful functions presentation code (UI part)

[Flavio Pompermaier]

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio

[vino yang]

Hi Flavio,

Please see this link.[1]

Best,

Vino

[1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example

Flavio Pompermaier <[hidden email]> 于2019年10月31日周四 下午4:53写道：

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio

[Flavio Pompermaier]

Hi Vino,

I already checked that code but I can't find the UI part :(

On Thu, Oct 31, 2019 at 12:32 PM vino yang <[hidden email]> wrote:

Hi Flavio,

Please see this link.[1]

Best,

Vino

[1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example

Flavio Pompermaier <[hidden email]> 于2019年10月31日周四 下午4:53写道：

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio

[Igal Shilman]

Hi Flavio,

We haven't included the UI source code just yet, we've only used it for demos and talks.

The reason is that (1) we didn't put a lot of effort and time there (2) didn't check the time to go through the individual dependencies and licences.

But we will add that very soon.

Would having the UI code there would improve your understanding? or is there another reason? 

Thanks,

Igal

On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <[hidden email]> wrote:

Hi Vino,

I already checked that code but I can't find the UI part :(

On Thu, Oct 31, 2019 at 12:32 PM vino yang <[hidden email]> wrote:

Hi Flavio,

Please see this link.[1]

Best,

Vino

[1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example

Flavio Pompermaier <[hidden email]> 于2019年10月31日周四 下午4:53写道：

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio

[Flavio Pompermaier]

Yes, I'm interested in how to read data from a UI..which egress should I use? If we use a kafka queue, how to filter data received in the topic? Should I use a correlation id or use a new topic per user?

Il Gio 31 Ott 2019, 16:08 Igal Shilman <[hidden email]> ha scritto:

Hi Flavio,

We haven't included the UI source code just yet, we've only used it for demos and talks.

The reason is that (1) we didn't put a lot of effort and time there (2) didn't check the time to go through the individual dependencies and licences.

But we will add that very soon.

Would having the UI code there would improve your understanding? or is there another reason? 

Thanks,

Igal

On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <[hidden email]> wrote:

Hi Vino,

I already checked that code but I can't find the UI part :(

On Thu, Oct 31, 2019 at 12:32 PM vino yang <[hidden email]> wrote:

Hi Flavio,

Please see this link.[1]

Best,

Vino

[1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example

Flavio Pompermaier <[hidden email]> 于2019年10月31日周四 下午4:53写道：

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio

[Igal Shilman]

For that particular example, the simulator [1] is responsible for simulating physical drivers and passengers that interact with their corresponding 

stateful functions [2].

The interaction between the simulator and the stateful functions is happening via four Kafka topics:

* to-driver - messages that are sent from the physical drivers (the simulator [1]) to the stateful functions. The messages always carry a driver-id which acts as a routing key (I think that this is what you mean by correlation id) to a specific driver stateful function (FnDriver)

* from-driver - messages that are sent from a stateful function with a specific driver id to the simulator

* to-passenger - symmetric to to-driver

* from-passenger - symmetric to from-driver.

The ingress and egress definition are specified here [3], and you may want to checkout how to router is defined as well [4][5].

In addition the simulator is also feeding the UI directly by duplicating the messages to a web socket (see [6])

I hope this clarifies the examples.

Igal.

[1] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator

[2] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions

[3] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43

[4] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33

[5] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26

[6] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70

On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <[hidden email]> wrote:

Yes, I'm interested in how to read data from a UI..which egress should I use? If we use a kafka queue, how to filter data received in the topic? Should I use a correlation id or use a new topic per user?

Il Gio 31 Ott 2019, 16:08 Igal Shilman <[hidden email]> ha scritto:

Hi Flavio,

We haven't included the UI source code just yet, we've only used it for demos and talks.

The reason is that (1) we didn't put a lot of effort and time there (2) didn't check the time to go through the individual dependencies and licences.

But we will add that very soon.

Would having the UI code there would improve your understanding? or is there another reason? 

Thanks,

Igal

On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <[hidden email]> wrote:

Hi Vino,

I already checked that code but I can't find the UI part :(

On Thu, Oct 31, 2019 at 12:32 PM vino yang <[hidden email]> wrote:

Hi Flavio,

Please see this link.[1]

Best,

Vino

[1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example

Flavio Pompermaier <[hidden email]> 于2019年10月31日周四 下午4:53写道：

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio

[Flavio Pompermaier]

Thanks Igal, this is more or less what I was expecting..this implies that ALL events are received on the UI side.

I was concerned about the tradeoffs of this choice: when I zoom on the map I could simply ignore messages outside the boundaries (but I still spend many cpu resource in the reading of useless messages).

In the case of a worldwide company (eg uber or simolar) it's probably better to create a topic per geographical arean..but also in this case then the UI should know when to attach or detach from queue topics when it reach a section of the map served by different topics..in another case I could just have too many events also in the single map section (let's think about some big city with many user).

Is there any talk at FF ir someone else that faced those issues too?

Il Gio 31 Ott 2019, 17:44 Igal Shilman <[hidden email]> ha scritto:

For that particular example, the simulator [1] is responsible for simulating physical drivers and passengers that interact with their corresponding 

stateful functions [2].

The interaction between the simulator and the stateful functions is happening via four Kafka topics:

* to-driver - messages that are sent from the physical drivers (the simulator [1]) to the stateful functions. The messages always carry a driver-id which acts as a routing key (I think that this is what you mean by correlation id) to a specific driver stateful function (FnDriver)

* from-driver - messages that are sent from a stateful function with a specific driver id to the simulator

* to-passenger - symmetric to to-driver

* from-passenger - symmetric to from-driver.

The ingress and egress definition are specified here [3], and you may want to checkout how to router is defined as well [4][5].

In addition the simulator is also feeding the UI directly by duplicating the messages to a web socket (see [6])

I hope this clarifies the examples.

Igal.

[1] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator

[2] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions

[3] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43

[4] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33

[5] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26

[6] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70

On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <[hidden email]> wrote:

Yes, I'm interested in how to read data from a UI..which egress should I use? If we use a kafka queue, how to filter data received in the topic? Should I use a correlation id or use a new topic per user?

Il Gio 31 Ott 2019, 16:08 Igal Shilman <[hidden email]> ha scritto:

Hi Flavio,

We haven't included the UI source code just yet, we've only used it for demos and talks.

The reason is that (1) we didn't put a lot of effort and time there (2) didn't check the time to go through the individual dependencies and licences.

But we will add that very soon.

Would having the UI code there would improve your understanding? or is there another reason? 

Thanks,

Igal

On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <[hidden email]> wrote:

Hi Vino,

I already checked that code but I can't find the UI part :(

On Thu, Oct 31, 2019 at 12:32 PM vino yang <[hidden email]> wrote:

Hi Flavio,

Please see this link.[1]

Best,

Vino

[1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example

Flavio Pompermaier <[hidden email]> 于2019年10月31日周四 下午4:53写道：

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio

[Igal Shilman]

Hi Flavio, let me try to clarify:

The intention of this example is to demonstrate how different entities (drivers, passengers, etc') participates in a protocol (ride matching). For that we have the stateful functions application, and a standalone java application that just generates the events to trigger the simulation (i.e. a passenger requests a ride would be an event that the simulator emits).

The visualization aspect of this example is not a part of what we are trying to demonstrate but rather, an out-of-band peek into the live system (websocket that duplicates the events directly to the UI) which we thought is a nice addition to the example that visualizes how cars are moving on the grid :-) 

If your goal is to create a scalable visualization of car locations, then I guess your suggestion to break the grid into distinct geo-regions can work. For that you would need:

1. An additional egress backed by a Kafka topic.

2. Modify the driver function to send location changes to that egress, where you'd use the region id (geohash with a resolution to your liking) as key [1]. 

3. At the consuming side subscribe to the partition that holds the region of interest.

But, realistically speaking for this kind of a visualization you would be better off consuming from the locations update topic (from-driver topic) and populating an optimized geo-indexed store,

and querying it directly.

I hope this clarifies things,

Igal 

[1] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121

On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier <[hidden email]> wrote:

Thanks Igal, this is more or less what I was expecting..this implies that ALL events are received on the UI side.

I was concerned about the tradeoffs of this choice: when I zoom on the map I could simply ignore messages outside the boundaries (but I still spend many cpu resource in the reading of useless messages).

In the case of a worldwide company (eg uber or simolar) it's probably better to create a topic per geographical arean..but also in this case then the UI should know when to attach or detach from queue topics when it reach a section of the map served by different topics..in another case I could just have too many events also in the single map section (let's think about some big city with many user).

Is there any talk at FF ir someone else that faced those issues too?

Il Gio 31 Ott 2019, 17:44 Igal Shilman <[hidden email]> ha scritto:

For that particular example, the simulator [1] is responsible for simulating physical drivers and passengers that interact with their corresponding 

stateful functions [2].

The interaction between the simulator and the stateful functions is happening via four Kafka topics:

* to-driver - messages that are sent from the physical drivers (the simulator [1]) to the stateful functions. The messages always carry a driver-id which acts as a routing key (I think that this is what you mean by correlation id) to a specific driver stateful function (FnDriver)

* from-driver - messages that are sent from a stateful function with a specific driver id to the simulator

* to-passenger - symmetric to to-driver

* from-passenger - symmetric to from-driver.

The ingress and egress definition are specified here [3], and you may want to checkout how to router is defined as well [4][5].

In addition the simulator is also feeding the UI directly by duplicating the messages to a web socket (see [6])

I hope this clarifies the examples.

Igal.

[1] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator

[2] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions

[3] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43

[4] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33

[5] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26

[6] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70

On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <[hidden email]> wrote:

Yes, I'm interested in how to read data from a UI..which egress should I use? If we use a kafka queue, how to filter data received in the topic? Should I use a correlation id or use a new topic per user?

Il Gio 31 Ott 2019, 16:08 Igal Shilman <[hidden email]> ha scritto:

Hi Flavio,

We haven't included the UI source code just yet, we've only used it for demos and talks.

The reason is that (1) we didn't put a lot of effort and time there (2) didn't check the time to go through the individual dependencies and licences.

But we will add that very soon.

Would having the UI code there would improve your understanding? or is there another reason? 

Thanks,

Igal

On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <[hidden email]> wrote:

Hi Vino,

I already checked that code but I can't find the UI part :(

On Thu, Oct 31, 2019 at 12:32 PM vino yang <[hidden email]> wrote:

Hi Flavio,

Please see this link.[1]

Best,

Vino

[1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example

Flavio Pompermaier <[hidden email]> 于2019年10月31日周四 下午4:53写道：

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio

[Flavio Pompermaier]

Thanks Igal for the detailed explanantion. I know that this was only a demo, I just wanted to reason a bit on the pros and cons of sending data to an UI from Flink.

Best,

Flavio

Il Ven 1 Nov 2019, 12:21 Igal Shilman <[hidden email]> ha scritto:

Hi Flavio, let me try to clarify:

The intention of this example is to demonstrate how different entities (drivers, passengers, etc') participates in a protocol (ride matching). For that we have the stateful functions application, and a standalone java application that just generates the events to trigger the simulation (i.e. a passenger requests a ride would be an event that the simulator emits).

The visualization aspect of this example is not a part of what we are trying to demonstrate but rather, an out-of-band peek into the live system (websocket that duplicates the events directly to the UI) which we thought is a nice addition to the example that visualizes how cars are moving on the grid :-) 

If your goal is to create a scalable visualization of car locations, then I guess your suggestion to break the grid into distinct geo-regions can work. For that you would need:

1. An additional egress backed by a Kafka topic.

2. Modify the driver function to send location changes to that egress, where you'd use the region id (geohash with a resolution to your liking) as key [1]. 

3. At the consuming side subscribe to the partition that holds the region of interest.

But, realistically speaking for this kind of a visualization you would be better off consuming from the locations update topic (from-driver topic) and populating an optimized geo-indexed store,

and querying it directly.

I hope this clarifies things,

Igal 

[1] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L121

On Thu, Oct 31, 2019 at 6:21 PM Flavio Pompermaier <[hidden email]> wrote:

Thanks Igal, this is more or less what I was expecting..this implies that ALL events are received on the UI side.

I was concerned about the tradeoffs of this choice: when I zoom on the map I could simply ignore messages outside the boundaries (but I still spend many cpu resource in the reading of useless messages).

In the case of a worldwide company (eg uber or simolar) it's probably better to create a topic per geographical arean..but also in this case then the UI should know when to attach or detach from queue topics when it reach a section of the map served by different topics..in another case I could just have too many events also in the single map section (let's think about some big city with many user).

Is there any talk at FF ir someone else that faced those issues too?

Il Gio 31 Ott 2019, 17:44 Igal Shilman <[hidden email]> ha scritto:

For that particular example, the simulator [1] is responsible for simulating physical drivers and passengers that interact with their corresponding 

stateful functions [2].

The interaction between the simulator and the stateful functions is happening via four Kafka topics:

* to-driver - messages that are sent from the physical drivers (the simulator [1]) to the stateful functions. The messages always carry a driver-id which acts as a routing key (I think that this is what you mean by correlation id) to a specific driver stateful function (FnDriver)

* from-driver - messages that are sent from a stateful function with a specific driver id to the simulator

* to-passenger - symmetric to to-driver

* from-passenger - symmetric to from-driver.

The ingress and egress definition are specified here [3], and you may want to checkout how to router is defined as well [4][5].

In addition the simulator is also feeding the UI directly by duplicating the messages to a web socket (see [6])

I hope this clarifies the examples.

Igal.

[1] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator

[2] https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions

[3] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/KafkaSpecs.java#L43

[4] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/Module.java#L33

[5] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-functions/src/main/java/com/ververica/statefun/examples/ridesharing/InboundDriverRouter.java#L26

[6] https://github.com/ververica/stateful-functions/blob/master/stateful-functions-examples/stateful-functions-ridesharing-example/stateful-functions-ridesharing-example-simulator/src/main/java/com/ververica/statefun/examples/ridesharing/simulator/simulation/Driver.java#L70

On Thu, Oct 31, 2019 at 4:15 PM Flavio Pompermaier <[hidden email]> wrote:

Yes, I'm interested in how to read data from a UI..which egress should I use? If we use a kafka queue, how to filter data received in the topic? Should I use a correlation id or use a new topic per user?

Il Gio 31 Ott 2019, 16:08 Igal Shilman <[hidden email]> ha scritto:

Hi Flavio,

We haven't included the UI source code just yet, we've only used it for demos and talks.

The reason is that (1) we didn't put a lot of effort and time there (2) didn't check the time to go through the individual dependencies and licences.

But we will add that very soon.

Would having the UI code there would improve your understanding? or is there another reason? 

Thanks,

Igal

On Thu, Oct 31, 2019 at 5:44 AM Flavio Pompermaier <[hidden email]> wrote:

Hi Vino,

I already checked that code but I can't find the UI part :(

On Thu, Oct 31, 2019 at 12:32 PM vino yang <[hidden email]> wrote:

Hi Flavio,

Please see this link.[1]

Best,

Vino

[1]: https://github.com/ververica/stateful-functions/tree/master/stateful-functions-examples/stateful-functions-ridesharing-example

Flavio Pompermaier <[hidden email]> 于2019年10月31日周四 下午4:53写道：

Hi to all,

yould it be possible to provide also the source code of the UI part of the ride sharing example? It would be interesting to me how the UI is reading the data from the Kafka egress.

Best,

Flavio