sampling function
Locked
 
|
Hi all, The second question is that are you planning to support the sample function for DataStream (within windows) since I did not see it in DataStream code ? Thank you, Do |
|
|
Hi Do, DataSet provides a stable @Public interface. DataSetUtils is marked @PublicEvolving which is intended for public use, has stable behavior, but method signatures may change. It's also good to limit DataSet to common methods whereas the utility methods tend to be used for specific applications. I don't have the pulse of streaming but this sounds like a useful feature that could be added. Greg On Sat, Jul 9, 2016 at 10:47 AM, Le Quoc Do <[hidden email]> wrote:
|
|
|
Hi Do,
In DataStream you can always implement your own sampling function, hopefully without too much effort. Adding such functionality it to the API could be a good idea. But given that in sampling there is no “one-size-fits-all” solution (as not every use case needs random sampling and not all random samplers fit to all workloads), I am not sure if we should start adding different sampling operators. Thanks, Kostas > On Jul 9, 2016, at 5:43 PM, Greg Hogan <[hidden email]> wrote: > > Hi Do, > > DataSet provides a stable @Public interface. DataSetUtils is marked > @PublicEvolving which is intended for public use, has stable behavior, but > method signatures may change. It's also good to limit DataSet to common > methods whereas the utility methods tend to be used for specific > applications. > > I don't have the pulse of streaming but this sounds like a useful feature > that could be added. > > Greg > > On Sat, Jul 9, 2016 at 10:47 AM, Le Quoc Do <[hidden email]> wrote: > >> Hi all, >> >> I'm working on approximate computing using sampling techniques. I >> recognized that Flink supports the sample function for Dataset >> (org/apache/flink/api/java/utils/DataSetUtils.java). I'm just wondering why >> you didn't merge the function to org/apache/flink/api/java/DataSet.java >> since the sample function works as a transformation operator? >> >> The second question is that are you planning to support the sample >> function for DataStream (within windows) since I did not see it in >> DataStream code ? >> >> Thank you, >> Do >> |
|
|
Hi Do, Paris and Martha worked on sampling techniques for data streams on Flink last year. If you want to implement your own samplers, you might find Martha's master thesis helpful [1]. -Vasia. On 11 July 2016 at 11:31, Kostas Kloudas <[hidden email]> wrote: Hi Do, |
|
|
Hi all,
Thank you all for your answers. By the way, I also recognized that Flink doesn't support "stratified sampling" function (only simple random sampling) for DataSet. It would be nice if someone can create a Jira for it, and assign the task to me so that I can work for it. Thank you, Do On Mon, Jul 11, 2016 at 11:44 AM, Vasiliki Kalavri <[hidden email]> wrote: Hi Do, |
|
|
Hey Do,
I think that more sophisticated samplers could make a better fit in the ML library and not in the core API but I am not very familiar with the milestones there. Maybe the maintainers of the batch ML library could check if sampling techniques could be useful there I guess. Paris > On 11 Jul 2016, at 16:15, Le Quoc Do <[hidden email]> wrote: > > Hi all, > > Thank you all for your answers. > By the way, I also recognized that Flink doesn't support "stratified > sampling" function (only simple random sampling) for DataSet. > It would be nice if someone can create a Jira for it, and assign the task > to me so that I can work for it. > > Thank you, > Do > > On Mon, Jul 11, 2016 at 11:44 AM, Vasiliki Kalavri < > [hidden email]> wrote: > >> Hi Do, >> >> Paris and Martha worked on sampling techniques for data streams on Flink >> last year. If you want to implement your own samplers, you might find >> Martha's master thesis helpful [1]. >> >> -Vasia. >> >> [1]: http://kth.diva-portal.org/smash/get/diva2:910695/FULLTEXT01.pdf >> >> On 11 July 2016 at 11:31, Kostas Kloudas <[hidden email]> >> wrote: >> >>> Hi Do, >>> >>> In DataStream you can always implement your own >>> sampling function, hopefully without too much effort. >>> >>> Adding such functionality it to the API could be a good idea. >>> But given that in sampling there is no “one-size-fits-all” >>> solution (as not every use case needs random sampling and not >>> all random samplers fit to all workloads), I am not sure if we >>> should start adding different sampling operators. >>> >>> Thanks, >>> Kostas >>> >>>> On Jul 9, 2016, at 5:43 PM, Greg Hogan <[hidden email]> wrote: >>>> >>>> Hi Do, >>>> >>>> DataSet provides a stable @Public interface. DataSetUtils is marked >>>> @PublicEvolving which is intended for public use, has stable behavior, >>> but >>>> method signatures may change. It's also good to limit DataSet to common >>>> methods whereas the utility methods tend to be used for specific >>>> applications. >>>> >>>> I don't have the pulse of streaming but this sounds like a useful >> feature >>>> that could be added. >>>> >>>> Greg >>>> >>>> On Sat, Jul 9, 2016 at 10:47 AM, Le Quoc Do <[hidden email]> >> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm working on approximate computing using sampling techniques. I >>>>> recognized that Flink supports the sample function for Dataset >>>>> (org/apache/flink/api/java/utils/DataSetUtils.java). I'm just >> wondering >>> why >>>>> you didn't merge the function to >> org/apache/flink/api/java/DataSet.java >>>>> since the sample function works as a transformation operator? >>>>> >>>>> The second question is that are you planning to support the sample >>>>> function for DataStream (within windows) since I did not see it in >>>>> DataStream code ? >>>>> >>>>> Thank you, >>>>> Do >>>>> >>> >>> >> |
|
|
Stratified sampling would also be beneficial for the DataSet API. I think it would be best if this method is also added to DataSetUtils or made available via the flink-contrib module. Furthermore, I think that it would be easiest if you created the JIRA for this feature, because you know what you want to add. For that you have to register at https://issues.apache.org/jira, if you haven't done this, and then we can add you as a contributor. Based on the JIRA description we can discuss possible implementations then. Cheers, Till On Tue, Jul 12, 2016 at 12:11 PM, Paris Carbone <[hidden email]> wrote: Hey Do, |
|
|
Hi Till,
I have created the JIRA: https://issues.apache.org/jira/browse/FLINK-4205 Thank you, Do On Tue, Jul 12, 2016 at 6:05 PM, Till Rohrmann <[hidden email]> wrote: Stratified sampling would also be beneficial for the DataSet API. I think |
«
Return to (DEPRECATED) Apache Flink User Mailing List archive.
|
1 view|%1 views
| Free forum by Nabble | Edit this page |