CKAN inputFormat (batch)

[Flavio Pompermaier]

Hi to all,
we've just published an example of a simple CKAN input format that downloads a CKAN resource (in parallel) from a CKAN catalog and produce a DataSet<Row>.
This can be very helpful in setting up a Flink demo using an OpenData dataset available online (see [1] for a list of available catalogs).

An example of its usage can be found here: [2].

Best,
Flavio

[1] https://ckan.org/about/instances/

[2] https://github.com/okkam-it/flink-examples/blob/master/src/main/java/it/okkam/flink/CkanDatasetImporter.java

[vino yang]

Hi Flavio,

I can not open the first link[1] you provided.

And what is your purpose? Introduce your CKAN input format to the community?

Thanks, vino.

[1]: https://ckan.org/about/instances/

Flavio Pompermaier <[hidden email]> 于2018年12月4日周二 上午1:09写道：

Hi to all,
we've just published an example of a simple CKAN input format that downloads a CKAN resource (in parallel) from a CKAN catalog and produce a DataSet<Row>.
This can be very helpful in setting up a Flink demo using an OpenData dataset available online (see [1] for a list of available catalogs).

An example of its usage can be found here: [2].

Best,
Flavio

[1] https://ckan.org/about/instances/

[2] https://github.com/okkam-it/flink-examples/blob/master/src/main/java/it/okkam/flink/CkanDatasetImporter.java

[Flavio Pompermaier]

Yesterday it was working...alternatively you can look at 

https://github.com/ckan/ckan-instances/blob/gh-pages/config/instances.json

The purpose of this code is to share with the community an example of a useful input format that coul be interesting for people working with open data (indeed there are lots of useful datasets in Ckan catalogues in the web). 

On Tue, 4 Dec 2018, 07:10 vino yang <[hidden email] wrote:

Hi Flavio,

I can not open the first link[1] you provided.

And what is your purpose? Introduce your CKAN input format to the community?

Thanks, vino.

[1]: https://ckan.org/about/instances/

Flavio Pompermaier <[hidden email]> 于2018年12月4日周二 上午1:09写道：

Hi to all,
we've just published an example of a simple CKAN input format that downloads a CKAN resource (in parallel) from a CKAN catalog and produce a DataSet<Row>.
This can be very helpful in setting up a Flink demo using an OpenData dataset available online (see [1] for a list of available catalogs).

An example of its usage can be found here: [2].

Best,
Flavio

[1] https://ckan.org/about/instances/

[2] https://github.com/okkam-it/flink-examples/blob/master/src/main/java/it/okkam/flink/CkanDatasetImporter.java

[Dawid Wysakowicz-2]

Hi Flavio,

Thank you for the example. It is definitely gonna be helpful for some people!

Best,

Dawid

On 04/12/2018 09:05, Flavio Pompermaier wrote:

Yesterday it was working...alternatively you can look at 

https://github.com/ckan/ckan-instances/blob/gh-pages/config/instances.json

The purpose of this code is to share with the community an example of a useful input format that coul be interesting for people working with open data (indeed there are lots of useful datasets in Ckan catalogues in the web). 

On Tue, 4 Dec 2018, 07:10 vino yang <[hidden email] wrote:

Hi Flavio,

I can not open the first link[1] you provided.

And what is your purpose? Introduce your CKAN input format to the community?

Thanks, vino.

[1]: https://ckan.org/about/instances/

Flavio Pompermaier <[hidden email]> 于2018年12月4日周二 上午1:09写道：

Hi to all,
we've just published an example of a simple CKAN input format that downloads a CKAN resource (in parallel) from a CKAN catalog and produce a DataSet<Row>.
This can be very helpful in setting up a Flink demo using an OpenData dataset available online (see [1] for a list of available catalogs).

An example of its usage can be found here: [2].

Best,
Flavio

[1] https://ckan.org/about/instances/

[2] https://github.com/okkam-it/flink-examples/blob/master/src/main/java/it/okkam/flink/CkanDatasetImporter.java