CSV Table source as data-stream in environment file

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

CSV Table source as data-stream in environment file

Nishant Gupta
Hi Team,

How do we define csv table source as a data-stream instead of data-set in environment file.?

Whether or not i mention  update-mode: append  or not... I takes only csv file as data-set.
Is there any detailed reference to environment file configuration where sinks and sources are defined.

 - name: badips
    type: source-table
    update-mode: append
    connector:
      type: filesystem
      path: "/home/user/file.csv"
    format:
      type: csv
      fields:
        - name: col1
          type: VARCHAR
      comment-prefix: "#"
    schema:
      - name: col1
        type: VARCHAR
Reply | Threaded
Open this post in threaded view
|

Re: CSV Table source as data-stream in environment file

Dian Fu
You need to add the following configuration to configure it run in streaming mode[1].

execution:
  type: streaming

Regards,
Dian


在 2019年9月26日,下午5:35,Nishant Gupta <[hidden email]> 写道:

Hi Team,

How do we define csv table source as a data-stream instead of data-set in environment file.?

Whether or not i mention  update-mode: append  or not... I takes only csv file as data-set.
Is there any detailed reference to environment file configuration where sinks and sources are defined.

 - name: badips
    type: source-table
    update-mode: append
    connector:
      type: filesystem
      path: "/home/user/file.csv"
    format:
      type: csv
      fields:
        - name: col1
          type: VARCHAR
      comment-prefix: "#"
    schema:
      - name: col1
        type: VARCHAR