Partitioned tables in SQL client configuration.

Posted by Maciek Próchniak on
URL: http://deprecated-apache-flink-user-mailing-list-archive.369.s1.nabble.com/Partitioned-tables-in-SQL-client-configuration-tp39843.html

Hello,

I try to configure SQL Client to query partitioned ORC data on local
filesystem. I have directory structure like that:

/tmp/table1/startdate=2020-11-28

/tmp/table1/startdate=2020-11-27

etc.


If I run SQL Client session and create table by hand:

create table tst (column1 string, startdate string) partitioned by
(startdate) with ('connector'='filesystem', 'format'='orc',
'path'='/tmp/table1');

everything runs fine:

explain select * from tst where startdate='2020-11-27'

shows that only one partition in 'readPartitions'


However, I struggle to configure table in .yaml config.

I tried like this (after some struggle, as "partition.keys" setting
doesn't seem to be documented...) :

tables:
   - name: tst2
     type: source-table
     connector: filesystem
     path: "/tmp/table1"
     format: orc
     partition.keys:
       - name: startdate
     schema:
       - name: column1
         data-type: string
       - name: startdate
         data-type: string

and it more or less works - queries are executed properly. However,
partitions are not pruned:

explain select * from tst2 where startdate='2020-11-27'

show all partitions in 'readPartitions'


Any idea what can be wrong? I'm using Flink 1.11.2


thanks,

maciek