Hello, the performance of apply function after join. Just for your information, I am running Flink job on the cluster consisted of 9 machine with each 48 cores. I am working on some benchmark with comparison of Flink, Spark-Sql, and Hive. I tried to optimize join function with Hint for better performance. I want to increase the performance as much as possible. Here are Questions=== 1) When seeing job progress log, apply() after join function seems like it takes a bit long time. Do you think if I do not use apply() to format tuples, I would gain the better performance? Well, I could set just the column number instead of apply() 2) on using join with Hint like Huge or Tiny, is there the ideal ratio regarding to the size of two tables? For me, if some table is 10 times bigger than the other table, I use join with Hint. Otherwise, I usually use the general join(). Best, Phil |
Hi Phil, an apply method after a join runs pipelined with the join, i.e., it starts processing when the first join result is emitted and finishes after it handled the last join result. small.join(large, JoinHint.REPARTITION_HASH_SECOND) which will execute the join by shuffling both inputs and building a hash table on the partition of the second input. If you want to optimize for performance, you should try both hints: REPARTITION_HASH_*small* and BROADCAST_HASH_*small*. Best, Fabian 2015-12-01 21:34 GMT+01:00 Philip Lee <[hidden email]>:
|
Free forum by Nabble | Edit this page |