Answer by Alfilercio for While writing to hdfs path getting error java.io.IOException: Failed to rename

You can do all the selects in one single job, get all the selects and union in a single table.

Dataset<Row> resultDs = givenItemList.parallelStream().map( item -> {       String query = "select $item  as itemCol , avg($item) as mean groupBy year";    return sparkSession.sql(query);}).reduce((a, b) -> a.union(b)).getsaveDsToHdfs(hdfsPath, resultDs );