Interested to Buy Any Domain ? << Click Here >> for more details...
How can you trigger automatic clean-ups in Spark to handle accumulated metadata?
- 1 Answers
- 467 Views
- I also Faced
- E-Mail Answers
Answer / Mukesh Kumar Gangwar
To trigger automatic clean-ups in Apache Spark, you can use the "spark.sql.execution.arrow.enableHadoopMetadataCleaner" configuration property. This property enables a metadata cleaner that cleans up Hadoop Distributed File System (HDFS) blocks when they are no longer needed. By default, it is set to false.
| Is This Answer Correct ? | 0 Yes | 0 No |
Is there a module to implement sql in spark? How does it work?
What are the roles and responsibilities of worker nodes in the Apache Spark cluster? Is Worker Node in Spark is same as Slave Node?
What do we mean by Paraquet?
Can you explain benefits of spark over mapreduce?
How to process data using Transformation operation in Spark?
What is the difference between map and flatmap?
Can you run spark on windows?
What is spark architecture?
To use Spark on an existing Hadoop Cluster, do we need to install Spark on all nodes of Hadoop?
What is lineage graph in Apache Spark?
How to create a Sparse vector from a dense vector?
How does reducebykey work in spark?
-
Apache Hadoop (394) - MapReduce (354)
- Apache Hive (345)
- Apache Pig (225)
- Apache Spark (991)
- Apache HBase (164)
- Apache Flume (95)
- Apache Impala (72)
- Apache Cassandra (392)
- Apache Mahout (35)
- Apache Sqoop (82)
- Apache ZooKeeper (65)
- Apache Ambari (93)
- Apache HCatalog (34)
- Apache HDFS Hadoop Distributed File System (214)
- Apache Kafka (189)
- Apache Avro (26)
- Apache Presto (15)
- Apache Tajo (26)
- Hadoop General (407)
