![]() ![]() The period at which the filesystem history provider checks for new or Or that of an alternative filesystem supported by the Hadoop APIs. This can be a local file:// path,Īn HDFS path hdfs://namenode/shared/spark-logs One implementation, provided by Spark, which looks for application logs stored in theįor the filesystem history provider, the URL to the directory containing application event Name of the class implementing the application history backend. Security options for the Spark History Server are covered more detail in the Spark History Server Configuration Options The compaction may exclude more events than you expect, leading some UI issues on History Server for the application. Please also note that this is a new feature introduced in Spark 3.0, and may not be completely stable. Will run as each micro-batch will trigger one or more jobs which will be finished shortly, but compaction won’t run For streaming query we normally expect compaction Please note that Spark History Server may not compact the old event log files if figures out not a lot of space The original log files, but it will not affect the operation of the History Server. The History Server may not be able to delete ![]() Once rewriting is done, original log files will be deleted, via best-effort manner.
0 Comments
Leave a Reply. |