How to efficiently manage state store files in Apache Spark streaming applications

To prevent the indefinite growth of your State Store (even when the watermark is updated), you can improve how efficiently you manage the lifecycle of your state store files in  Apache Spark Structured Streaming applications .  This applies to both Hadoop Distributed File System and RocksDB-based providers. Handling instructions In any Stateful Stre...

1 min reading time