Problem

You have a streaming job that was reading an Event Hubs source and failed with an illegal argument exception error about incorrect partitions.

java.lang.IllegalArgumentException: requirement failed: partitions(2).partition == 3, but it should equal 2

Cause

The stream was reading an Event Hubs topic with 4 partitions (part-0, part-1, part-2, part-3). The error message indicates that the RDD partition at index 2 is part-3 but the expected partition is part-2.

The root cause was an issue in the Azure Event Hubs connector. The process caused a mismatch in the number of RDD partitions calculated between the ongoing RDD and the checkpoint metadata. When a partition offset rebalancing occurred and the sequence number was lagging and marked as expired by the Event Hubs connector, it led to a reconfiguration of the Event Hubs state which was incorrect.

Solution

You need to upgrade your Event Hubs connector to version 2.3.19 or above. This version contains a fix for the issue. Versions below 2.3.19 do not contain the fix.

Databricks recommends upgrading to Event Hubs connector version 2.3.21 or above. 

For more information, please review the Azure Event Hubs connector release notes.