[GLUTEN-12225][CORE] Fix arrow.c shading: exclude memory/vector packages so public API stays unshaded

[sezruby]

What changes were proposed in this pull request?

Extend package/pom.xml's org.apache.arrow relocation excludes to also keep org.apache.arrow.memory.** and org.apache.arrow.vector.** unshaded.

The bundled Arrow C-Data classes (org.apache.arrow.c.*) are correctly excluded from relocation because their native JNI binds to the original class names. However, their public API signatures take and return org.apache.arrow.memory.* and org.apache.arrow.vector.* types — which were being relocated. The result: the bundled ArrowArrayStream / ArrowSchema / ArrowArray / Data classes get compiled against the shaded BufferAllocator / VectorSchemaRoot, so any caller passing a vanilla Apache Arrow allocator hits NoSuchMethodError.

This affects any Spark workload that combines gluten with another library using Arrow C-Data (Iceberg's Arrow vector layer, Lance Java's writer, Snowflake JDBC's Arrow result decoder, etc.) when gluten's bundle wins classloader resolution against vanilla Arrow.

How was this patch tested?

Adds dev/check-arrow-c-shading.sh which runs javap on the produced bundle jar and asserts that public method signatures reference unshaded Arrow types. Wired into package/pom.xml's verify phase via exec-maven-plugin so regressions are caught in CI.

Tested against the upstream gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.6.0.jar:

$ dev/check-arrow-c-shading.sh /path/to/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.6.0.jar
  FAIL org/apache/arrow/c/ArrowArrayStream — public API references gluten-shaded Arrow types:
      public static org.apache.arrow.c.ArrowArrayStream allocateNew(
        org.apache.gluten.shaded.org.apache.arrow.memory.BufferAllocator);
  FAIL org/apache/arrow/c/ArrowSchema — public API references gluten-shaded Arrow types:
      public static org.apache.arrow.c.ArrowSchema allocateNew(
        org.apache.gluten.shaded.org.apache.arrow.memory.BufferAllocator);
  FAIL org/apache/arrow/c/ArrowArray — public API references gluten-shaded Arrow types:
      public static org.apache.arrow.c.ArrowArray allocateNew(
        org.apache.gluten.shaded.org.apache.arrow.memory.BufferAllocator);
  FAIL org/apache/arrow/c/Data — public API references gluten-shaded Arrow types:
      [16 methods touching shaded org.apache.arrow.memory/vector types]

Bundle has 4 Arrow C-Data class(es) with shaded API types.
exit code: 1

After applying the relocation exclude change, a freshly-built bundle should pass the same check (script exits 0). The repro from #12225 (3 lines calling ArrowArrayStream.allocateNew(new RootAllocator(...)) ) goes from NoSuchMethodError to OK.

Closes

#12225

[sezruby]

@philo-he, @zhouyuan could you have a look at the PR?

[philo-he]

@sezruby, thanks for the PR. This fix makes sense. I recall there's a related issue that occurs at compile time when an external project introduces the Gluten JAR as a dependency: a Scala type mismatch caused by the Maven Shade Plugin not rewriting ScalaSignature annotations. My understanding is this PR also fixes that case (see https://chungmin.hashnode.dev/unraveling-a-scala-type-mismatch-mystery).

One small concern is potential Arrow version conflicts, since these packages are no longer shaded. That said, the memory and vector APIs should be stable across minor versions, so I assume the risk should be low in practice.

cc @zhztheplayer

[zhztheplayer]

nit: Should we directly exclude all arrow packages? E.g., org.apache.arrow.*

[sezruby]

The full org.apache.arrow.* exclusion would lose gluten's isolation from the user's Arrow version everywhere, not just on the C-Data boundary. The C-Data classes have to be unshaded because their JNI native lib hardcodes the original class names; arrow.memory.* and arrow.vector.* follow because they appear in arrow.c.* public method signatures. Anything else under org.apache.arrow.* (flight, algorithm, adapter, etc.) is internal to gluten's columnar batch handling and safer to keep shaded so it doesn't conflict with user Arrow. The narrow exclusion is the minimum that makes the public C-Data API self-consistent without giving up isolation elsewhere.

[zhztheplayer]

That said, the memory and vector APIs should be stable across minor versions

This sounds a real risk. Moving forward, can we completely remove Arrow from the bundled Gluten Jar, and let user rely on Spark's bundled Arrow instead?

I assume we don't have any customized Arrow code now: #12130

[sezruby]

can we completely remove Arrow from the bundled Gluten Jar, and let user rely on Spark's bundled Arrow instead? I assume we don't have any customized Arrow code now: #12130

Worth doing, but a couple of things worth confirming first:

#12130 removed the dead Arrow-CSV / Arrow-Dataset JVM paths. That's progress, but there are still ~34 files under gluten-arrow/ and elsewhere that call org.apache.arrow.* (ColumnarBatches, ArrowColumnVector, ArrowWritableColumnVector, ColumnarBatchSerializer, JNI columnar batch bridges, SparkArrowUtil, etc.). Those code paths are alive on the Velox hot path and the bundled Arrow is what they currently resolve against.

If we drop the bundled Arrow:

1. arrow.version in gluten's poms would need to align with whatever the lowest-common-denominator target Spark ships — Arrow 12 for Spark 3.5 (DBR 16.4 ships 12.0.1, vanilla Spark 3.5.x ships 15.0.0), Arrow 18 for Spark 4.x. Compile profile per Spark version, similar to how <spark.version> is already keyed.
2. No 15.0.0-gluten source patches relied on — needs confirmation. The arrow-gluten.version = 15.0.0-gluten in pom.xml suggests there's at least some customization, even if the dead code in [MINOR][VL] Remove dead Arrow-CSV / Arrow-Dataset JVM code paths #12130 was the main consumer. Worth a sweep for any remaining call site that depends on a non-public Arrow API.
3. Testing matrix — minor: confirm no NoSuchMethodError on the Spark distros gluten claims support for, especially DBR / Cloudera flavors that ship older Arrow.

So I think it's the right long-term direction. As an immediate fix this PR makes the current shading approach internally consistent (which is independently a valid bug fix, since the partial-shading is a latent bug regardless of whether Arrow gets unbundled later). Happy to take either path — let me know if you'd prefer I close this and pursue the Arrow-unbundling work instead, or merge this as the short-term fix and treat unbundling as a follow-up.

a Scala type mismatch caused by the Maven Shade Plugin not rewriting ScalaSignature annotations

@philo-he Good link, the partial-shading also breaks downstream Scala consumers that pull the gluten jar as a dep, since their compile-time Arrow type doesn't match what gluten expects across the API boundary. This PR fixes that as a side effect by keeping the boundary types unshaded so they match the public Apache Arrow types every other Scala consumer compiles against. cc author @clee704