[SPARK-56221][SQL][PYTHON] Feature parity between spark.catalog.* vs DDL commands

### What changes were proposed in this pull request?

**SQL**
- `SHOW CACHED TABLES`: lists relations cached with an explicit name (`CACHE TABLE`, `catalog.cacheTable`, etc.); unnamed `Dataset.cache()` entries are not listed.

**`Catalog` API (Scala / Java / PySpark)**
- `listCachedTables()`: same information as `SHOW CACHED TABLES` (`CachedTable`: name, storage level).
- `dropTable` / `dropView`: drop persistent table or view (with `ifExists`, `purge` where applicable).
- `createDatabase` / `dropDatabase`: create or drop a namespace (with options: `ifNotExists` / `ifExists`, `cascade`, properties map).
- `listPartitions`: partition strings for a table (aligned with `SHOW PARTITIONS`).
- `listViews`: list views in the current or given namespace; optional name pattern.
- `getTableProperties`: all table properties (aligned with `SHOW TBLPROPERTIES`).
- `getCreateTableString`: DDL from `SHOW CREATE TABLE` (optional `asSerde`).
- `truncateTable`: remove all table data (not for views).
- `analyzeTable`: `ANALYZE TABLE ... COMPUTE STATISTICS` (optional `noScan`).

### Why are the changes needed?

Gives stable programmatic ways to do what users already do with SQL (SHOW CACHED TABLES, SHOW PARTITIONS, etc.), without routing everything through raw SQL.

### Does this PR introduce any user-facing change?

Yes. New SQL command, new Catalog API API.

### How was this patch tested?

Unittests were added.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #55025 from HyukjinKwon/SPARK-56221.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

Author: HyukjinKwon

common/utils/src/main/resources/error/error-conditions.json

@@ -7463,6 +7463,11 @@
           "The ANALYZE TABLE command does not support views."
         ]
       },
+      "CATALOG_INTERFACE_METHOD" : {
+        "message" : [
+          "Catalog API <methodName> is not supported by <catalogClass>."
+        ]
+      },
       "CATALOG_OPERATION" : {
         "message" : [
           "Catalog <catalogName> does not support <operation>."

docs/sql-performance-tuning.md

@@ -30,6 +30,8 @@ Spark SQL can cache tables using an in-memory columnar format by calling `spark.
 Then Spark SQL will scan only required columns and will automatically tune compression to minimize
 memory usage and GC pressure. You can call `spark.catalog.uncacheTable("tableName")` or `dataFrame.unpersist()` to remove the table from memory.
 
+To list relations cached with an explicit name, use `SHOW CACHED TABLES` in SQL or `spark.catalog.listCachedTables()`. Entries cached only via `Dataset.cache()` without a name are not included.
+
 Configuration of in-memory caching can be done via `spark.conf.set` or by running
 `SET key=value` commands using SQL.
 

docs/sql-ref-ansi-compliance.md

@@ -440,6 +440,7 @@ Below is a list of all the keywords in Spark SQL.
 |BY|non-reserved|non-reserved|reserved|
 |BYTE|non-reserved|non-reserved|non-reserved|
 |CACHE|non-reserved|non-reserved|non-reserved|
+|CACHED|non-reserved|non-reserved|non-reserved|
 |CALL|reserved|non-reserved|reserved|
 |CALLED|non-reserved|non-reserved|non-reserved|
 |CASCADE|non-reserved|non-reserved|non-reserved|

docs/sql-ref-syntax-aux-cache-cache-table.md

@@ -79,6 +79,7 @@ CACHE TABLE testCache OPTIONS ('storageLevel' 'DISK_ONLY') SELECT * FROM testDat
 
 ### Related Statements
 
+* [SHOW CACHED TABLES](sql-ref-syntax-aux-show-cached-tables.html)
 * [CLEAR CACHE](sql-ref-syntax-aux-cache-clear-cache.html)
 * [UNCACHE TABLE](sql-ref-syntax-aux-cache-uncache-table.html)
 * [REFRESH TABLE](sql-ref-syntax-aux-cache-refresh-table.html)

docs/sql-ref-syntax-aux-cache-uncache-table.md

@@ -49,6 +49,7 @@ UNCACHE TABLE t1;
 ### Related Statements
 
 * [CACHE TABLE](sql-ref-syntax-aux-cache-cache-table.html)
+* [SHOW CACHED TABLES](sql-ref-syntax-aux-show-cached-tables.html)
 * [CLEAR CACHE](sql-ref-syntax-aux-cache-clear-cache.html)
 * [REFRESH TABLE](sql-ref-syntax-aux-cache-refresh-table.html)
 * [REFRESH](sql-ref-syntax-aux-cache-refresh.html)

docs/sql-ref-syntax-aux-show-cached-tables.md

@@ -0,0 +1,51 @@
+---
+layout: global
+title: SHOW CACHED TABLES
+displayTitle: SHOW CACHED TABLES
+license: |
+  Licensed to the Apache Software Foundation (ASF) under one or more
+  contributor license agreements.  See the NOTICE file distributed with
+  this work for additional information regarding copyright ownership.
+  The ASF licenses this file to You under the Apache License, Version 2.0
+  (the "License"); you may not use this file except in compliance with
+  the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+---
+
+### Description
+
+The `SHOW CACHED TABLES` statement returns every in-memory cache entry that was registered with an explicit table or view name, for example via [`CACHE TABLE`](sql-ref-syntax-aux-cache-cache-table.html) or `spark.catalog.cacheTable`. The result has two columns: `tableName` (the name used when caching) and `storageLevel` (a string description of how the data is cached).
+
+Relations cached only through `Dataset.cache()` / `DataFrame.cache()` without assigning a catalog name are **not** listed.
+
+### Syntax
+
+```sql
+SHOW CACHED TABLES
+```
+
+### Examples
+
+```sql
+CACHE TABLE my_table AS SELECT * FROM src;
+
+SHOW CACHED TABLES;
++----------+--------------------------------------+
+| tableName|                          storageLevel|
++----------+--------------------------------------+
+|  my_table|Disk Memory Deserialized 1x Replicated|
++----------+--------------------------------------+
+```
+
+### Related Statements
+
+* [CACHE TABLE](sql-ref-syntax-aux-cache-cache-table.html)
+* [UNCACHE TABLE](sql-ref-syntax-aux-cache-uncache-table.html)
+* [CLEAR CACHE](sql-ref-syntax-aux-cache-clear-cache.html)

docs/sql-ref-syntax-aux-show.md

@@ -19,6 +19,7 @@ license: |
   limitations under the License.
 ---
 
+ * [SHOW CACHED TABLES](sql-ref-syntax-aux-show-cached-tables.html)
  * [SHOW COLUMNS](sql-ref-syntax-aux-show-columns.html)
  * [SHOW CREATE TABLE](sql-ref-syntax-aux-show-create-table.html)
  * [SHOW DATABASES](sql-ref-syntax-aux-show-databases.html)

docs/sql-ref-syntax.md

@@ -124,6 +124,7 @@ You use SQL scripting to execute procedural logic in SQL.
  * [RESET](sql-ref-syntax-aux-conf-mgmt-reset.html)
  * [SET](sql-ref-syntax-aux-conf-mgmt-set.html)
  * [SET VAR](sql-ref-syntax-aux-set-var.html)
+ * [SHOW CACHED TABLES](sql-ref-syntax-aux-show-cached-tables.html)
  * [SHOW COLUMNS](sql-ref-syntax-aux-show-columns.html)
  * [SHOW CREATE TABLE](sql-ref-syntax-aux-show-create-table.html)
  * [SHOW DATABASES](sql-ref-syntax-aux-show-databases.html)

project/MimaExcludes.scala

@@ -35,6 +35,10 @@ object MimaExcludes {
 
   // Exclude rules for 4.2.x from 4.1.0
   lazy val v42excludes = v41excludes ++ Seq(
+    // [SQL] SafeJsonSerializer.safeMapToJValue: second parameter widened from Function1 to
+    // Function2 so the key is passed to the value serializer (progress.scala). Binary-incompatible
+    // vs spark-sql-api 4.0.0; not part of the public supported API (private[streaming] package).
+    ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.streaming.SafeJsonSerializer.safeMapToJValue"),
     // Add DEBUG format to ErrorMessageFormat enum
     ProblemFilters.exclude[Problem]("org.apache.spark.ErrorMessageFormat*"),
     // [SPARK-47086][BUILD][CORE][WEBUI] Upgrade Jetty to 12.1.4

python/docs/source/reference/pyspark.sql/catalog.rst

@@ -25,30 +25,41 @@ Catalog
 .. autosummary::
     :toctree: api/
 
+    Catalog.analyzeTable
     Catalog.cacheTable
     Catalog.clearCache
+    Catalog.createDatabase
     Catalog.createExternalTable
     Catalog.createTable
     Catalog.currentCatalog
     Catalog.currentDatabase
     Catalog.databaseExists
+    Catalog.dropDatabase
     Catalog.dropGlobalTempView
+    Catalog.dropTable
     Catalog.dropTempView
+    Catalog.dropView
     Catalog.functionExists
+    Catalog.getCreateTableString
     Catalog.getDatabase
     Catalog.getFunction
     Catalog.getTable
+    Catalog.getTableProperties
     Catalog.isCached
+    Catalog.listCachedTables
     Catalog.listCatalogs
     Catalog.listColumns
     Catalog.listDatabases
     Catalog.listFunctions
+    Catalog.listPartitions
     Catalog.listTables
+    Catalog.listViews
     Catalog.recoverPartitions
     Catalog.refreshByPath
     Catalog.refreshTable
     Catalog.registerFunction
     Catalog.setCurrentCatalog
     Catalog.setCurrentDatabase
     Catalog.tableExists
+    Catalog.truncateTable
     Catalog.uncacheTable