|
| 1 | +# Apache Spark |
| 2 | + |
| 3 | +## Pre-flight Checks |
| 4 | + |
| 5 | +Before the first code edit or running test in a session, ensure a clean working environment. DO NOT skip these checks: |
| 6 | + |
| 7 | +1. Run `git remote -v` to identify the personal fork and upstream (`apache/spark`). If unclear, ask the user to configure their remotes following the standard convention (`origin` for the fork, `upstream` for `apache/spark`). |
| 8 | +2. If the latest commit on `<upstream>/master` is more than a day old (check with `git log -1 --format="%ci" <upstream>/master`), run `git fetch <upstream> master`. |
| 9 | +3. If there are uncommitted changes (check with `git status`), ask the user to stash them before proceeding. |
| 10 | +4. Switch to the appropriate branch: |
| 11 | + - **Existing PR**: resolve the PR branch name via `gh api repos/apache/spark/pulls/<number> --jq '.head.ref'`, then look for a local branch matching that name. If found, switch to it and inform the user. If not found, ask whether to fetch it or if there is a local branch under a different name. |
| 12 | + - **New edits**: ask the user to choose: create a new git worktree from `<upstream>/master` and work from there (recommended), or create and switch to a new branch from `<upstream>/master`. |
| 13 | + - **Running tests**: use `<upstream>/master`. |
| 14 | + |
| 15 | +## Development Notes |
| 16 | + |
| 17 | +SQL golden file tests are managed by `SQLQueryTestSuite` and its variants. Read the class documentation before running or updating these tests. DO NOT edit the generated golden files (`.sql.out`) directly. Always regenerate them when needed, and carefully review the diff to make sure it's expected. |
| 18 | + |
| 19 | +Spark Connect protocol is defined in proto files under `sql/connect/common/src/main/protobuf/`. Read the README there before modifying proto definitions. |
| 20 | + |
| 21 | +Avoid introducing non-ASCII characters in code or comments. String literals may contain non-ASCII when the content requires it (error messages, test data, etc.). Identifiers are ASCII by convention. The common failure mode is typographic characters (em-dash, smart quotes, ellipsis, non-breaking space) sneaking into comments; scalastyle flags some of these. Spot-check before committing: `grep -rn -P "[^\x00-\x7F]" <files>`. |
| 22 | + |
| 23 | +## Scala Test Base Classes |
| 24 | + |
| 25 | +When writing a new Scala test suite, pick the lowest base class that provides what the test actually needs. Spark uses the `AnyFunSuite` ScalaTest style throughout, so the bases below are the chain to choose from. Each adds capability on top of the previous: |
| 26 | + |
| 27 | + SparkFunSuite (core) |
| 28 | + <- PlanTest (sql/catalyst) |
| 29 | + <- QueryTest (sql/core) |
| 30 | + |
| 31 | +| Test scope | Base | Notes | |
| 32 | +|------------|------|-------| |
| 33 | +| Plain JVM/Scala — no Spark SQL | `SparkFunSuite` | `core` utilities, RDD, network, util classes, etc. Adds per-test timeout, `testRetry`, `gridTest`, thread audit, fixed timezone/locale, `withTempDir`, `withLogAppender`, `checkError`. | |
| 34 | +| Catalyst plan tests — no `SparkSession` | `PlanTest` | Adds `comparePlans`, `normalizePlan`, `normalizeExprIds`. For analyzer / optimizer / planner rule tests. | |
| 35 | +| SQL/DataFrame tests — needs a `SparkSession` | `QueryTest` | Adds `checkAnswer`, codegen-on/off helpers. `spark: SparkSession` is abstract and must be supplied by a session-providing trait (see below). | |
| 36 | + |
| 37 | +### Providing a `SparkSession` for `QueryTest` |
| 38 | + |
| 39 | +`QueryTest` declares `spark: SparkSession` abstractly via `SparkSessionProvider`, so it cannot be instantiated on its own. A concrete suite mixes in one of the session-providing traits below: |
| 40 | + |
| 41 | + QueryTest (abstract `spark`) |
| 42 | + + SharedSparkSession (sql/core) -> classic in-process `TestSparkSession` |
| 43 | + + TestHiveSingleton (sql/hive) -> Hive-backed `TestHive` session |
| 44 | + |
| 45 | +| Session provider | Module / location | Typical usage | |
| 46 | +|---|---|---| |
| 47 | +| `SharedSparkSession` | `sql/core` | Already extends `QueryTest` for historical reasons, but still mix in `QueryTest` explicitly, e.g. `class X extends QueryTest with SharedSparkSession`. Default for tests under `sql/core`. | |
| 48 | +| `TestHiveSingleton` | `sql/hive` | Mixed in alongside `QueryTest`, e.g. `class X extends QueryTest with TestHiveSingleton`. Used by tests under `sql/hive`. | |
| 49 | + |
| 50 | +## Build and Test |
| 51 | + |
| 52 | +Build and tests can take a long time. If the user explicitly asked to run tests, run them. Otherwise (you are running tests on your own to verify a change), first ask the user if they have more changes to make. |
| 53 | + |
| 54 | +Prefer SBT over Maven for faster incremental compilation. Module names are defined in `project/SparkBuild.scala`. |
| 55 | + |
| 56 | +Compile a single module: |
| 57 | + |
| 58 | + build/sbt <module>/compile |
| 59 | + |
| 60 | +Compile test code for a single module: |
| 61 | + |
| 62 | + build/sbt <module>/Test/compile |
| 63 | + |
| 64 | +Run test suites by wildcard or full class name: |
| 65 | + |
| 66 | + build/sbt '<module>/testOnly *MySuite' |
| 67 | + build/sbt '<module>/testOnly org.apache.spark.sql.MySuite' |
| 68 | + |
| 69 | +Run test cases matching a substring: |
| 70 | + |
| 71 | + build/sbt '<module>/testOnly *MySuite -- -z "test name"' |
| 72 | + |
| 73 | +For faster iteration, keep SBT open in interactive mode: |
| 74 | + |
| 75 | + build/sbt |
| 76 | + > project <module> |
| 77 | + > testOnly *MySuite |
| 78 | + |
| 79 | +### PySpark Tests |
| 80 | + |
| 81 | +PySpark tests require building Spark with Hive support first: |
| 82 | + |
| 83 | + build/sbt -Phive package |
| 84 | + |
| 85 | +Activate the virtual environment specified by the user, or default to `.venv`: |
| 86 | + |
| 87 | + source <venv>/bin/activate |
| 88 | + |
| 89 | +If the default venv does not exist, create it: |
| 90 | + |
| 91 | + python3 -m venv .venv |
| 92 | + source .venv/bin/activate |
| 93 | + pip install -r dev/requirements.txt |
| 94 | + |
| 95 | +Run a single test suite: |
| 96 | + |
| 97 | + python/run-tests --testnames pyspark.sql.tests.arrow.test_arrow |
| 98 | + |
| 99 | +Run a single test case: |
| 100 | + |
| 101 | + python/run-tests --testnames "pyspark.sql.tests.test_catalog CatalogTests.test_current_database" |
| 102 | + |
| 103 | +## Investigating PR CI Failures |
| 104 | + |
| 105 | +Do NOT download full job logs to grep for errors — they are very large and slow. Instead, use the test report annotations on the fork. |
| 106 | + |
| 107 | +Step 1 — Get the fork owner and the latest commit SHA of the PR: |
| 108 | + |
| 109 | + gh api repos/apache/spark/pulls/<PR_NUMBER> --jq '{owner: .head.repo.owner.login, sha: .head.sha}' |
| 110 | + |
| 111 | +Step 2 — Find the "Report test results" check run on the fork's commit: |
| 112 | + |
| 113 | + gh api repos/<OWNER>/spark/commits/<SHA>/check-runs \ |
| 114 | + --jq '.check_runs[] | select(.name == "Report test results") | {id: .id, annotations: .output.annotations_count}' |
| 115 | + |
| 116 | +Step 3 — Fetch failure annotations: |
| 117 | + |
| 118 | + gh api repos/<OWNER>/spark/check-runs/<CHECK_RUN_ID>/annotations |
| 119 | + |
| 120 | +Each annotation contains the test class, test name, and failure message. |
| 121 | + |
| 122 | +## Pull Request Workflow |
| 123 | + |
| 124 | +PR title format is `[SPARK-xxxx][COMPONENT] Title`. The component tag is derived from the JIRA component name: take the last word and uppercase it (e.g. `Project Infra` → `[INFRA]`, `Spark Core` → `[CORE]`, `Structured Streaming` → `[STREAMING]`, `SQL` → `[SQL]`). |
| 125 | + |
| 126 | +Infer the PR title from the changes. If no ticket ID is given, create one using `dev/create_spark_jira.py`, using the PR title (without the JIRA ID and component tag) as the ticket title. |
| 127 | + |
| 128 | + python3 dev/create_spark_jira.py "<title>" -c <component> { -t <type> | -p <parent-jira-id> } |
| 129 | + |
| 130 | +- **Component** (`-c`): the exact JIRA component name (not the PR title shorthand), e.g. "SQL", "Spark Core", "PySpark", "Connect". Run `python3 dev/create_spark_jira.py --list-components` for the full list. |
| 131 | +- **Issue type** (`-t`): "Bug", "Improvement", "New Feature", "Test", "Documentation", or "Dependency upgrade". |
| 132 | +- **Parent** (`-p`): if the user mentions a parent JIRA ticket (e.g., "this is a subtask of SPARK-12345"), pass it instead of `-t`. The issue type is automatically "Sub-task". |
| 133 | + |
| 134 | +The script sets the latest unreleased version as the default affected version. |
| 135 | + |
| 136 | +After creating a JIRA ticket, print a prominent notice so the user does not miss it: |
| 137 | + |
| 138 | + ============================================================ |
| 139 | + JIRA ticket created: SPARK-XXXXX |
| 140 | + https://issues.apache.org/jira/browse/SPARK-XXXXX |
| 141 | + |
| 142 | + Title: <title> |
| 143 | + Component(s): <component> |
| 144 | + Issue type: <type> |
| 145 | + Affected version(s): <version> |
| 146 | + Priority: <priority> |
| 147 | + |
| 148 | + Please review and adjust these fields if needed. |
| 149 | + ============================================================ |
| 150 | + |
| 151 | +Before writing the PR description, read `.github/PULL_REQUEST_TEMPLATE` and fill in every section from that file. |
| 152 | + |
| 153 | +DO NOT push to the upstream repo. Always push to the personal fork. Open PRs against `master` on the upstream repo. |
| 154 | + |
| 155 | +DO NOT force push or use `--amend` on pushed commits unless the user explicitly asks. If the remote branch has new commits, fetch and rebase before pushing. |
| 156 | + |
| 157 | +Always get user approval before external operations such as pushing commits, creating PRs, or posting comments. Use `gh pr create` to open PRs. If `gh` is not installed, generate the GitHub PR URL for the user and recommend installing the GitHub CLI. |
| 158 | + |
| 159 | +## Security |
| 160 | + |
| 161 | +Security model: [SECURITY.md](./SECURITY.md) |
| 162 | + |
| 163 | +Agents that scan this repository should consult `SECURITY.md` for the project's threat model, in-scope / out-of-scope declarations, and known non-findings before reporting issues. |
0 commit comments