Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: apache/celeborn
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main@{1day}
Choose a base ref
...
head repository: apache/celeborn
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
  • 4 commits
  • 49 files changed
  • 3 contributors

Commits on May 30, 2025

  1. [CELEBORN-1413][FOLLOWUP] Support dependencies of spark-4.0 profile 

    ### What changes were proposed in this pull request?

Support dependencies of `spark-4.0` profile.

Follow up #3282.

### Why are the changes needed?

#3282 is lack of dependencies support of `spark-4.0` profile.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Dependencies check: maven-jdk17 (spark-4.0).

Closes #3298 from SteNicholas/CELEBORN-1413.

Authored-by: SteNicholas <[email protected]>
Signed-off-by: SteNicholas <[email protected]>
    @SteNicholas
    SteNicholas committed May 30, 2025
    Configuration menu
    Copy the full SHA
    3fb6d5b View commit details
    Browse the repository at this point in the history

  2. [CELEBORN-1627][FOLLOWUP] Fix the issue where the case of name affect… 

    …s the metrics dashboard

### What changes were proposed in this pull request?
Revert role name change in [CELEBORN-1627](#2777)

### Why are the changes needed?
Fix the issue where the case of name affects the metrics dashboard

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
Manual

Closes #3299 from RexXiong/CELEBORN-1627-FOLLOWUP.

Authored-by: Shuang <[email protected]>
Signed-off-by: Wang, Fei <[email protected]>
    @RexXiong @turboFei
    RexXiong authored and turboFei committed May 30, 2025
    Configuration menu
    Copy the full SHA
    0227a1a View commit details
    Browse the repository at this point in the history

  3. [CELEBORN-2005][FOLLOWUP] Introduce ShuffleMetricGroup for numBytesIn… 

    …, numBytesOut, numRecordsOut, numBytesInPerSecond, numBytesOutPerSecond, numRecordsOutPerSecond metrics

### What changes were proposed in this pull request?

Introduce `ShuffleMetricGroup` for `numBytesIn`, `numBytesOut`, `numRecordsOut`, `numBytesInPerSecond`, `numBytesOutPerSecond`, `numRecordsOutPerSecond` metrics.

Follow up #3272.

### Why are the changes needed?

`numBytesIn`, `numBytesOut`, `numRecordsOut`, `numBytesInPerSecond`, `numBytesOutPerSecond`, `numRecordsOutPerSecond` metrics should put shuffle id into variables, which could introduce `ShuffleMetricGroup` to support. Meanwhile, #3272 would print many same logs as follows that shoud be improved:

```
2025-05-28 10:48:54,433 WARN  [flink-akka.actor.default-dispatcher-18] org.apache.flink.metrics.MetricGroup                         [] - Name collision: Group already contains a Metric with the name 'numRecordsOut'. Metric will not be reported.[11.66.62.202, taskmanager, antc4flink3980005426-taskmanager-3-70, antc4flink3980005426, [vertex-2]HashJoin(joinType=[LeftOuterJoin], where=[(f0 = f00)], select=[f0, f1, f2, f3, f4, f5, f6, f00, f10, f20, f30, f40, f50, f60], build=[right]) -> Sink: Sink(table=[default_catalog.default_database.sink], fields=[f0, f1, f2, f3, f4, f5, f6, f00, f10, f20, f30, f40, f50, f60]), 2, Shuffle, Remote, 1]
```

### Does this PR introduce _any_ user-facing change?

Introduce `celeborn.client.flink.metrics.scope.shuffle` config option to define the scope format string that is applied to all metrics scoped to a shuffle:

- Variables:
   - Shuffle: `<task_id>, <task_name>, <task_attempt_id>, <task_attempt_num>, <subtask_index>, <shuffle_id>`.
- Metrics:

Scope | Metrics | Description | Type
-- | -- | -- | --
Shuffle | numBytesIn | The total number of bytes this shuffle has read. | Counter |
Shuffle | numBytesOut| The total number of bytes this shuffle has written. | Counter |
Shuffle | numRecordsOut | The total number of records this shuffle has written. | Counter |
Shuffle | numBytesInPerSecond | The number of bytes this shuffle reads per second. | Meter |
Shuffle | numBytesOutPerSecond | The number of bytes this shuffle writes per second. | Meter |
Shuffle | numRecordsOutPerSecond | The number of records this shuffle writes per second. | Meter |

### How was this patch tested?

Manual test.

![image](https://github.com/user-attachments/assets/a10c18ab-84f9-44f5-bb2d-e6b08e5bc64e)
![image](https://github.com/user-attachments/assets/0cb29c17-3388-4608-b7a4-ee7e3c9b43c1)

Closes #3296 from SteNicholas/CELEBORN-2005.

Authored-by: SteNicholas <[email protected]>
Signed-off-by: Weijie Guo <[email protected]>
    @SteNicholas @reswqa
    SteNicholas authored and reswqa committed May 30, 2025
    Configuration menu
    Copy the full SHA
    68a1db1 View commit details
    Browse the repository at this point in the history

  4. [CELEBORN-2020] Support http authentication for Celeborn CLI 

    ### What changes were proposed in this pull request?

Support http authentication for Celeborn CLI.

### Why are the changes needed?

Current CLI does not work if the authentication is enabled for master or worker.

### Does this PR introduce _any_ user-facing change?
Yes, a new option.

### How was this patch tested?

UT.

Closes #3300 from turboFei/cli_auth.

Authored-by: Wang, Fei <[email protected]>
Signed-off-by: Wang, Fei <[email protected]>
    @turboFei
    turboFei committed May 30, 2025
    Configuration menu
    Copy the full SHA
    5f58fb1 View commit details
    Browse the repository at this point in the history
Loading