Skip to content

Commit 2b22308

Browse files
aswinshakilaswinshakil
committed
Merge branch 'master' into HDDS-10239-container-reconciliation
COMMITS: 360fea5 HDDS-11494. Improve the duration option of freon ombg (apache#7246) 10d3b21 HDDS-11504. Update Ratis to 3.1.1. (apache#7257) ce46297 HDDS-11162. Improve Disk Usage page UI (apache#7214) c91f1c7 HDDS-11491. Avoid sharing clientId among deleting services (apache#7250) b0943d5 HDDS-11501. Improve logging in XceiverServerRatis (apache#7252) 55925ab HDDS-11502. Class path contains multiple SLF4J providers (apache#7255) d0ad836 HDDS-11472. Avoid recreating external access authorizer on OM state reload (apache#7238) 254db9e HDDS-11500. RootCARotationManager cancelling wrong task in notifyStatusChanged (apache#7251) 1e6e4b3 HDDS-11499. Remove redundant code from ECReconstructionCoordinator. (apache#7248) adb2821 HDDS-11490. Bump rollup to 3.29.5 (apache#7232) 189a9fe HDDS-11484. Validate javadoc in CI (apache#7245) 64a29c6 HDDS-11497. Bump commons-configuration2 to 2.11.0 (apache#7242) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snapshot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForPrefixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyInFailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snaps hot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForP refixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyIn FailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) e0060a8 HDDS-11196. Improve SCM WebUI Display (apache#6960) 22ddfb9 Revert "HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192)" 9f5bf43 HDDS-11457. Internal error on S3 CompleteMultipartUpload if parts are not specified (apache#7195) 10c47a1 HDDS-11459. Bump develocity-maven-extension to 1.22.1 (apache#7201) 50f2563 HDDS-11419. Fix waitForCheckpointDirectoryExist log message (apache#7199) a7d7e37 HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192) 5feb9ea HDDS-11453. OmSnapshotPurge should be in a different ozone manager double buffer batch (apache#7188) 703c4d5 HDDS-10984. Tool to restore SCM certificates from RocksDB. (apache#6781) d221065 HDDS-11440. Add a lastTransactionInfo field in SnapshotInfo to check for transactions in flight on the snapshot (apache#7179) e573701 HDDS-11448. Improve documentation in ContainerStateMachine (apache#7183) 0e49f7a HDDS-11449. Remove unnecessary log from client console. (apache#7184) cd251f2 HDDS-11438. Ensure DataInputBuffer is closed in OMPBHelper#convert (apache#7182) 4b47812 HDDS-11389. Incorrect number of deleted containers shown in Recon UI. (apache#7149) 0915f0b HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks. (apache#7009) 0f16195 HDDS-11416. refactor ratis submit request avoid code duplicate (apache#7166) 86fe920 HDDS-11376. Improve ReplicationSupervisor to record replication metrics (apache#7140) CONFLICT: Merge conflict in hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandlerWithUnhealthyContainer.java MODIFIED: /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/DNContainerOperationClient.java /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java
2 parents 9765a6a + 360fea5 commit 2b22308

File tree

440 files changed

+15001
-5905
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

440 files changed

+15001
-5905
lines changed

‎.github/workflows/ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,7 @@ jobs:
218218
distribution: 'temurin'
219219
java-version: ${{ matrix.java }}
220220
- name: Compile Ozone using Java ${{ matrix.java }}
221-
run: hadoop-ozone/dev-support/checks/build.sh -Pdist -Dskip.npx -Dskip.installnpx -Djavac.version=${{ matrix.java }} ${{ inputs.ratis_args }}
221+
run: hadoop-ozone/dev-support/checks/build.sh -Pdist -Dskip.npx -Dskip.installnpx -Dmaven.javadoc.failOnWarnings=${{ matrix.java != 8 }} -Djavac.version=${{ matrix.java }} ${{ inputs.ratis_args }}
222222
env:
223223
OZONE_WITH_COVERAGE: false
224224
DEVELOCITY_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}

‎.mvn/extensions.xml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
<extension>
2525
<groupId>com.gradle</groupId>
2626
<artifactId>develocity-maven-extension</artifactId>
27-
<version>1.21.5</version>
27+
<version>1.22.1</version>
2828
</extension>
2929
<extension>
3030
<groupId>com.gradle</groupId>

‎NOTICE.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
Apache Ozone
2-
Copyright 2022 The Apache Software Foundation
2+
Copyright 2024 The Apache Software Foundation
33

44
This product includes software developed at
55
The Apache Software Foundation (http://www.apache.org/).

‎dev-support/ci/selective_ci_checks.bats

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -429,3 +429,15 @@ load bats-assert/load.bash
429429
assert_output -p needs-integration-tests=false
430430
assert_output -p needs-kubernetes-tests=false
431431
}
432+
433+
@test "properties file in resources" {
434+
run dev-support/ci/selective_ci_checks.sh 71b8bdd8becf72d6f7d4e7986895504b8259b3e5
435+
436+
assert_output -p 'basic-checks=["rat","checkstyle","native"]'
437+
assert_output -p needs-build=false
438+
assert_output -p needs-compile=false
439+
assert_output -p needs-compose-tests=false
440+
assert_output -p needs-dependency-check=false
441+
assert_output -p needs-integration-tests=true
442+
assert_output -p needs-kubernetes-tests=false
443+
}

‎dev-support/ci/selective_ci_checks.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -373,6 +373,7 @@ function check_needs_checkstyle() {
373373
"^hadoop-hdds/dev-support/checkstyle"
374374
"pom.xml"
375375
"src/..../java"
376+
"src/..../resources/.*\.properties"
376377
)
377378
local ignore_array=(
378379
"^hadoop-ozone/dist"

‎dev-support/pom.xml

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License. See accompanying LICENSE file.
14+
-->
15+
<project xmlns="http://maven.apache.org/POM/4.0.0"
16+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
17+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
18+
<parent>
19+
<artifactId>ozone-main</artifactId>
20+
<groupId>org.apache.ozone</groupId>
21+
<version>1.5.0-SNAPSHOT</version>
22+
</parent>
23+
<modelVersion>4.0.0</modelVersion>
24+
<artifactId>ozone-dev-support</artifactId>
25+
<description>Helper module for sharing resources among projects</description>
26+
<name>Apache Ozone Dev Support</name>
27+
28+
<properties>
29+
<failIfNoTests>false</failIfNoTests>
30+
</properties>
31+
<build>
32+
<resources>
33+
<resource>
34+
<directory>${project.build.directory}/extra-resources</directory>
35+
<targetPath>META-INF</targetPath>
36+
<includes>
37+
<include>LICENSE.txt</include>
38+
<include>NOTICE.txt</include>
39+
</includes>
40+
</resource>
41+
</resources>
42+
<plugins>
43+
<plugin>
44+
<groupId>org.apache.maven.plugins</groupId>
45+
<artifactId>maven-site-plugin</artifactId>
46+
<configuration>
47+
<skip>true</skip>
48+
</configuration>
49+
</plugin>
50+
<!-- copy L&N files to target/extra-resources -->
51+
<plugin>
52+
<groupId>org.apache.maven.plugins</groupId>
53+
<artifactId>maven-resources-plugin</artifactId>
54+
<executions>
55+
<execution>
56+
<id>copy-resources</id>
57+
<phase>validate</phase>
58+
<goals>
59+
<goal>copy-resources</goal>
60+
</goals>
61+
<configuration>
62+
<outputDirectory>${project.build.directory}/extra-resources</outputDirectory>
63+
<resources>
64+
<resource>
65+
<directory>../</directory>
66+
<includes>
67+
<include>LICENSE.txt</include>
68+
<include>NOTICE.txt</include>
69+
</includes>
70+
</resource>
71+
</resources>
72+
</configuration>
73+
</execution>
74+
</executions>
75+
</plugin>
76+
<!-- add entries for L&N files to remote-resources.xml in jar file -->
77+
<plugin>
78+
<groupId>org.apache.maven.plugins</groupId>
79+
<artifactId>maven-remote-resources-plugin</artifactId>
80+
<executions>
81+
<execution>
82+
<phase>process-resources</phase>
83+
<goals>
84+
<goal>bundle</goal>
85+
</goals>
86+
</execution>
87+
</executions>
88+
<configuration>
89+
<resourcesDirectory>${project.build.outputDirectory}</resourcesDirectory>
90+
<includes>
91+
<include>META-INF/LICENSE.txt</include>
92+
<include>META-INF/NOTICE.txt</include>
93+
</includes>
94+
</configuration>
95+
</plugin>
96+
</plugins>
97+
</build>
98+
</project>

‎hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java

Lines changed: 65 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -247,28 +247,49 @@ public enum ChecksumCombineMode {
247247
tags = ConfigTag.CLIENT)
248248
private String fsDefaultBucketLayout = "FILE_SYSTEM_OPTIMIZED";
249249

250+
// ozone.client.hbase.enhancements.allowed
251+
@Config(key = "hbase.enhancements.allowed",
252+
defaultValue = "false",
253+
description = "When set to false, client-side HBase enhancement-related Ozone (experimental) features " +
254+
"are disabled (not allowed to be enabled) regardless of whether those configs are set.\n" +
255+
"\n" +
256+
"Here is the list of configs and values overridden when this config is set to false:\n" +
257+
"1. ozone.fs.hsync.enabled = false\n" +
258+
"2. ozone.client.incremental.chunk.list = false\n" +
259+
"3. ozone.client.stream.putblock.piggybacking = false\n" +
260+
"4. ozone.client.key.write.concurrency = 1\n" +
261+
"\n" +
262+
"A warning message will be printed if any of the above configs are overridden by this.",
263+
tags = ConfigTag.CLIENT)
264+
private boolean hbaseEnhancementsAllowed = false;
265+
266+
// ozone.client.incremental.chunk.list
250267
@Config(key = "incremental.chunk.list",
251-
defaultValue = "true",
268+
defaultValue = "false",
252269
type = ConfigType.BOOLEAN,
253270
description = "Client PutBlock request can choose incremental chunk " +
254271
"list rather than full chunk list to optimize performance. " +
255-
"Critical to HBase. EC does not support this feature.",
272+
"Critical to HBase. EC does not support this feature. " +
273+
"Can be enabled only when ozone.client.hbase.enhancements.allowed = true",
256274
tags = ConfigTag.CLIENT)
257-
private boolean incrementalChunkList = true;
275+
private boolean incrementalChunkList = false;
258276

277+
// ozone.client.stream.putblock.piggybacking
259278
@Config(key = "stream.putblock.piggybacking",
260-
defaultValue = "true",
279+
defaultValue = "false",
261280
type = ConfigType.BOOLEAN,
262-
description = "Allow PutBlock to be piggybacked in WriteChunk " +
263-
"requests if the chunk is small.",
281+
description = "Allow PutBlock to be piggybacked in WriteChunk requests if the chunk is small. " +
282+
"Can be enabled only when ozone.client.hbase.enhancements.allowed = true",
264283
tags = ConfigTag.CLIENT)
265-
private boolean enablePutblockPiggybacking = true;
284+
private boolean enablePutblockPiggybacking = false;
266285

286+
// ozone.client.key.write.concurrency
267287
@Config(key = "key.write.concurrency",
268288
defaultValue = "1",
269289
description = "Maximum concurrent writes allowed on each key. " +
270290
"Defaults to 1 which matches the behavior before HDDS-9844. " +
271-
"For unlimited write concurrency, set this to -1 or any negative integer value.",
291+
"For unlimited write concurrency, set this to -1 or any negative integer value. " +
292+
"Any value other than 1 is effective only when ozone.client.hbase.enhancements.allowed = true",
272293
tags = ConfigTag.CLIENT)
273294
private int maxConcurrentWritePerKey = 1;
274295

@@ -298,6 +319,34 @@ public void validate() {
298319
OzoneConfigKeys.OZONE_CLIENT_BYTES_PER_CHECKSUM_MIN_SIZE;
299320
}
300321

322+
// Verify client configs related to HBase enhancements
323+
// Enforce check on ozone.client.hbase.enhancements.allowed
324+
if (!hbaseEnhancementsAllowed) {
325+
// ozone.client.hbase.enhancements.allowed = false
326+
if (incrementalChunkList) {
327+
LOG.warn("Ignoring ozone.client.incremental.chunk.list = true " +
328+
"because HBase enhancements are disallowed. " +
329+
"To enable it, set ozone.client.hbase.enhancements.allowed = true.");
330+
incrementalChunkList = false;
331+
LOG.debug("Final ozone.client.incremental.chunk.list = {}", incrementalChunkList);
332+
}
333+
if (enablePutblockPiggybacking) {
334+
LOG.warn("Ignoring ozone.client.stream.putblock.piggybacking = true " +
335+
"because HBase enhancements are disallowed. " +
336+
"To enable it, set ozone.client.hbase.enhancements.allowed = true.");
337+
enablePutblockPiggybacking = false;
338+
LOG.debug("Final ozone.client.stream.putblock.piggybacking = {}", enablePutblockPiggybacking);
339+
}
340+
if (maxConcurrentWritePerKey != 1) {
341+
LOG.warn("Ignoring ozone.client.key.write.concurrency = {} " +
342+
"because HBase enhancements are disallowed. " +
343+
"To enable it, set ozone.client.hbase.enhancements.allowed = true.",
344+
maxConcurrentWritePerKey);
345+
maxConcurrentWritePerKey = 1;
346+
LOG.debug("Final ozone.client.key.write.concurrency = {}", maxConcurrentWritePerKey);
347+
}
348+
// Note: ozone.fs.hsync.enabled is enforced by OzoneFSUtils#canEnableHsync, not here
349+
}
301350
}
302351

303352
public long getStreamBufferFlushSize() {
@@ -486,6 +535,14 @@ public void setDatastreamPipelineMode(boolean datastreamPipelineMode) {
486535
this.datastreamPipelineMode = datastreamPipelineMode;
487536
}
488537

538+
public void setHBaseEnhancementsAllowed(boolean isHBaseEnhancementsEnabled) {
539+
this.hbaseEnhancementsAllowed = isHBaseEnhancementsEnabled;
540+
}
541+
542+
public boolean getHBaseEnhancementsAllowed() {
543+
return this.hbaseEnhancementsAllowed;
544+
}
545+
489546
public void setIncrementalChunkList(boolean enable) {
490547
this.incrementalChunkList = enable;
491548
}

‎hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/AbstractCommitWatcher.java

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,6 @@ XceiverClientReply watchOnLastIndex() throws IOException {
124124
*
125125
* @param commitIndex log index to watch for
126126
* @return minimum commit index replicated to all nodes
127-
* @throws IOException IOException in case watch gets timed out
128127
*/
129128
CompletableFuture<XceiverClientReply> watchForCommitAsync(long commitIndex) {
130129
final MemoizedSupplier<CompletableFuture<XceiverClientReply>> supplier

‎hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -364,7 +364,6 @@ public void writeOnRetry(long len) throws IOException {
364364
* it is a no op.
365365
* @param bufferFull flag indicating whether bufferFull condition is hit or
366366
* its called as part flush/close
367-
* @return minimum commit index replicated to all nodes
368367
* @throws IOException IOException in case watch gets timed out
369368
*/
370369
public void watchForCommit(boolean bufferFull) throws IOException {

0 commit comments

Comments
 (0)