2024 April APAC Community Sync

[ChenSammi]

April 10th
Attender:Hao, Mingyu, Yiyang, Xi, Hongbing, Jianghua, Hualong, Yuanben, Conway, Wei-Chiu, Sammi

Qihoo
- Plan to use NVME as the read/write cache in each DN.
- Shared the POC code about how to reduce multiple serialization and deserialization of OMRequest object. Will submit patch later.
Cloudera
- Update of Hbase on Ozone progress
- Short circuit read support in Ozone design https://issues.apache.org/jira/browse/HDDS-10685.
Shopee
- 1.4.1 release is discussed and kicked off in the community.
- Looking forward the performance data of gRPC zero copy improvement.
- Current OM table cache uses the desalinized data as key and value. To reduce unnecessary serialization and deserialization of objects when accessing table cache, a new type of cache which holding Java object is suggested.
DiDi
- OM long GC (>60s) doesn't appear again after tuning the CMS GC parameters.
- EC data writing is a little bit slow. Suggested apply following two patches
https://issues.apache.org/jira/browse/HDDS-9911
https://issues.apache.org/jira/browse/HDDS-9912
- Tested and shared the follow read performance. https://docs.google.com/document/d/1xVkaQYDXJmztETJVZQHkkij_j8j6MGQ4XB8ehathhG8/edit#heading=h.r9ym4lnx88xm

[xichen01]

• Current OM table cache uses the desalinized data as key and value. To reduce unnecessary serialization and deserialization of objects when accessing table cache, a new type of cache which holding Java object is suggested.

About the protobuf conversion in the table cache

In fact, Ozone's CacheVlaue currently caches Java type objects instead of protobuf objects. However, when using cache, some types of objects require protobuf conversion, which consumes system resources.

For example, when OmBucketInfo executes getBucketTable().get(bucketKey) to obtain the cache, it will deep copy from
OmBucketInfo (java Object) -> BucketInfo (proto Object) -> OmBucketInfo (java Object) in the method DelegatedCode#copyObject

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DelegatedCodec.java

Lines 117 to 123 in 8795bcb

	// Deep copy
	try {
	return forward.apply(delegate.copyObject(backward.apply(message)));
	} catch (IOException e) {
	throw new IllegalStateException("Failed to copyObject", e);
	}
	}

---

About the DelegatedCodec#copyObject deep copy and shallow copy

However, I saw that some types implement interface CopyObject such as OMKeyInfo, so in DelegatedCodec#copyObject, OMKeyInfo only needs to call copyObject to make a shallow copy, without protobuf conversion.

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/DelegatedCodec.java

Lines 113 to 116 in 8795bcb

	final CopyObject<T> casted = ((CopyObject<T>) message);
	return casted.copyObject();
	}

@szetszwo
May I know why we did not let the OmBucketInfo implement interface CopyObject but chose to use serialization into protobuf type to implement deep copy? Is this due to type safety issues?

And OMKeyInfo#copyObject is just a shallow copy. Will this not cause thread safety issues? For example, for OmKeyInfo#acls, is doubleBuffer absolutely safe when calling OmKeyInfo#addAcl?

[szetszwo]

... why we did not let the OmBucketInfo implement interface CopyObject ...

This is a good catch. It is better to have OmBucketInfo implementing CopyObject .

And OMKeyInfo#copyObject is just a shallow copy. Will this not cause thread safety issues? ...

It seems a bug. OmKeyInfo.acls is just an ArrayList. It definitely is not thread safe.

[adoroszlai]

OzoneAcl is immutable (HDDS-10388), so copying only the acls list is safe.

[xichen01]

@adoroszlai
Currently, however, we don't explicitly declare classes that need to be immutable in the class name or class comments to prevent immutable from being broken.

Maybe we can use this annotation https://spotbugs.readthedocs.io/en/stable/detectors.html#checkimmutableannotation ?

[adoroszlai]

@xichen01 good idea, created HDDS-10741

[szetszwo]

... It is better to have OmBucketInfo implementing CopyObject .

Filed HDDS-10752.

... OmKeyInfo.acls is just an ArrayList. It definitely is not thread safe.

Filed HDDS-10753.

[ChenSammi]

April 26th
Attender:Hao, Mingyu, Yiyang, Xi, Hongbing, Hualong, Yuanben, Conway, leechungen, Wei-Chiu, Sammi

Mingyu

• Observer SCM GC pressure(>5s) which cause leader to transfer. It could be related with Container Balancer. With HDDS-10160. Cache sort results in ContainerBalancerSelectionCriteria, the container selection performance has great improvement. Now in every iterator, whole container info will be copied for analysis, which might lead to high GC pressure, when container count is more than 1m.

Xi

• AWS S3 Lifecycle Configurations is updated with design document HDDS-8342 . Review and feedback is welcome.

Hongbing

• After last time OM GC parameter tuning, the long GC(>60s) disappeared for a while. But this week, it happen again. Currently OM max heap is 128G. Will change ratis server close threshold from current 60s to 120s as a workaround. And will try to get heap dump to further analysis which are the tops objects in the OM heap.

[ivandika3]

AWS S3 Lifecycle Configurations is updated with design document HDDS-8342 . Review and feedback is welcome.

A design PR has been created #6589, all feedbacks are welcome.