What's Changed

Breaking Changes

New Features

Support yaml partially (#262)

• Now you can pass YAML file to decaton with central dogma. See https://github.com/line/decaton/blob/v9.4.0/docs/dynamic-property-configuration.adoc.

How to Migrate Dynamic Property Configuration in JSON to one in YAML

It's optional. If you want to use YAML instead of JSON, you can migrate with these steps

1. Create a YAML file from the JSON file, and store them to Central Dogma.
2. Upgrade Decaton from 9.y.z to 9.4.0 and change your decaton processors to read the YAML file retrieved from Central Dogma instead of JSON.
   Just changing file path that is passed to CentralDogmaPropertySupplier is enough.
3. (2) is completed, you can delete the unnecessary JSON files.

Note you should care inconsistency between JSON and YAML files until (2) is completed.

Bug Fixes

Improvements

Other Changes

Full Changelog: v9.3.0...v9.4.0

Breaking Changes

New Features

Allow build processors using kafka-clients Deserializer (#267)

• ProcessorsBuilder#consuming now have an overload that accepts kafka.common.serialization.Deserializer<T>, which can be used to deserialize tasks using topic, record headers and custom TaskMetadata extraction logic is not necessary

Support JSON schema for decaton processor properties used in central dogma (#263)

• See https://github.com/line/decaton/blob/v9.3.0/docs/dynamic-property-configuration.adoc#json-schema

Bug Fixes

Improvements

Other Changes

Full Changelog: v9.2.1...v9.3.0

Breaking Changes

New Features

Bug Fixes

• PerKeyQuota usage is double counted fixes #265 (#266)

Improvements

Other Changes

Breaking Changes

New Features

• Add a metric for v9 upgrade (#264)
  • The new metric decaton.records.consumed is helpful on v9 migration to check if all DecatonClient are producing with new format

Bug Fixes

• Fix CentralDogmaPropertySupplier#setValue (#257)

Improvements

Other Changes

• Update release doc (#255)
• Run unintentionally ignored tests (#260)
• Fix comment (#261)

Breaking Changes

New Features

Bug Fixes

• Fix decaton.processor.processed.time to measure task processing time (#253)

Improvements

Other Changes

• OSSRH is deprecated (#254)

Breaking Changes

New Features

Bug Fixes

• Fix task parse failure during migration to v9 fixes #250 (#251)

Improvements

Other Changes

Breaking Changes

New Features

• Put back support to produce in the old DecatonTaskRequest for migration #248

Bug Fixes

Improvements

Other Changes

Breaking Changes

New Features

Bug Fixes

Improvements

• Lower log level of partition processor core creation, to be virtual thread friendly #245

Other Changes

Breaking Changes

• TaskMetadata as header (#238)
  • decaton-client:
    • DecatonClient no longer wraps tasks with DecatonTaskRequest-protobuf
  • decaton-processor:
    • DecatonTaskRequest is moved to com.linecorp.decaton.protocol.internal and marked as deprecated
    • Retry tasks are no longer produced in DecatonTaskRequest format unless decaton.retry.task.in.legacy.format is set to true
    • KafkaProducerSupplier signature change
    • TaskExtractor signature change
  • Please read below upgrade guide carefully!!!
• Make Decaton can consume any topic with deserializer (#241)
  • decaton-processor:
    • ProcessorsBuilder#consuming(String topic, Deserializer<T> deserializer) no longer parses messages as DecatonTaskRequest-protobuf at all, unless decaton.legacy.parse.fallback.enabled
  • Please read below upgrade guide carefully!!!

New Features

Bug Fixes

Improvements

Other Changes

• Fix potential key exposure in PKQ (#237)
• Add task batching doc (#150)
• Update docs and examples for 9.0.0 (#239)

Upgrade guide from 8.0.1 or earlier

• As described in "Breaking Changes" section, Decaton made two big breaking changes in this release.
• You MUST read below guide carefully and follow the instructions, or your application may encounter deserialization errors and fail to process tasks.

Depending on your use cases, there are 4 different upgrade paths.

Case A: No retry queueing && non-DecatonClient

In this case, you can upgrade as usual without any special care.

• Step1: Upgrade decaton-processors to 9.0.0 or higher.
• That's it.

Case B: No retry queueing && DecatonClient

• Step1: Upgrade decaton-processors to 9.0.0 or higher with decaton.legacy.parse.fallback.enabled=true
• Step2: Upgrade decaton-clients to 9.0.0 or higher
• Step3: Wait all old-format tasks are processed after Step2.
  • You can monitor topic's committed offset is caught-up
• Step4: Set decaton.legacy.parse.fallback.enabled=false

Case C: Retry queueing && non-DecatonClient

• Step1: Upgrade decaton-processors to 9.0.0 or higher with decaton.retry.task.in.legacy.format=true, decaton.legacy.parse.fallback.enabled=true
• Step2: After all decaton-processors are upgraded, set decaton.retry.task.in.legacy.format=false
• Step3: Wait all retry tasks are processed after Step2.
  • You can monitor retry-topic's offset lag becomes 0.
• Step4: Set decaton.legacy.parse.fallback.enabled=false

Case D: Retry queueing && DecatonClient

• Step1: Upgrade decaton-processors to 9.0.0 or higher with decaton.retry.task.in.legacy.format=true, decaton.legacy.parse.fallback.enabled=true
• Step2: Upgrade decaton-clients to 9.0.0 or higher
• Step3: After all decaton-processors are upgraded, set decaton.retry.task.in.legacy.format=false
• Step4: Wait all old-format tasks and retry tasks are processed after Step3.
  • You can monitor topic's committed offset is caught-up and retry-topic's offset lag becomes 0.
• Step5: Set decaton.legacy.parse.fallback.enabled=false

EDIT 2025.01.29

In addition to the above migration procedure, if you are using DecatonClient (decaton-client) in your application, which also runs a ProcessorSubscription (decaton-processor module) AND the tasks produced by the same app are coming back to the processor of the same app (i.e, looping-back), by you upgrading decaton-processor dep it also pulls a newer decaton-client dep which no longer produces tasks in the deprecated DecatonTaskRequest format.
Under this situation we also need to keep the DecatonClient producing tasks in the old format until all destination processors are upgraded and reasy to parse thew new header-based metadata, and for this case we've recently added a new flag to DecatonClientBuilder to instruct it keep using the deprecated format - produceInOldTaskRequest.
Please make sure to set the parameter if your application matches the above case.
See also #248.

DecatonClient.producing(topic, new ProtocolBuffersSerializer<HelloTask>())
                        ...
                       .produceInOldTaskRequest(true) // This will make the resulting DecatonClient to keep using the old task format
                       .build();

Breaking Changes

New Features

Bug Fixes

• Use TopicPartition.topic for metrics #235

Improvements

Other Changes

• Bump dependencies (micrometer/tracing, spring-boot, central-dogma) #233
• Fix ResourceTracker in benchmark to work fine with VIRTUAL_THREAD #234