如何在test中模拟Kafka broker? EmbeddedKafka如何使用？

最新推荐文章于 2024-10-07 15:55:46 发布

creepcheck

最新推荐文章于 2024-10-07 15:55:46 发布

阅读量2.6k

点赞数

CC 4.0 BY-SA版权

文章标签： kafka junit

本文链接：https://blog.csdn.net/creepcheck/article/details/106052569

本文探讨了在测试中如何使用EmbeddedKafkaRule和EmbeddedKafkaBroker来模拟Kafka broker。区别在于EmbeddedKafkaRule是JUnit4的TestRule，而EmbeddedKafkaBroker在Spring应用上下文中作为bean管理。遇到的问题是配置 EmbeddedKafkaBroker 时需要提供broker的属性，并且通过@EmbeddedKafka注解可以在@Configuration文件中启用Kafka监听器并注册bean。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

EmbeddedKafkaBroker

1. EmbeddedKafkaRule vs. EmbeddedKafkaBroker

EmbeddedKafkaRule: A TestRule wrapper around an EmbeddedKafkaBroker.
查看源码，发现他俩都是创建一些虚拟的broker给test使用的。

然后看到这样一段话，不过对我没有什么用：
TopicAlreadyExists exceptions:
Some suggestions:

Since you are using JUnit5, don’t use the JUnit4 EmbeddedKafkaRule, use EmbeddedKafkaBrokerinstead; or simply add @EmbeddedKafkaand the broker will be added as a bean to the Spring application context and its life cycle managed by Spring (use @DirtiesContext to destroy); for non-Spring tests, the broker will be created (and destroyed) by the JUnit5 EmbeddedKafkaConditionand is available via EmbeddedKafkaCondition.getBroker().
Don’t use explicit ports; let the broker use its default random port and use embeddedKafka.getBrokersAsString() for the bootstrap servers property.
If you must manage the brokers yourself (in @BeforeAll), destroy() them in @AfterAll

1.1 EmbeddedKafkaRule

Spring-kafka-test provides an embedded Kafka broker. We can use a JUnit @ClassRule annotation to create this Kafka broker. This rule starts the Kafka and Zookeeper servers on a random port before the tests execute and shuts them down after the tests are complete. The embedded Kafka broker eliminates the need to have a real Kafka and Zookeeper instance running while running the test.

官方文档: A JUnit 4 @Rule wrapper for the EmbeddedKafkaBroker is provided to create an embedded Kafka and an embedded Zookeeper server. (See @EmbeddedKafka Annotation for information about using @EmbeddedKafka with JUnit 5).

Starting with version 2.0, if you use Spring’s test application context caching, you can also declare a EmbeddedKafkaBroker bean, so a single broker can be used across multiple test classes. For convenience, we provide a test class-level annotation called @EmbeddedKafka to register the EmbeddedKafkaBroker bean. The following example shows how to use it:

@RunWith(SpringRunner.class)
@DirtiesContext
@EmbeddedKafka(partitions = 1,
         topics = {
                 KafkaStreamsTests.STREAMING_TOPIC1,
                 KafkaStreamsTests.STREAMING_TOPIC2 })
public class KafkaStreamsTests {

    @Autowired
    private EmbeddedKafkaBroker embeddedKafka;

    @Test
    public void someTest() {
        Map<String, Object> consumerProps = KafkaTestUtils.consumerProps("testGroup", "true", this.embeddedKafka);
        consumerProps.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        ConsumerFactory<Integer, String> cf = new DefaultKafkaConsumerFactory<>(consumerProps);
        Consumer<Integer, String> consumer = cf.createConsumer();
        this.embeddedKafka.consumeFromAnEmbeddedTopic(consumer, KafkaStreamsTests.STREAMING_TOPIC2);
        ConsumerRecords<Integer, String> replies = KafkaTestUtils.getRecords(consumer);
        assertThat(replies.count()).isGreaterThanOrEqualTo(1);
    }

    @Configuration
    @EnableKafkaStreams
    public static class KafkaStreamsConfiguration {

        @Value("${" + EmbeddedKafkaBroker.SPRING_EMBEDDED_KAFKA_BROKERS + "}")
        private String brokerAddresses;

        @Bean(name = KafkaStreamsDefaultConfiguration.DEFAULT_STREAMS_CONFIG_BEAN_NAME)
        public KafkaStreamsConfiguration kStreamsConfigs() {
            Map<String, Object> props = new HashMap<>();
            props.put(StreamsConfig.APPLICATION_ID_CONFIG, "testStreams");
            props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, this.brokerAddresses);
            return new KafkaStreamsConfiguration(props);
        }

    }

}

2. 我遇到的问题：

我并没有用@EmbeddedKafka，我是想自己注入一个embeddedkafkabroker bean的，本来我直接用：

private EmbeddedKafkaBroker embeddedKafkaBroker = new EmbeddedKafkaBroker(1, false, 3, "topic");

然后再config

Map<String, Object> props = new HashMap<>(KafkaTestUtils.consumerProps("groupId", "auto-commit: false", embeddedKafkaBroker));

发现不行，因为没有设置broker的properties，broker并没有被Inject。
所以得自己写一个embeddedKafkaBroker的函数，标注为bean，然后注入此test class。用到的是brokerProperties，有点类似这里.:

@Bean
public EmbeddedKafkaBroker embeddedKafkaBroker() {
    return new EmbeddedKafkaBroker(1,false,2,"test-events")
        .brokerProperties(Collections.singletonMap(KafkaConfig.LogDirProp(), "/tmp/foo"));
}