Creation and management of Apache Pulsar partition topics

This article details the two main methods of creating partitioned topics in Apache Pulsar: automatic partitioned topic creation by configuring the Broker, and manual creation of partitioned topics using the Pulsar Admin API. It is emphasized that the partition topic must be specified at the time of creation. Once a topic is created as a non-partition type, it cannot be subsequently converted to a partition topic. This is crucial for system design and planning.

Understanding Apache Pulsar Topic Partitioning

Apache Pulsar's topic partitioning is a key mechanism for achieving high throughput and scalability. A partitioned topic is logically divided into multiple independent "partitions", each of which can process messages independently. Producers can send messages to specific partitions or automatically distributed by Pulsar, and each consumer instance in the consumer group can consume one or more partitions, thereby achieving parallel consumption. Proper planning and creation of partition topics is critical to building high-performance Pulsar applications.

In Pulsar, a topic's partition properties are determined when it is created, and once created, cannot be changed. This means that a non-partitioned topic cannot be converted to a partitioned topic in a subsequent operation. Therefore, when designing the Pulsar messaging system, it is crucial to determine in advance whether the topic needs partitions and the number of partitions.

How to create a partition topic

There are two main strategies for creating partition topics: automatic creation through Broker configuration, or manual creation using the Pulsar Admin API.

1. Automatic partition topic creation through Broker configuration

When a producer or consumer attempts to connect to a topic that does not yet exist, Pulsar Broker can automatically create the topic based on its configuration. By adjusting the Broker configuration, you can ensure that the automatically created topic is the partition topic by default.

The following are the relevant Broker configuration parameters:

• allowAutoTopicCreationType: This parameter controls the type of automatic topic creation. Setting it to partitioned means that all automatically created topics will be partitioned topics.
• defaultNumPartitions: This parameter specifies the default number of partitions for automatically created partition topics when allowAutoTopicCreationType is set to partitioned.

Configuration example:

Add or modify the following lines in Pulsar Broker's configuration file (usually conf/broker.conf):

 allowAutoTopicCreationType = partitioned
defaultNumPartitions = 5

Things to note:

• This method is suitable for scenarios that rely on Pulsar to automatically create themes. When your Java application first connects to a non-existent topic through PulsarClient, the topic will be automatically created with the specified number of partitions if the Broker is configured with the above parameters.
• This global configuration affects all automatically created themes. If you need to set a different number of partitions for a specific topic, or some topics do not require partitions, you need to create them manually.

2. Use Pulsar Admin API to manually create partition topics

For scenarios that require finer control, or when you do not want to rely on the automatic creation mechanism, you can use the Pulsar Admin API to manually create partition topics. The Pulsar Admin API provides a programming interface to manage various resources in the Pulsar cluster, including topics.

The Pulsar Admin client provides the topics().createPartitionedTopic(topicName, numPartitions) method to create a topic with a specified number of partitions.

Conceptual Java code example:

 import org.apache.pulsar.client.admin.PulsarAdmin;
import org.apache.pulsar.client.admin.PulsarAdminException;
import org.apache.pulsar.client.api.PulsarClientException;

public class PartitionedTopicCreator {

    private static final String BROKER_SERVICE_URL = "http://localhost:8080"; // Pulsar Admin Service URL

    public static void main(String[] args) {
        PulsarAdmin pulsarAdmin = null;
        try {
            //Initialize Pulsar Admin client pulsarAdmin = PulsarAdmin.builder()
                    .serviceHttpUrl(BROKER_SERVICE_URL)
                    .build();

            String topicName = "persistent://public/default/my-partitioned-topic";
            int numPartitions = 3; // Expected number of partitions // Check if the topic already exists to avoid repeated creation if (!pulsarAdmin.topics().getPartitionedTopicMetadata(topicName).getPartitions().isEmpty()) {
                System.out.println("Topic" topicName " already exists and is partitioned.");
            } else {
                // Create a partition topic pulsarAdmin.topics().createPartitionedTopic(topicName, numPartitions);
                System.out.println("Partition topic successfully created: " topicName ", number of partitions: " numPartitions);
            }

        } catch (PulsarClientException e) {
            System.err.println("Pulsar client exception: " e.getMessage());
            e.printStackTrace();
        } catch (PulsarAdminException e) {
            System.err.println("Pulsar Admin exception: " e.getMessage());
            e.printStackTrace();
        } finally {
            if (pulsarAdmin != null) {
                try {
                    pulsarAdmin.close();
                } catch (PulsarAdminException e) {
                    System.err.println("Failed to close Pulsar Admin client: " e.getMessage());
                }
            }
        }
    }
}

illustrate:

• PulsarAdmin.builder().serviceHttpUrl(...) is used to build the Pulsar Admin client instance.
• pulsarAdmin.topics().createPartitionedTopic(topicName, numPartitions) is the core method used to create a topic with a specified name and number of partitions.
• It is a good practice to check if the topic exists before creating it to avoid PulsarAdminException.

Key limitation: Immutability of partitioned topics

As highlighted earlier, once a topic is created as a non-partitioned topic, it cannot be converted to a partitioned topic. This is a core design principle of Pulsar. If you initially create a test topic without partitioning (for example, via pulsarClient.newProducer().topic("test").create() and the Broker is not configured for automatic partitioning), then the test topic will always be single-partitioned.

If subsequent business needs require the test topic to have partitioning capabilities, your only option is:

1. Create a new partition topic (e.g. test-partitioned) and use the partitioning strategy.
2. Migrate data (if required) from the old non-partitioned topic to the new partitioned topic.
3. Update all producers and consumers to connect to the new partition topic.

Therefore, in the early stages of system design, it is crucial to fully evaluate the partitioning requirements of the topic to avoid complex migration work later.

Summarize

Creating partitioned topics in Apache Pulsar is fundamental to achieving high performance and scalability. You can configure the Broker to automatically create partition topics, or use the Pulsar Admin API to manually and precisely control the topic creation process. Whichever approach you choose, you must keep in mind the immutability principle of partitioned topics: partition properties must be defined when the topic is created and cannot be changed subsequently. This feature requires developers to carefully consider the theme's partitioning strategy during the system design phase to ensure that future expansion needs are met.

The above is the detailed content of Creation and management of Apache Pulsar partition topics. For more information, please follow other related articles on the PHP Chinese website!