Simplifying Distributed App Complexity with App Platform’s Log Forwarding and OpenSearch

As applications become more complex both in functionality and deployment footprint, observability becomes increasingly crucial. App Platform supports metrics via our Insights features and logging via the Log Forwarding feature. In this post, we will build on the microservices architecture example by demonstrating how to enhance it with log forwarding to Managed OpenSearch.

The Complexity of Distributed Systems

Distributed applications often lead to distributed problems that are difficult to debug. Microservices architecture, while powerful, introduces a layer of complexity that can be daunting. Services need to communicate effectively, manage state, and scale independently. All these factors contribute to an intricate web of interdependencies that can make troubleshooting a nightmare.

Enter App Platform

App Platform simplifies the deployment and management of distributed apps. It allows developers to focus on coding and application logic, while App Platform handles the infrastructure. With features like Log Forwarding, Insights, and Alerts, it offers a streamlined approach to handling complex architectures.

Building on the Microservices Architecture

In a previous post, Deploying your Microservices Architecture App in App Platform, we discussed how to deploy a microservices app with a managed Kafka cluster and two components:

1. Web Service API: Receives requests, publishes messages to Kafka, and responds to the user.

2. Backend Processing Service: Consumes messages from Kafka and processes them.

This architecture allows for independent scaling of the web service API and backend processing service, providing flexibility and efficiency.

Enhancing Observability with Log Forwarding to OpenSearch (or ElasticSearch)

Now, let’s enhance this architecture with improved observability using App Platform’s Log Forwarding feature. Logs are vital for understanding what is happening within your application. By forwarding logs to OpenSearch, you can gain deeper insights and make debugging easier.

App Platform supports OpenSearch in two different ways:

1. Bring-Your-Own: In this mode, you can use any OpenSearch or ElasticSearch endpoint as long as it supports TLS and authentication via a username and password.

2. DigitalOcean Managed OpenSearch: You can create a cluster in a few minutes via the Cloud Console’s Databases page.

In this example, we’ll use a managed OpenSearch cluster for simplicity.

Step 1: Deploy the App

1. Deploy your app following the steps at described in this post

Step 2: Configuring Log Forwarding

1. Enable Log Forwarding: In the App Platform dashboard, navigate to your app and enable log forwarding from the settings tab.

2. Configure OpenSearch: Set up an OpenSearch instance where your logs will be sent. Ensure your OpenSearch endpoint is accessible and properly secured.

Alternatively, you can modify the App Spec located in the Kafka post repository.

Here’s what the changes to the app spec would look like

yaml

 name: sample-golang-kafka-2

# env and databases sections removed for brevity

services:

  - name: producer

    source_dir: producer

    github:

      repo: digitalocean/sample-golang-kafka

      branch: main

    log_destinations:

      - name: logs-producer

      open_search:

        cluster_name: my-opensearch-cluster

workers:

  - name: consumer

    source_dir: consumer

    github:

      repo: digitalocean/sample-golang-kafka

      branch: main

    log_destinations:

      - name: logs-consumer

      open_search:

        cluster_name: my-opensearch-cluster

Step 3: Configure OpenSearch Dashboard

1. Login via the link provided in the OpenSearch database dashboard using the credentials provided

2. Create an Index Pattern by navigating to Management → Dashboards Management → Index Patterns

3. Use the Default datasource

4. Specify a pattern name of ‘logs*’

5. Specify @timestamp as the Time field and click the Create index pattern button

6. Navigate to OpenSearch Dashboards → Discover

Conclusion

App Platform is designed to simplify the complexities of deploying and managing distributed, cloud-native apps. By leveraging features like Log Forwarding and Managed OpenSearch, you can enhance observability and streamline your operational experience. Try out these features and let us know how they help you manage your distributed applications more effectively.